1
00:00:07,992 --> 00:00:09,923
Welcome back to Adventures in DevOps.

2
00:00:09,923 --> 00:00:15,687
We've been talking a lot about browsers lately, reliability, testing, browser automation.

3
00:00:15,687 --> 00:00:25,043
And so to fit with that theme, I figured we'd try taking one step further and getting a
feeling about how companies decide, good or bad, to hook an LLM directly into their

4
00:00:25,043 --> 00:00:26,714
browser for some sort of automation.

5
00:00:26,714 --> 00:00:31,677
We brought on Tukaten Singh, founder and CEO at Skyvern, previously ML and Search.

6
00:00:31,677 --> 00:00:33,134
Welcome to the show.

7
00:00:33,134 --> 00:00:33,808
Good to here, Warren.

8
00:00:33,808 --> 00:00:35,394
Thanks for the thanks for having me on.

9
00:00:35,394 --> 00:00:37,095
So maybe let's just get to it.

10
00:00:37,095 --> 00:00:46,963
two weeks ago uh on this podcast, we talked a lot about Web MCP and providing better
capabilities to make it easier to potentially use LLMs to interact with web pages and not

11
00:00:46,963 --> 00:00:48,354
just the browser directly.

12
00:00:48,354 --> 00:00:49,295
Why do this at all?

13
00:00:49,295 --> 00:00:55,439
Like, why not design, say, technical processes to interact directly with like we've had
APIs forever, right?

14
00:00:55,439 --> 00:01:01,612
So if you have some sort of business process that you're running in your company, why not
try to go to your

15
00:01:01,612 --> 00:01:08,184
you know, third party vendors or or providers or the source technology and try to interact
with the APIs that are available there.

16
00:01:08,184 --> 00:01:11,897
There's a couple really interesting reasons why that that's not likely to happen.

17
00:01:11,897 --> 00:01:13,778
How many websites exist in the world?

18
00:01:13,778 --> 00:01:15,269
It's about 1.1 billion.

19
00:01:15,269 --> 00:01:17,750
But there's only about 900,000 APIs that exist.

20
00:01:17,750 --> 00:01:22,774
You know, 1.01 billion shortfall between like number of websites to APIs.

21
00:01:22,774 --> 00:01:29,188
And and if you look at why that's the case, all the w websites that like benefit by making
the customers' lives easier have APIs.

22
00:01:29,188 --> 00:01:33,851
So think like technical products, think like, you know, e commerce even to some degree has
APIs.

23
00:01:33,851 --> 00:01:36,653
Basically, with a profit center for a company, they'll have an API.

24
00:01:36,653 --> 00:01:37,888
And if it's cost center,

25
00:01:37,888 --> 00:01:40,598
Like think of any government website or your cell phone bill.

26
00:01:40,598 --> 00:01:42,620
You gotta log into like Comcast to fetch it.

27
00:01:42,620 --> 00:01:49,273
Like those guys just have the portals because legalities comply them force them to have
it, or there's some regulation that requires open data from the government.

28
00:01:49,273 --> 00:01:52,811
Those guys don't have any APIs and they have no incentives to create it.

29
00:01:52,811 --> 00:01:59,767
And so it's very unlikely that cohort of companies will adopt something like WebIn MCP,
where it requires the website owner to make a change.

30
00:01:59,767 --> 00:02:06,754
And as long as that's true, it's easier for agents to just behave like humans and access
the internet the same way that people do.

31
00:02:06,754 --> 00:02:15,070
first thought that comes to my mind is like a list of all of the restaurants that you
could potentially go to and you maybe want to make a reservation or even just see what the

32
00:02:15,070 --> 00:02:15,841
menus are.

33
00:02:15,841 --> 00:02:20,845
And that's a that's a challenge from a consumer standpoint because obviously the tools
available are limited there.

34
00:02:20,845 --> 00:02:22,496
And I think it was

35
00:02:22,496 --> 00:02:30,678
Google, a long time ago, was suggesting, they can automate the reservation process for us
by by doing something by sidestepping this and inserting an agent there.

36
00:02:30,678 --> 00:02:33,749
I think it was a duplex, I think is what they had originally called it.

37
00:02:33,749 --> 00:02:37,730
But there's still this human element and there's a lot more than just calling a phone
number.

38
00:02:37,730 --> 00:02:43,502
And those companies that are running a restaurant, basically, they they may not get any
benefit from having an API.

39
00:02:43,502 --> 00:02:48,213
Who's gonna order food from them programmatically in a way, or maybe reservations?

40
00:02:48,213 --> 00:02:51,170
Uh, but then I I do see the flip side though.

41
00:02:51,170 --> 00:03:00,594
Whereas like Domino's got a lot of publicity early on, I think it was like almost 30 years
ago, by having a like a order dominoes command inside of EverQuest.

42
00:03:00,594 --> 00:03:08,638
And you do see restaurant reservation platforms pop up all the time or ordering platforms
that want to interact with restaurants to be able to order stuff.

43
00:03:08,638 --> 00:03:19,436
And there was a big one that just got a lot of slack recently because they were AI
generating the food images for their platform where no images existed.

44
00:03:19,436 --> 00:03:26,838
And so like I think about all these things and I and I still see like I'm still praying
for this better world where it's cheaper and more reliable and you get server-side form

45
00:03:26,838 --> 00:03:27,908
validation.

46
00:03:27,908 --> 00:03:31,446
But on the flip side, you're right, the incentive may not be there, right?

47
00:03:31,446 --> 00:03:34,710
You know, maybe restaurants say, we only care about this local area.

48
00:03:34,710 --> 00:03:38,221
We already are producing as much food as we can at a particular price point.

49
00:03:38,221 --> 00:03:39,412
We don't need more customers.

50
00:03:39,412 --> 00:03:43,353
Being accessible on a particular platform or doing reservations doesn't help us.

51
00:03:43,353 --> 00:03:48,930
How do you make the decision on whether or not like do you even consider like whether it's
a good thing?

52
00:03:48,930 --> 00:03:58,177
to automate some particular accessing or of a website or an action before going and and
taking it or is just like, okay, customers have a need, they have a business process and

53
00:03:58,177 --> 00:04:00,140
we have to replicate that as best as possible.

54
00:04:00,140 --> 00:04:02,432
do definitely do care about the ethical side of the question.

55
00:04:02,432 --> 00:04:05,234
Just to preface that answer, like we are an open source company.

56
00:04:05,234 --> 00:04:07,445
You can use that product, you can like clone it, run it.

57
00:04:07,445 --> 00:04:10,697
But with our cloud platform we actually get a bunch of antibot behavior as well.

58
00:04:10,697 --> 00:04:12,979
And we did we chose actually actively not to open source that.

59
00:04:12,979 --> 00:04:17,302
Um very intentionally because we got a lot of requests to do harmful things on the
internet.

60
00:04:17,302 --> 00:04:23,192
Somebody came into our Discord one day and they're like how can I use Skyvern to create
ten thousand Reddit accounts to promote my thing.

61
00:04:23,192 --> 00:04:26,363
Like obviously like like we're like, okay, don't do that.

62
00:04:26,363 --> 00:04:29,755
Like, you know, go go maybe go to our competitors or something and do that, you know.

63
00:04:29,755 --> 00:04:40,059
Not our but but like, you know, social media spamming, like it's there's a lot of
incentive and value to be created for people by doing that, but we don't wanna be the the

64
00:04:40,059 --> 00:04:41,720
facilitators of that kind of bad behavior, right?

65
00:04:41,720 --> 00:04:46,392
So there's definitely like an open question on the ethics side of it, like like what kind
of behavior do we want to support?

66
00:04:46,392 --> 00:04:50,528
If somebody comes in, they're like, Hey, I want I wanna like fetch my telecom bill.

67
00:04:50,528 --> 00:04:53,622
every month and like the telecom website is like cloud player protected.

68
00:04:53,622 --> 00:04:55,124
Like is that unethical?

69
00:04:55,124 --> 00:04:56,585
And it's like definitely not.

70
00:04:56,585 --> 00:04:56,816
Right?

71
00:04:56,816 --> 00:04:59,038
That's in the realm of like it's okay to go do that.

72
00:04:59,038 --> 00:05:00,220
We do try to screen for that.

73
00:05:00,220 --> 00:05:04,535
Like just last week somebody like tried to like create like ten thousand anthropic
accounts through our product.

74
00:05:04,535 --> 00:05:10,231
We already banned that kind of behavior, but like but to like remind the users to not do
do nefarious things like that.

75
00:05:10,231 --> 00:05:10,646
Could you share

76
00:05:10,646 --> 00:05:19,648
a little bit about the primary use case for Skyburn and how customers end up using it and
then once we have that contact maybe we'll we can dive into some of the more interesting

77
00:05:19,648 --> 00:05:20,428
scenarios.

78
00:05:20,428 --> 00:05:26,880
Yeah, so um Skyvern is a like um a vibe coding platform for non technical people to create
their own browser automations.

79
00:05:26,880 --> 00:05:34,696
j we work with a lot of enterprises that use our product to automatically generate
insurance quotes so they can provide the lowest like surface the lowest cost thing for

80
00:05:34,696 --> 00:05:35,346
their customers.

81
00:05:35,346 --> 00:05:41,200
Um they use us to fetch invoices for like regional locations, each having their own
telecom portal.

82
00:05:41,200 --> 00:05:48,274
they'll use us for things like healthcare automations, like um every time you hire a
doctor, you have to like prove they are allowed to practice in the area they're allowed

83
00:05:48,274 --> 00:05:49,058
practice.

84
00:05:49,058 --> 00:05:54,673
By the regulation, so they need to like go to a bunch of websites to collect that data and
to make sure they get paid by the insurance carriers and so on.

85
00:05:54,673 --> 00:05:57,586
so we help with a lot of like back office use cases like that.

86
00:05:57,586 --> 00:06:03,962
Um usually it's like companies that'll like start and start scaling by having like people
do that work, where it's like literally just like go to the same website and fill out the

87
00:06:03,962 --> 00:06:07,355
same like five fields and grab a screenshot and save it over and over again.

88
00:06:07,355 --> 00:06:10,616
And then they want to scale their business, they're sort of looking for automation to do
that.

89
00:06:10,616 --> 00:06:20,433
Do you find that a lot of the scenarios where there may be a customer doing or wanting to
do something that you would maybe flavor as unethical, are they r real customers?

90
00:06:20,433 --> 00:06:23,715
Are are they people that are doing something within their business domain?

91
00:06:23,715 --> 00:06:33,842
Or is it like a completely separate uh group of users that are potentially doing something
illegal or unethical specifically and you don't really you didn't really design the

92
00:06:33,842 --> 00:06:38,926
product for them, but they they feel like they have an opportunity because it's open
source to somehow utilize it.

93
00:06:38,926 --> 00:06:42,679
We d we didn't want to limit like our UX for certain good or bad behaviors.

94
00:06:42,679 --> 00:06:48,052
However, it's pretty clear that some behavior is not good for the world, even though like
people individually might get value out of it.

95
00:06:48,052 --> 00:06:53,416
From the open source side, we don't really restrict it, but um we use playwright under the
hood in in our open source to run things.

96
00:06:53,416 --> 00:06:56,078
And if you run stuff with Playwright, basically broadcast the internet.

97
00:06:56,078 --> 00:06:58,280
It's like I'm a bot, I'm a bot, I'm a bot.

98
00:06:58,280 --> 00:07:01,392
Like, you know, stop like just FYI, like I'm a bot running the stuff.

99
00:07:01,392 --> 00:07:03,943
Uh and so that kind of takes care of itself.

100
00:07:03,943 --> 00:07:07,820
On the cloud side, we actually do have some some better stealth, if that makes sense.

101
00:07:07,820 --> 00:07:18,884
Maybe an ethical question, but if uh customer is using your platform and doing something
that will get them banned or by association or proxy their account through Skyvern, then

102
00:07:18,884 --> 00:07:25,816
it's it's a real risk for you to even execute those those actions in that way and have
that behavior associated with your your cloud accounts.

103
00:07:25,816 --> 00:07:30,117
Like if you're running on prem, then your IP addresses could be would be at risk
potentially.

104
00:07:30,117 --> 00:07:32,782
And if you're running in a cloud, then your cloud accounts.

105
00:07:32,782 --> 00:07:39,349
Those are actually surprisingly easy because w if you run stuff from like AWS, like AWS
has their own data center IP range.

106
00:07:39,349 --> 00:07:43,810
So if you run automation there, like every website can be like this coming from an AWS
data center.

107
00:07:43,810 --> 00:07:49,956
Do you find that blocking that sort of behavior is because you consider it unethical to do
that in the first place?

108
00:07:49,956 --> 00:07:53,739
Or there is some real ramification if you sort of allowed it?

109
00:07:53,739 --> 00:08:02,377
And on the flip side, I'd be like, well, if anthropic or Reddit had a concern with
allowing that, wouldn't they already need to have a mechanism in place to block that sort

110
00:08:02,377 --> 00:08:03,338
of behavior in the first place?

111
00:08:03,338 --> 00:08:10,158
And so it wouldn't really be your responsibility or your company's responsibility or the
or the product's responsibility to take an extra step and do something about it.

112
00:08:10,158 --> 00:08:12,189
I think the legalities there are quite ambig ambiguous.

113
00:08:12,189 --> 00:08:18,955
There's not not a lot of case law that goes the one way or the other because our customers
direct the AI to do what it needs to do.

114
00:08:18,955 --> 00:08:21,647
And so they're f they actually take liability because they direct it.

115
00:08:21,647 --> 00:08:24,509
But it is still our job to make sure that they are behaving well.

116
00:08:24,509 --> 00:08:25,010
To our ethics.

117
00:08:25,010 --> 00:08:26,351
I want to feel like I'm doing good in the world.

118
00:08:26,351 --> 00:08:30,414
Like I didn't start a company to be evil, but I do take some pride in blocking the bad
stuff.

119
00:08:30,414 --> 00:08:39,539
Do you find that there's like direct and expected like categories of particular unethical
or illegal behavior that you're often dealing with and you you're just like, okay, it's

120
00:08:39,539 --> 00:08:41,860
like one of three these three buckets and we're good?

121
00:08:41,860 --> 00:08:48,563
Or is every sort of scenario a potential like, well, we don't know upfront in how to
categorize this stuff.

122
00:08:48,563 --> 00:08:53,944
Everything seems like a little snowflake and we have to evaluate them like on a case by
case basis.

123
00:08:53,944 --> 00:08:55,756
So I think there's some cases where it's obvious.

124
00:08:55,756 --> 00:08:57,327
Like the the three examples I gave.

125
00:08:57,327 --> 00:09:02,901
Um LinkedIn is maybe okay, but we block it just because of the potential of abuse there.

126
00:09:02,901 --> 00:09:06,234
Reddit, I I haven't seen any good use case of people wanting to go to Reddit.

127
00:09:06,234 --> 00:09:10,187
That that that that isn't already it doesn't already have a publicly sanctioned way of
doing it.

128
00:09:10,187 --> 00:09:10,457
Yeah.

129
00:09:10,457 --> 00:09:15,992
Like if you wanna know what people are talking about, you they have RSS feeds you can
subscribe to, you don't need to go through use our product.

130
00:09:15,992 --> 00:09:16,452
Right.

131
00:09:16,452 --> 00:09:19,244
Um s same with like all the social media websites.

132
00:09:19,244 --> 00:09:24,694
Like they have they have business APIs you can use to actually access them legitimately,
so there's no reason for them to go through our product.

133
00:09:24,694 --> 00:09:28,953
and then there's a set of ambiguous use cases where it's like somebody wants to apply to a
job.

134
00:09:28,953 --> 00:09:29,784
Mm-hmm.

135
00:09:29,784 --> 00:09:31,055
Like, is it bad?

136
00:09:31,055 --> 00:09:32,026
Is it good?

137
00:09:32,026 --> 00:09:33,428
You know, it's kind of ambiguous.

138
00:09:33,428 --> 00:09:39,033
Another one is like they want to go to Amazon, but they want to like manage their
transactions for their business.

139
00:09:39,033 --> 00:09:42,797
Like Amazon doesn't have great APIs for that and like is that bad?

140
00:09:42,797 --> 00:09:43,447
Or is that good?

141
00:09:43,447 --> 00:09:44,344
Like, you know

142
00:09:44,344 --> 00:09:46,567
Just saying like going to Amazon isn't necessarily a problem.

143
00:09:46,567 --> 00:09:47,968
But then like what are they doing when they get there?

144
00:09:47,968 --> 00:09:56,588
Are they trying to go to every single web page and download all the product details and
whatnot from their competitors to do you know dynamic pricing for their own items?

145
00:09:56,588 --> 00:10:05,336
Or is it about actually just trying to automate the buying of particular items or the
management of a shopping cart or you know history of supplies?

146
00:10:05,336 --> 00:10:13,901
whatever you got vendored out so that you can manage your resource planning, you know, in
your own ERP system and just the buy through Amazon specifically without having to deploy

147
00:10:13,901 --> 00:10:21,494
your own Amazon partner web app that interacts with the APIs directly, because that may
not even exist to support you and your needs.

148
00:10:21,494 --> 00:10:31,631
I see like this weird duality where it seems like a lot of the use cases which could help
a a business automate particular internal processes or change the user experience that the

149
00:10:31,631 --> 00:10:32,231
user is getting.

150
00:10:32,231 --> 00:10:33,846
So instead of having to go through

151
00:10:33,846 --> 00:10:39,518
a bunch of different platforms or different websites to list out all of the, I don't know,
products that they need to purchase.

152
00:10:39,518 --> 00:10:45,709
Like if you're maybe if you're building a data center and you want some sort of Ethernet
cables and uh and the jacks and and wall outlets and whatever.

153
00:10:45,709 --> 00:10:50,418
And obviously not probably buying this stuff from Amazon, but even if you were, you may
have that in a spreadsheet somewhere.

154
00:10:50,418 --> 00:10:56,026
And then you have to go and you know go and actually buy each of those things and jumping
over that gap is is useful.

155
00:10:56,026 --> 00:11:01,954
I I I want to say that it does seem like the business use cases for process automation
make a lot of sense.

156
00:11:01,954 --> 00:11:02,664
And

157
00:11:02,664 --> 00:11:06,348
feel more ethical than the ones that are on the end consumer side.

158
00:11:06,348 --> 00:11:09,741
Like I I feel like the consumer things are all like, I don't know.

159
00:11:09,741 --> 00:11:10,860
That's the gray arrow.

160
00:11:10,860 --> 00:11:17,855
Well, w the other the other thing when you deal with businesses is that y often they'll
you'll have to sign a contract with them and in the contract you can just say that you're

161
00:11:17,855 --> 00:11:19,176
not allowed to do anything illegal.

162
00:11:19,176 --> 00:11:19,536
Right.

163
00:11:19,536 --> 00:11:24,139
And and like like a an actual business has a lot of incentive to follow the contract.

164
00:11:24,139 --> 00:11:30,523
Like obviously like a startup like a very small startup or like a one person shop, those
those companies they can just go declare bankruptcy and move on.

165
00:11:30,523 --> 00:11:36,127
But like an established business with like, you know, fifty employees, the probability
they want to survive for the next ten years is extremely high.

166
00:11:36,127 --> 00:11:40,590
So when you put something like that in the contract, you can just generally trust that
they're gonna do reasonable things.

167
00:11:40,590 --> 00:11:45,042
Do you have like a complex like KYC or know your customer uh process setup?

168
00:11:45,042 --> 00:11:48,536
we don't we don't enable our anti bot stuff unless we have a contract in place with our
customers.

169
00:11:48,536 --> 00:11:55,744
Uh a and at which point we would we would put in the in the MSA the like the master
service agreement that they're only allowed to use us for legally sanctioned stuff.

170
00:11:55,744 --> 00:12:01,828
In order for a customer to come on the platform and and really get the value out of it, it
will always be the case that you'll have to know who they are and what business that

171
00:12:01,828 --> 00:12:03,219
they're trying to automate.

172
00:12:03,219 --> 00:12:07,212
Or do you see in the future this shifting closer to the self service onboarding side?

173
00:12:07,212 --> 00:12:13,172
I have not yet seen where somebody wants to spend more than five hundred bucks a month
with their product and they don't want a contract in place.

174
00:12:13,172 --> 00:12:14,754
Will that happen in the future?

175
00:12:14,754 --> 00:12:20,783
Either the company wants discounts or they need some like protections in place like
healthcare companies like need like HIPAA compliance and so on.

176
00:12:20,783 --> 00:12:22,606
That naturally forces to go down this path.

177
00:12:22,606 --> 00:12:32,220
So as I mentioned, there's like WebMCP that has come out, but you obviously rightly
identified that this actually requires platforms to actually implement the mechanisms and

178
00:12:32,220 --> 00:12:35,931
basically write a whole new API just for interacting through the website.

179
00:12:35,931 --> 00:12:42,260
And if they're going to do that anyway and they're technical, why are they just not
offering an MCP server or offering a real API to interact with?

180
00:12:42,260 --> 00:12:51,205
I think it like a fifth customer, was this company in Ireland and they wanted to go
automate some like tack data retrieval from like the I Irish government and their like VP

181
00:12:51,205 --> 00:12:52,075
of operations emailed me.

182
00:12:52,075 --> 00:12:53,466
It's like, Why would anybody use your product?

183
00:12:53,466 --> 00:12:55,337
This tax authority has APIs we can use.

184
00:12:55,337 --> 00:12:55,837
That's great.

185
00:12:55,837 --> 00:12:56,477
Like, that's great.

186
00:12:56,477 --> 00:12:58,008
If there's an API use it, don't our product.

187
00:12:58,008 --> 00:12:58,378
Great.

188
00:12:58,378 --> 00:12:59,659
Can the API do this operation?

189
00:12:59,659 --> 00:13:01,920
And he's like, Oh no, it d it that has a gap.

190
00:13:02,160 --> 00:13:04,261
Can't do it And like, what about this other one?

191
00:13:04,261 --> 00:13:06,062
He's like, no, can't do that one either and I'm like

192
00:13:06,062 --> 00:13:06,662
You see the problem?

193
00:13:06,662 --> 00:13:09,693
Like even if they have an API or an MCP, what if the tool call is not available?

194
00:13:09,693 --> 00:13:13,915
We use this product called Hex for data analytics and the MCP doesn't let you schedule
reports.

195
00:13:13,915 --> 00:13:15,985
Like like you're just kinda screwed.

196
00:13:15,985 --> 00:13:19,756
You know, like if all my agents just schedule reports, I gotta go through the browser now,
there's no other option.

197
00:13:19,756 --> 00:13:21,027
And it's just like a tech company.

198
00:13:21,027 --> 00:13:24,228
You it's like not a website with zero incentive to actually create a good MCP.

199
00:13:24,228 --> 00:13:30,380
Even if there isn't a API, it might not have the coverage it the website needs to actually
be useful for people to the fullest degree.

200
00:13:30,380 --> 00:13:30,670
Or

201
00:13:30,670 --> 00:13:36,732
Worse, they think about it and then they decide that they specifically don't want to solve
the problem for that particular customer set.

202
00:13:36,732 --> 00:13:46,044
So a few weeks back, we had an architect from the Wikimedia Foundation come on where you
know they're powering Wikipedia, they're they're writing the product uh open source and

203
00:13:46,044 --> 00:13:46,634
everything.

204
00:13:46,634 --> 00:13:54,316
And you get a lot of LLM usage, a lot of customers coming in and trying to answer
questions whose answers are on the wiki somewhere.

205
00:13:54,316 --> 00:13:55,507
And it's a lot of use.

206
00:13:55,507 --> 00:14:00,258
The thought is like, well, you could be one of those companies that's like, we're not
going to offer an API for

207
00:14:00,258 --> 00:14:01,339
These users.

208
00:14:01,339 --> 00:14:03,981
And I just feel like, but you're losing so much value.

209
00:14:03,981 --> 00:14:08,301
You're like, you're basically saying we don't want users just because of the tool they're
using.

210
00:14:08,301 --> 00:14:15,130
It's like there used to be a thing where, oh, they're using the wrong browser, so they
can't access our website, or you know, that you need to have flash installed.

211
00:14:15,130 --> 00:14:17,652
And at some point you realize, wait, these are real users.

212
00:14:17,652 --> 00:14:19,323
Uh, we we should do something for them.

213
00:14:19,323 --> 00:14:24,778
And it's really surprising when tech companies don't have the APIs available to actually
even do the integration.

214
00:14:24,778 --> 00:14:30,132
And I think over time, it has to happen that we that every single company is offering.

215
00:14:30,230 --> 00:14:32,811
Endpoints that solve whatever their users want.

216
00:14:32,811 --> 00:14:37,232
Now, if you don't have any users that use LLMs, interact with your your APIs, then
whatever.

217
00:14:37,232 --> 00:14:38,683
You don't need to automate in that way.

218
00:14:38,683 --> 00:14:42,714
The mind-boggling part for me is as a company, I want to power that.

219
00:14:42,714 --> 00:14:44,815
So we offer like a very technical product.

220
00:14:44,815 --> 00:14:50,803
Our UI is like administrative and it's not like a pristine users use it eight hours a day,
every single day as part of their job.

221
00:14:50,803 --> 00:14:52,077
They almost never have to go there.

222
00:14:52,077 --> 00:14:56,290
If they're like, we want to build our own administrative portal that uses your APIs, I'm
like,

223
00:14:56,290 --> 00:14:57,492
Yes, please go do that.

224
00:14:57,492 --> 00:15:05,052
You know, because then you can get the value out that you exactly need and not have to
worry about whatever new features we're putting in or having the exact graphs or data

225
00:15:05,052 --> 00:15:05,843
that's available.

226
00:15:05,843 --> 00:15:06,975
And just seems like a no-brainer.

227
00:15:06,975 --> 00:15:10,850
And yet you see lots of companies that don't offer a way to do this out of the box.

228
00:15:10,850 --> 00:15:12,051
And it's not hard anymore.

229
00:15:12,051 --> 00:15:17,263
Like like if we were having this conversation a year ago, we'd be like, okay, we've got to
like allocate like an engineer for two weeks.

230
00:15:17,263 --> 00:15:18,934
And it's like, okay, what's the opportunity cost for that?

231
00:15:18,934 --> 00:15:22,252
Like, what are we gonna give up on the roadmap to add this feature that the small
percentage people need?

232
00:15:22,252 --> 00:15:26,856
But now it's like now I can just be like, hey, like go go go build an MCP for this.

233
00:15:26,856 --> 00:15:30,939
Like, you know, like I can literally just like automate that and comp company still don't
do it.

234
00:15:30,939 --> 00:15:36,561
There's some PM in the company that's like deciding the roadmap with opportunity costs
associated with like each each line item, right?

235
00:15:36,561 --> 00:15:37,912
And it's just not important to them.

236
00:15:37,912 --> 00:15:40,074
For government websites, they might not even have a PM, right?

237
00:15:40,074 --> 00:15:43,366
They might just not like put this on like a zombie, zombie budget, right?

238
00:15:43,366 --> 00:15:45,758
Where there's like zero like zero like maintenance.

239
00:15:45,758 --> 00:15:47,350
Like we had a customer that's so funny.

240
00:15:47,350 --> 00:15:52,754
Um, they run an automation with us that goes to like this government portal to like look
at the inventory accounts or something.

241
00:15:52,754 --> 00:15:57,048
And their IT team reached out to them and they're like, Can you stop running this
automation?

242
00:15:57,048 --> 00:15:59,800
'Cause it's running at the same time every day and it's crashing our servers.

243
00:15:59,800 --> 00:16:07,426
And instead of like making it sc instead of making it scale, they're like, actually, can
you just like stagger it so we can stop having our servers crash?

244
00:16:07,426 --> 00:16:09,969
when this thing runs and it it was like one run a day.

245
00:16:09,969 --> 00:16:11,250
It wasn't anything substantial.

246
00:16:11,250 --> 00:16:12,571
It's not even like the quality of engineering.

247
00:16:12,571 --> 00:16:18,982
It's really like the attitude that the organization has towards our web offering that that
defines this lack of functionality.

248
00:16:18,982 --> 00:16:29,219
It's it's surprising how often we see challenges show up in our own technology or like
workarounds that we have to build because our providers didn't build it in the optimal

249
00:16:29,219 --> 00:16:29,429
way.

250
00:16:29,429 --> 00:16:33,732
I think the number one that always shows up for us is lack of idempotency on APIs.

251
00:16:33,732 --> 00:16:37,325
So like y something is gonna crash at some point for sure.

252
00:16:37,325 --> 00:16:39,576
And you're like, I have to retry this.

253
00:16:39,596 --> 00:16:42,798
And you go to retry it and you're like, Well, wait a second.

254
00:16:42,798 --> 00:16:46,271
This is like a dangerous operation that we don't ever want to do twice.

255
00:16:46,271 --> 00:16:48,330
And yet this API doesn't support.

256
00:16:48,330 --> 00:16:51,171
A way to actually uh deduplicate it.

257
00:16:51,171 --> 00:16:52,690
We had we have we have these all the time.

258
00:16:52,690 --> 00:16:57,092
We interact with email, which SMTP doesn't support item potency really.

259
00:16:57,092 --> 00:17:02,174
I mean, you can use a message ID sort of, but most email senders like it's so ridiculous
to me.

260
00:17:02,174 --> 00:17:10,516
Most email senders don't let you specify in an API like what the message ID is that's
generated by the provider after they've sent the email.

261
00:17:10,516 --> 00:17:12,457
And like that's the number one absurd thing.

262
00:17:12,457 --> 00:17:14,897
But even like Slack and Discord don't do this.

263
00:17:14,897 --> 00:17:17,514
There's no way to deduplicate sending messages there.

264
00:17:17,514 --> 00:17:19,405
Stripe for the longest time didn't support this.

265
00:17:19,405 --> 00:17:27,228
Like you want to send a charge and you can sort of send a message to deduplicate it, but
they only support like specific IDs and only within five minutes.

266
00:17:27,228 --> 00:17:28,659
And they didn't do that always.

267
00:17:28,659 --> 00:17:31,060
And I'm I'm just like, I don't want to double charge this.

268
00:17:31,060 --> 00:17:34,121
And so you you write some ridiculous extra automation.

269
00:17:34,121 --> 00:17:38,158
And like these are companies that that are so fundamentally technical.

270
00:17:38,158 --> 00:17:43,305
Like they're offering like first class APIs to do this thing, and they don't even have the
necessary functionality.

271
00:17:43,305 --> 00:17:46,550
So it makes a lot of sense to me that we're getting to the point where.

272
00:17:46,550 --> 00:17:49,396
You don't even trust a company to write the API correctly.

273
00:17:49,578 --> 00:17:53,260
If the the UI exists, you know, that becomes the interface that everyone's using.

274
00:17:53,260 --> 00:17:55,951
Like item points is like an advanced topic, right, for engineers.

275
00:17:55,951 --> 00:18:02,934
Like like the stuff I'm talking about is like you you wanna you go on a to like register a
company and you you're like shit it's one AM.

276
00:18:02,934 --> 00:18:06,165
I wanna go to the IRS website and get an EIN number so I can do so I can do payroll.

277
00:18:06,165 --> 00:18:10,577
It's it's it's offline from one AM to s s nine AM every day.

278
00:18:10,577 --> 00:18:11,848
You can't access the website.

279
00:18:11,848 --> 00:18:19,581
You know, like I'm talking like like stuff that, you know, was common I guess in like the
two thousands, early two thousands and like late nineties, but still there.

280
00:18:19,581 --> 00:18:21,492
That's the kind of websites our customers send out to send to.

281
00:18:21,492 --> 00:18:22,152
It's different world.

282
00:18:22,152 --> 00:18:22,828
It's like

283
00:18:22,828 --> 00:18:24,369
Blows my mind every day.

284
00:18:24,558 --> 00:18:30,230
Do you find that a certain kind of company is focused more on the process automation than
than others?

285
00:18:30,230 --> 00:18:32,821
Actually, we solve like a very specific problem in the world.

286
00:18:32,821 --> 00:18:35,872
There's this like entire industry called the RPA industry.

287
00:18:35,872 --> 00:18:37,633
it stands for robotic process automation.

288
00:18:37,633 --> 00:18:42,455
The the category exists to like automate repetitive work within large businesses.

289
00:18:42,455 --> 00:18:51,399
And the way the in in industry worked is there's these like people called RPA engineers
who are good engineers and they learn how to use these like archaic tools like UiPath.

290
00:18:51,399 --> 00:18:59,586
What they do is they like go around the company and they like interview these like
business leaders and business team members to figure out what work they do all the time.

291
00:18:59,586 --> 00:19:01,067
Write that down and then automate it.

292
00:19:01,067 --> 00:19:06,258
The person who actually knows the work to automate is like some non technical person who
is doing the actually doing the work every day.

293
00:19:06,258 --> 00:19:07,341
They actually know it inside out.

294
00:19:07,341 --> 00:19:08,826
They all the edge cases that could happen.

295
00:19:08,826 --> 00:19:12,555
Uh if the website changes in a small way, they can like they they exactly what to do to
fix it.

296
00:19:12,555 --> 00:19:19,900
Where they'd have like they know everything, they have to write it down, forget like half
the stuff that they know in their head, hand it off to this guy to go program, which is

297
00:19:19,900 --> 00:19:22,461
honestly like not super interesting programming work anyways.

298
00:19:22,461 --> 00:19:26,464
When it would be deployed, they'd run into a bunch of edge cases and they'd have to go
back and forth.

299
00:19:26,464 --> 00:19:27,805
It would take weeks to automate.

300
00:19:27,805 --> 00:19:29,346
Like with our tool, this person

301
00:19:29,346 --> 00:19:31,559
can now actually just go build their own automation.

302
00:19:31,559 --> 00:19:37,437
Like we we we just go after companies are already kind of know the problem that they want
to automate and they just want like a shorter feedback cycle.

303
00:19:37,720 --> 00:19:45,485
That's sort of the interesting learning that a lot of companies will face is that it's
like the amount that you spend on automation actually increases over time if the cost of

304
00:19:45,485 --> 00:19:47,366
the automation decreases, right?

305
00:19:47,366 --> 00:19:50,398
So when the cost of the automation decreases, the ROI increases.

306
00:19:50,398 --> 00:19:58,563
And so like the amount that you're willing, the number of things that you're willing to
automate actually increases over time because you're now getting a positive ROI on on more

307
00:19:58,563 --> 00:19:59,103
stuff.

308
00:19:59,103 --> 00:20:04,108
And so companies actually end up spending more over time ah now that automation is
cheaper.

309
00:20:04,108 --> 00:20:06,390
Which is sort of this like confusing paradox.

310
00:20:06,390 --> 00:20:07,881
but I I I think it makes a lot of sense.

311
00:20:07,881 --> 00:20:12,185
Programming stuff for the browser environment is just a huge headache from my standpoint.

312
00:20:12,185 --> 00:20:19,372
And in the browser episode that where we talked about this, there was like there's a lot
of things that just seem like they go wrong all the time, or not built well, just a

313
00:20:19,372 --> 00:20:20,101
challenge to get around.

314
00:20:20,101 --> 00:20:26,586
Are are you fundamentally trusting Playwright to do most of the management of the
integration with the browser?

315
00:20:26,586 --> 00:20:27,907
Remember iframes?

316
00:20:27,907 --> 00:20:30,289
Remember Shadow DOM roots?

317
00:20:30,289 --> 00:20:31,220
they're not over.

318
00:20:31,220 --> 00:20:32,271
Like all this crap.

319
00:20:32,271 --> 00:20:37,816
Like I remember when we first open sourced our product, the first PR we got was somebody
adding ice i frames before Discovered.

320
00:20:37,816 --> 00:20:39,786
They're like, I need this shit to work.

321
00:20:39,786 --> 00:20:40,342
You know?

322
00:20:40,342 --> 00:20:47,345
Uh and it's like everybody's got these battle scars from like when when when they're
building automations, but uh the Chrome browser.

323
00:20:47,345 --> 00:20:49,387
It's like an unintelligible protocol.

324
00:20:49,387 --> 00:20:53,250
So playwright just makes it like legible for people to use.

325
00:20:53,250 --> 00:20:59,622
They're being replayed against Chrome primarily, or is the browser selection part of the
automation that you're creating?

326
00:20:59,622 --> 00:21:02,536
this is actually a very common feature request we get is to support other browsers.

327
00:21:02,536 --> 00:21:06,712
I if I listen to our customers who want to do QA, they care a lot about the different set
of browsers.

328
00:21:06,712 --> 00:21:09,335
But if I listen to our customers who want to do automation, they don't get they only want
Chrome.

329
00:21:09,335 --> 00:21:10,517
Chrome or like Microsoft Edge.

330
00:21:10,517 --> 00:21:12,339
Like like that's all they care about.

331
00:21:12,500 --> 00:21:13,321
Which is also Chrome.

332
00:21:13,321 --> 00:21:14,291
Yeah, exactly.

333
00:21:14,291 --> 00:21:16,301
So I think I think I'm gonna get some emails about this.

334
00:21:16,301 --> 00:21:24,514
I will say I keep trying to like Firefox, and every couple of years, they always do
something where I'm I'm like, I'm using Chrome.

335
00:21:24,514 --> 00:21:27,655
I mean, I'm I'm not using Chrome, I'm using Brave, but I'll say Chromium for now.

336
00:21:27,655 --> 00:21:30,816
Everyone can just plug in whatever version of Chromium they're using.

337
00:21:30,816 --> 00:21:32,547
And I'm like, you know what?

338
00:21:32,547 --> 00:21:33,827
This is just getting worse.

339
00:21:33,827 --> 00:21:35,357
I really wish I could switch to Firefox.

340
00:21:35,357 --> 00:21:37,186
And like then I go and I'm like, wait.

341
00:21:37,186 --> 00:21:42,890
Firefox last year just had another whatever terrible, you know, and then insert whatever
scandal thing.

342
00:21:42,890 --> 00:21:44,361
And it just happens over and over again.

343
00:21:44,361 --> 00:21:53,328
Like I think the biggest one that just happened this year was a change to their their
terms and services about like training data, like training on they they they made a change

344
00:21:53,328 --> 00:21:58,532
to that like they're allowed to potentially train on the gestures that you're using or wh
whatever your history is.

345
00:21:58,532 --> 00:22:01,265
And I'm just like, okay, I guess I'm not switching this year either.

346
00:22:01,265 --> 00:22:04,888
And it was really sad because early on I was definitely a Firefox user.

347
00:22:04,888 --> 00:22:05,346
Same.

348
00:22:05,346 --> 00:22:13,901
I ended switching for one of those reasons and I keep wanting to go back because I hear
about great things and then I hear about wait, they don't really support pass keys in

349
00:22:13,901 --> 00:22:19,574
every scenario, or wait, they don't they don't support this security thing or that thing
and uh it's just it's just over.

350
00:22:19,574 --> 00:22:21,435
I do get it from an automation standpoint.

351
00:22:21,435 --> 00:22:24,977
The user shouldn't care or know about what where the automation is running.

352
00:22:24,977 --> 00:22:26,638
That's they don't they don't care about that.

353
00:22:26,638 --> 00:22:33,602
They care more that it's happening and the right thing is happening and a a tool that's
dedicated to a single set of browsers.

354
00:22:33,602 --> 00:22:37,064
That does the right thing more reliably makes a lot more sense.

355
00:22:37,064 --> 00:22:43,669
In an episode where we talked about test automation, we were reviewing the challenges with
doing any sort of testing with a browser.

356
00:22:43,669 --> 00:22:55,097
And one of the things that came up was there was a company that had done an amazing thing
where they allowed you to write prompts to execute QA, you know, automation tests against

357
00:22:55,097 --> 00:22:56,618
like your website running in the browser.

358
00:22:56,618 --> 00:22:58,589
And I'm just like, why would you do that?

359
00:22:58,589 --> 00:23:01,361
Did all the tools exist to do this deterministically?

360
00:23:01,361 --> 00:23:02,520
You don't want test.

361
00:23:02,520 --> 00:23:07,393
To run through a prompt and they're like, yeah, well, we did it, we did it because we
could.

362
00:23:07,393 --> 00:23:11,675
Just like the we we've had solutions to this for for so long.

363
00:23:11,675 --> 00:23:16,658
Uh it's not like we needed LLMs to start doing unit tests against the browser.

364
00:23:16,658 --> 00:23:23,896
And if anything, we with products like Skyburn, we definitely see the market moving more
towards forgetting about the browser, forgetting about the website.

365
00:23:23,896 --> 00:23:31,689
Google published like a really interesting paper on this in twenty in twenty twenty one,
so about three years before LMs well four two years before LMs got really big.

366
00:23:31,689 --> 00:23:33,150
And it was called semantic locators.

367
00:23:33,150 --> 00:23:37,692
Basically, can you use natural language to identify which selectors to interact with on a
website?

368
00:23:37,692 --> 00:23:38,592
No.

369
00:23:38,652 --> 00:23:44,740
And and QA was actually the primary topic of that because if somebody updates like the
location of that button, should should that test case break?

370
00:23:44,740 --> 00:23:47,506
Like unit tests, another spicy topic, like unit tests are mostly garbage.

371
00:23:47,506 --> 00:23:52,298
Like you know, like no sorry, sorry for all the numbers, you know, like they're mostly
garbage.

372
00:23:52,694 --> 00:23:59,266
Let's say you've got a search bar on your website and you want to test that searching for
a particular product in your e-commerce catalog is actually returned and someone can add

373
00:23:59,266 --> 00:23:59,900
it to their card.

374
00:23:59,900 --> 00:24:07,725
The way you do this is you may write some automation, which first finds the search bar, it
then enters some characters in there, then it clicks the search button and then finds the

375
00:24:07,725 --> 00:24:11,008
list of product results and clicks on the right one and goes to the rest of the flow.

376
00:24:11,008 --> 00:24:14,200
The challenge is like, where is the search bar on the screen?

377
00:24:14,200 --> 00:24:15,211
Like, how do you even find that?

378
00:24:15,211 --> 00:24:19,168
Now, the strategy has historically has been well, there is an input.

379
00:24:19,168 --> 00:24:24,433
element in in HTML where it's probably that, but there's lots of other input elements on
the screen.

380
00:24:24,433 --> 00:24:27,305
So you can add a class or an ID to that element.

381
00:24:27,305 --> 00:24:30,689
And so you can easily find that if you're escraping the DOM.

382
00:24:30,689 --> 00:24:33,098
Now the problem is that you may not have done that.

383
00:24:33,098 --> 00:24:35,173
Uh the class may have changed.

384
00:24:35,173 --> 00:24:39,417
Or if you're using an LLM to sort of generate the test, you're like, yeah, go to the
search bar and enter the text.

385
00:24:39,417 --> 00:24:40,898
There may it may not say search.

386
00:24:40,898 --> 00:24:47,562
Like if you've done any sort of uh LinkedIn automation, you'll note that like all the
classes and all the things that are in the DOM, they don't they don't reference anything

387
00:24:47,562 --> 00:24:47,962
that's there.

388
00:24:47,962 --> 00:24:49,543
You have no idea what the text is.

389
00:24:49,543 --> 00:24:57,827
So you could try to find the word search on a button somewhere and hopefully that that's
the the search button, but it may not say search, it may say something else.

390
00:24:57,827 --> 00:25:02,500
And the text of that may change over time, especially if you're using any sort of locale
specific stuff.

391
00:25:02,500 --> 00:25:06,372
The button may not actually say search in any sort of QA environment, it may say something
else.

392
00:25:06,372 --> 00:25:09,556
And also king off of the text uh

393
00:25:09,556 --> 00:25:12,128
in uh an element seems like it's it's risky.

394
00:25:12,128 --> 00:25:21,875
So when I see these tests, yeah, the the whole goal of automation is obviously the where
we want to get to, but it really does require the website or the the development of the

395
00:25:21,875 --> 00:25:24,817
website to be implemented in a way that still allows it.

396
00:25:24,817 --> 00:25:32,392
And so I I think over time, one direction that we will see that going is more specific
elements that pop up, right?

397
00:25:32,392 --> 00:25:37,548
Like instead of just a search an input for for search, it will be a search input which has

398
00:25:37,548 --> 00:25:39,539
you know, additional expectations around it.

399
00:25:39,539 --> 00:25:47,104
And I think browsers are very slow to get there, but I think it is a requirement to make
the next step, just like we've added a lot of things for accessibility, like paragraphs

400
00:25:47,104 --> 00:25:56,159
and summaries and descriptions that actually have first class meanings and can be used in
this way, rather than, yes, find the fourth paragraph object in the third div that's in

401
00:25:56,159 --> 00:26:03,994
there and the one that has the span is the one we want to, you know, uh, you know, look at
for the actual value because there's no class or ID

402
00:26:04,034 --> 00:26:10,564
I see more things moving towards like real time rendering generation, client side in
information being generated generated and so on.

403
00:26:10,564 --> 00:26:11,915
let's let's fight about this.

404
00:26:11,915 --> 00:26:21,383
so I I uh I'm actually doing a talk this month, the challenges of of getting a dashboard
spun up for a reasonable amount of money looking at some business metrics.

405
00:26:21,644 --> 00:26:31,212
And that that that's not really where I'm going, but part of this was an experiment that
that we did using what what is commonly coined as generative UI now, which is basically

406
00:26:31,212 --> 00:26:35,895
dynamically generate a UI to render some specific data that you care about.

407
00:26:35,895 --> 00:26:39,128
So maybe you have the database of data and you're like, I don't want to.

408
00:26:39,128 --> 00:26:40,759
have a static dashboard.

409
00:26:40,759 --> 00:26:43,721
I want a dashboard that shows me the most relevant thing.

410
00:26:43,721 --> 00:26:44,982
Well, what's most relevant?

411
00:26:44,982 --> 00:26:45,763
How do you figure that out?

412
00:26:45,763 --> 00:26:50,386
Well, you can look at some data, perform some queries, and then feed it into an LLM and
get back a result.

413
00:26:50,386 --> 00:26:58,781
But every single step along the way where we were doing this is still so early that we
pretty much had to define every possible thing you could possibly imagine.

414
00:26:58,781 --> 00:27:02,614
So like the colors that the graphs have to be, just so that you can see it like

415
00:27:02,614 --> 00:27:05,696
And it's from an accessibility standpoint or user experience standpoint.

416
00:27:05,696 --> 00:27:06,966
Like what is even a graph?

417
00:27:06,966 --> 00:27:12,089
Like the difference between when something should be a line graph and when something
should be a bar graph or when it should be a table or a pie graph.

418
00:27:12,089 --> 00:27:14,671
Like you had to explain to it what those things are.

419
00:27:14,671 --> 00:27:18,332
And so like I'm sitting there and I'm going over uh the whole development that we've done.

420
00:27:18,332 --> 00:27:27,077
And it's like we pretty much just explained how to build a dashboard product in natural
language so it could be passed to the system prompt of the model so it could generate this

421
00:27:27,077 --> 00:27:27,528
stuff.

422
00:27:27,528 --> 00:27:29,659
And we also had to take a lot of extra annoying steps.

423
00:27:29,659 --> 00:27:32,334
Like we had to find a graphing framework and then we had to

424
00:27:32,334 --> 00:27:40,457
proxy all of the elements and components that we would use that to put it in a form that
the that we could pass into the system prompt so that the LM would understand like what to

425
00:27:40,457 --> 00:27:42,317
actually generate specifically.

426
00:27:42,317 --> 00:27:51,640
And then we had it generate SQL queries to fetch the data dynamically, but we had to
explain what SQL was, like what type of queries make sense, like what is a what what

427
00:27:51,640 --> 00:27:56,111
belongs in a like a like statement and what you know what's valid, what's not valuable.

428
00:27:56,111 --> 00:27:58,198
And then we still weren't done.

429
00:27:58,198 --> 00:28:01,581
We we're like, well, we need that we need the prompt to actually generate the right
graphs.

430
00:28:01,581 --> 00:28:05,094
But if you say show me what's actually relevant, it still was doing the wrong thing.

431
00:28:05,094 --> 00:28:12,501
And so now we have a bunch of prompts where it's like, okay, this is prompt A, where it
has like, I want show me four graphs where the first graph is a line graph and the x-axis

432
00:28:12,501 --> 00:28:18,297
is a date and the y-axis is a list of counts, and you know, here are the lines that they
should be, and the second graph is and so on and so forth.

433
00:28:18,297 --> 00:28:20,519
And so like after we're done, I'm just like, why did we even do this?

434
00:28:20,519 --> 00:28:24,246
We pretty much did all of the software development and then converted into national
language.

435
00:28:24,246 --> 00:28:28,252
natural language just so an LLM could generate it for us after the fact.

436
00:28:28,252 --> 00:28:30,374
That was definitely not the right solution.

437
00:28:30,438 --> 00:28:32,909
what what like the question I'll ask is like what model?

438
00:28:32,909 --> 00:28:40,125
Because like when we started Skyburn, this is GPT four era of models and people would be
like, go apply to this job.

439
00:28:40,305 --> 00:28:45,548
And it would fill out the form and click click submit and then maybe there's like a
question missing and it'd be like I'm done.

440
00:28:45,969 --> 00:28:46,980
I did it successfully.

441
00:28:46,980 --> 00:28:53,404
Like and we had to like we had to do write so much code around the model to make sure it
was actually gonna do the right thing.

442
00:28:53,404 --> 00:28:59,682
Um but with the latest generation of models, like I I did we did a full redesign of our
product a few months ago and

443
00:28:59,682 --> 00:29:07,546
We the b basically the redesign was just deleting all the shit we wrote like last year
like the previous year 'cause the new models just did it implicitly.

444
00:29:07,546 --> 00:29:08,686
Like the new models just like did it.

445
00:29:08,686 --> 00:29:11,848
Like they they they they didn't need to be instructed to be that specific.

446
00:29:11,848 --> 00:29:15,249
They they took reasonable I mean, they made reasonable assumptions.

447
00:29:15,249 --> 00:29:22,162
Like we still have I would say we still have too many instructions in there, but like the
the cohort of instructions and post processing like maybe this long and now it's like this

448
00:29:22,162 --> 00:29:22,673
long.

449
00:29:22,673 --> 00:29:23,663
And a lot of it was deletion.

450
00:29:23,663 --> 00:29:25,932
So I wonder I wonder if that's gonna

451
00:29:25,932 --> 00:29:27,242
It it could be coming.

452
00:29:27,242 --> 00:29:32,344
like I yeah, I literally so I literally wrote this prompt like a week ago.

453
00:29:33,943 --> 00:29:36,065
yeah, uh so don't get me wrong.

454
00:29:36,065 --> 00:29:43,907
It for sure is much better since there has been more documentation on basically the
semantic layer for writing queries.

455
00:29:43,907 --> 00:29:50,789
So the TLDR there is a way to inform the model like what sort of data you're saving and
like what different columns actually mean.

456
00:29:50,789 --> 00:29:53,930
And before that it was definitely atrocious, like no question.

457
00:29:53,930 --> 00:29:55,478
Good good luck even getting

458
00:29:55,478 --> 00:29:59,840
something that matched the system prompt or understanding that we are trying to generate a
dashboard.

459
00:29:59,840 --> 00:30:07,503
But I I just at this point with a system prompt and with a with the actual user prompts,
we are getting the right things to show up there.

460
00:30:07,503 --> 00:30:08,884
It's not what we want.

461
00:30:08,884 --> 00:30:14,666
Like it's not intelligent about about what should actually show up or what is important in
a particular use case.

462
00:30:14,666 --> 00:30:17,337
And that's the part where it would feel good to automate.

463
00:30:17,337 --> 00:30:21,299
Like let's say you have an incident and there's some data around the incident.

464
00:30:21,299 --> 00:30:24,450
Maybe there's like some null reference exceptions or some customer reports.

465
00:30:24,450 --> 00:30:31,134
Or some five hundreds or whatever have you, and you want a bunch of dashboards that are
automatically generated based off of the data that you have.

466
00:30:31,134 --> 00:30:37,238
I don't think it's I don't think we're at the point where it's going to know what
dashboards to show you, like what to even figure out what the incident is.

467
00:30:37,238 --> 00:30:40,639
Yeah, or or worse, like it might show you insufficient information.

468
00:30:40,639 --> 00:30:42,360
I find I find outlines do that a lot.

469
00:30:42,360 --> 00:30:45,611
Like particularly like if you use the open ad models, like they're they're lazy.

470
00:30:45,611 --> 00:30:50,103
And and they'll be like, Here here's three of the ten things you might care about.

471
00:30:50,343 --> 00:30:51,704
Let me know if you want more.

472
00:30:52,244 --> 00:30:55,065
And I'm like, God, I just need I I'm having it to it right now, give me everything.

473
00:30:55,065 --> 00:30:55,816
What are you doing?

474
00:30:55,816 --> 00:30:56,826
Yeah.

475
00:30:57,312 --> 00:31:05,979
One of the huge critical failings of the current LLM technology though, what's based up,
you know, from the uh BERT paper that came out of Google, it's basically a transformer

476
00:31:05,979 --> 00:31:06,569
architecture.

477
00:31:06,569 --> 00:31:08,171
It can't generate new things.

478
00:31:08,171 --> 00:31:10,793
It just takes whatever the input is and transforms it.

479
00:31:10,793 --> 00:31:18,579
So realistically, if you don't tell it I want 10, you know, graphs, it it's just gonna
make up what it thinks is the right number of graphs.

480
00:31:18,579 --> 00:31:21,862
If you don't tell it you want a particular thing, it's not gonna know to include that.

481
00:31:21,862 --> 00:31:27,212
You can sort of try to get it to prompt in the real stuff, like have an output that
contains some.

482
00:31:27,212 --> 00:31:31,305
new and novel ideas, but you still had to sort of be like, make a breakthrough.

483
00:31:31,305 --> 00:31:33,847
And as far as you know what it's gonna actually generate.

484
00:31:33,847 --> 00:31:38,260
And it's just unless you're very specific, I I just find it's never really gonna get to
that level.

485
00:31:38,260 --> 00:31:44,875
Like obviously the graphs it generates will be maybe it'll be better accessible because
we'll have the right tools, or maybe it will include more relevant data.

486
00:31:44,875 --> 00:31:45,466
that would be better.

487
00:31:45,466 --> 00:31:48,344
But it's like it's always gonna have a problem in some area.

488
00:31:48,344 --> 00:31:48,675
Yeah.

489
00:31:48,675 --> 00:31:56,244
But this is the dimension that I'm like really excited about improving 'cause like I I
write really short prompts and I I get pissed when they don't follow.

490
00:31:56,244 --> 00:31:58,667
I'll be like F you like come on, like you don't be an idiot.

491
00:31:58,667 --> 00:31:58,947
Right.

492
00:31:58,947 --> 00:32:01,270
But I found myself doing that less with Fable.

493
00:32:01,270 --> 00:32:04,152
Um so I wonder how true that'll be in six months.

494
00:32:04,152 --> 00:32:14,120
will still be at the point where given a prompt and getting a response, I'll still feel
the n need to share expletives back to the L L based off of my current

495
00:32:14,140 --> 00:32:15,153
Those are legal now.

496
00:32:15,153 --> 00:32:18,009
Dorbike changing their terms and servers we can't swear swear at them anymore.

497
00:32:18,009 --> 00:32:19,040
So it's

498
00:32:20,431 --> 00:32:23,533
I I can't like if I swear, will they terminate my account?

499
00:32:23,533 --> 00:32:27,105
Because because I I was already planning on leaving.

500
00:32:27,105 --> 00:32:37,562
Uh I I know a lot of our company has already transitioned off of Claude, but yeah, I mean
if they close my account because I do that, then uh yeah, I'll I'll go I'm going somewhere

501
00:32:37,562 --> 00:32:38,742
else anyway.

502
00:32:39,690 --> 00:32:40,937
I get really angry.

503
00:32:40,937 --> 00:32:41,865
I I definitely do.

504
00:32:41,865 --> 00:32:48,449
So uh stuff not going my way, especially if I'm tell it to, you know, interact with the
browser in a particular way and it doesn't do that.

505
00:32:48,449 --> 00:32:49,534
but I'm not there yet.

506
00:32:49,534 --> 00:32:53,644
I I don't feel like I want to automate my social media stuff.

507
00:32:53,644 --> 00:33:01,732
No, but I I I I set up a personal agent and I swear a lot of mine to the point where like
our one of our interns like started forwarding all my like expletives to like this like

508
00:33:01,732 --> 00:33:09,819
highlights channel and I like, Okay, I gotta stop stop doing that because like these are
being memorialized whenever I'm like, my god, don't be an idiot

509
00:33:09,819 --> 00:33:13,163
when when you become the meme you've you know you've gone too far.

510
00:33:14,146 --> 00:33:18,574
Week to week, month to month, when new releases come out, there must be some breaking
changes.

511
00:33:18,574 --> 00:33:25,388
Know if you see many captchas as you browse, but I went to my neighbor's house one day and
his friend had convinced him to start using some like stretchy VPN for his privacy.

512
00:33:25,388 --> 00:33:27,979
He's a plumber, he doesn't really know.

513
00:33:28,260 --> 00:33:35,404
And every time he clicked anything, he would see like these like nasty captures where it
would be like it it would basically be like, you can figure out which button is flashing,

514
00:33:35,404 --> 00:33:37,856
it's like this cadence and other things are happening.

515
00:33:37,856 --> 00:33:40,347
And to be honest, those were like easier for machines to solve than people.

516
00:33:40,347 --> 00:33:41,207
Yeah.

517
00:33:42,469 --> 00:33:42,819
Done.

518
00:33:42,819 --> 00:33:44,150
Like some of them are getting really crazy.

519
00:33:44,150 --> 00:33:44,994
Like I

520
00:33:44,994 --> 00:33:48,657
Like there was one where I saw the other day where it was like basically build a
platformer.

521
00:33:48,657 --> 00:33:53,040
Like you have to like move these things around to get like the character to go across and
like reach the flag at the end.

522
00:33:53,040 --> 00:33:55,502
To sell the captains basically like build a Mario level.

523
00:33:55,502 --> 00:33:56,963
Captures are just so wrong.

524
00:33:56,963 --> 00:33:58,083
They're just targeted the wrong way.

525
00:33:58,083 --> 00:34:01,107
Like, sure, they may have a problem with too many account signups, right?

526
00:34:01,107 --> 00:34:04,890
And historically, the way to do that is sort of user identity verification.

527
00:34:04,890 --> 00:34:05,871
No, know your customer.

528
00:34:05,871 --> 00:34:07,523
Now, obviously, that's challenging.

529
00:34:07,523 --> 00:34:14,979
There are very few countries in the world that have a giant program set up to make sure
that your online identity matches a you know real real identity.

530
00:34:14,979 --> 00:34:21,585
And so as long as you allow virtual people, you know, virtual accounts set up, you're
gonna end up with this problem somewhere.

531
00:34:21,585 --> 00:34:22,306
You're absolutely right.

532
00:34:22,306 --> 00:34:22,837
As

533
00:34:22,837 --> 00:34:27,260
the agility, you know, goes forward, the complexity of the captures will continue to
change.

534
00:34:27,260 --> 00:34:31,352
And the only time that they were nice for humans was never.

535
00:34:31,352 --> 00:34:34,384
Like they've always been a challenge for us to solve.

536
00:34:34,384 --> 00:34:37,217
This is like proof of work for humans, right?

537
00:34:37,217 --> 00:34:41,959
PayPal invented captchry, but they're the ones that invented it to fight like Russian mob
rigs that like scamming money off their thing.

538
00:34:41,959 --> 00:34:49,623
PayPal like had this like huge uh fraud problem and they were trying to find ways to
figure out like which users are are automated, which ones are not.

539
00:34:49,623 --> 00:34:52,665
because they kept getting fraudulent accounts that like siphoning money from their users.

540
00:34:52,665 --> 00:34:54,686
They invented captchas to solve that problem.

541
00:34:54,686 --> 00:34:57,572
It was the captchas, you the one you where it's like slightly tinted words.

542
00:34:57,572 --> 00:35:02,470
back then, like the none of the machine learning algorithms could actually solve it and
basically felt their fraud problem overnight.

543
00:35:02,470 --> 00:35:03,362
And so if you want

544
00:35:03,362 --> 00:35:06,636
to thank a company for captives or or or blame them.

545
00:35:06,636 --> 00:35:08,408
You know, that's that's kind of the root of that.

546
00:35:08,408 --> 00:35:11,041
So before nineteen ninety nine, you know, you could rouse the web, no problem.

547
00:35:11,041 --> 00:35:13,260
Like the good old days, as they say.

548
00:35:13,260 --> 00:35:24,435
I find that this whole sort of thing is just a a good approximation of how bad we are at
long term strategy as humans, because realistically we know that on one hand, we believe

549
00:35:24,435 --> 00:35:27,297
that LLMs will handle more things for us.

550
00:35:27,297 --> 00:35:29,958
And on the other hand, we're like, but we can have captures to stop them.

551
00:35:29,958 --> 00:35:33,770
I'm like, well, yes, but then we'll have the captches solved by the LLMs.

552
00:35:33,770 --> 00:35:39,256
And then they're only getting to a point where the problem is so challenging to solve, as
you pointed out, you have to create a

553
00:35:39,256 --> 00:35:44,023
perfectly working platformer game and then you know also run that in order to prove that
you are a human.

554
00:35:44,023 --> 00:35:48,290
And I think we're gonna very quickly get to the philosophical problem, which is like, what
is a human?

555
00:35:48,290 --> 00:35:51,692
I think the line is deturing test is I think we're well past that point now.

556
00:35:51,692 --> 00:35:53,093
The captchas are a Turing test, right?

557
00:35:53,093 --> 00:35:53,373
Yeah.

558
00:35:53,373 --> 00:35:54,364
This is how bad we are.

559
00:35:54,364 --> 00:35:57,747
Companies believe that the appropriate Turing test to have is a captcha.

560
00:35:57,747 --> 00:36:00,009
And clearly it's not working, right?

561
00:36:00,009 --> 00:36:02,511
Uh I will say we found a much better strategy.

562
00:36:02,511 --> 00:36:05,174
So we have login and access control as a product.

563
00:36:05,174 --> 00:36:09,618
And there are some interactions that we want to basically prevent malicious attackers from
utilizing.

564
00:36:09,618 --> 00:36:14,082
The one thing that a lot of people forget is that attackers don't have an infinite source
of resources and money.

565
00:36:14,082 --> 00:36:16,974
So they will only spend so much on doing something.

566
00:36:16,974 --> 00:36:19,308
And so for us, we just have a proof of work.

567
00:36:19,308 --> 00:36:24,159
hidden from the user running in the browser on every single dangerous interaction.

568
00:36:24,159 --> 00:36:33,438
So if it is a malicious automation that's trying to uh either do credential stuffing to
guess accounts or email addresses, they can do that all they want, but they're gonna have

569
00:36:33,438 --> 00:36:35,082
to spend a lot of money to make this happen.

570
00:36:35,082 --> 00:36:36,903
And it it increases over time.

571
00:36:36,903 --> 00:36:41,384
Like if you want to do two requests or three requests, you have to spend more and more
users never find out about this.

572
00:36:41,384 --> 00:36:46,946
They go once, they enter their username and hopefully they have a passkey or unfortunately
a password.

573
00:36:46,946 --> 00:36:47,746
And if they

574
00:36:47,746 --> 00:36:48,916
They get it right, then they're fine.

575
00:36:48,916 --> 00:36:51,909
And if they get it wrong, they're well, then they suffer a little penalty.

576
00:36:51,909 --> 00:36:55,881
But with automation, they just hit the wall, you know, almost instantaneously.

577
00:36:55,881 --> 00:36:57,774
And then all the attackers drop off there.

578
00:36:57,774 --> 00:37:03,554
And like this feels way more appropriate than making people play a little mini game that
they're not good at.

579
00:37:03,554 --> 00:37:05,375
Part of it's also just finding the malice, right?

580
00:37:05,375 --> 00:37:09,159
Like maybe not all automation deserves to be stopped and now we put a credit wall in.

581
00:37:09,159 --> 00:37:12,091
And agents can actually submit the credit wall and keep going if they want.

582
00:37:12,091 --> 00:37:13,326
I'm I'm happy with that.

583
00:37:13,326 --> 00:37:14,206
That's a good point, right?

584
00:37:14,206 --> 00:37:20,198
The block is in the wrong place because fundamentally you want to block a particular kind
of action and not some step leading up to it.

585
00:37:20,198 --> 00:37:21,388
It's the wrong solution, right?

586
00:37:21,388 --> 00:37:27,140
Oh, maybe if we put an obstacle in this completely unrelated other part, you know, the
malicious activity won't actually happen.

587
00:37:27,140 --> 00:37:31,171
It's like, well, it happens in this other part, so it's they're totally unrelated to each
other.

588
00:37:31,171 --> 00:37:32,491
You know, people will find stuff.

589
00:37:32,491 --> 00:37:39,894
And then you end up with a story where, well, now they solve that capture and you're like
your whole strategy for blocking malicious activity is totally gone overnight.

590
00:37:39,894 --> 00:37:41,506
But if you focus on it differently,

591
00:37:41,506 --> 00:37:42,316
Then you can solve it.

592
00:37:42,316 --> 00:37:43,677
Automation is not the issue.

593
00:37:43,677 --> 00:37:52,669
You should understand when users want to use your platform, whether it's through
automation or by manual intervention with a with a screen and a UI or a browser and do the

594
00:37:52,669 --> 00:37:53,390
right thing.

595
00:37:53,390 --> 00:38:02,812
And one of the things I see a lot from startups is that they don't understand why users
come in, they take the free trial, then they drop off and make a new account, a new free

596
00:38:02,812 --> 00:38:05,913
trial, and go through this cycle to get value added with their product.

597
00:38:05,913 --> 00:38:08,174
And I think this happens when the

598
00:38:08,174 --> 00:38:11,437
thing you're charging for isn't well aligned with the value you're getting out.

599
00:38:11,437 --> 00:38:11,597
Right.

600
00:38:11,597 --> 00:38:17,152
If you give someone a free trial for the value they get out, then they're going to keep
getting free trials to get the value out.

601
00:38:17,152 --> 00:38:19,625
There's no reason for them to make a long lived account.

602
00:38:19,625 --> 00:38:27,032
If you want them to do the right thing, then you need to organize or orchestrate the
product or website that's all that's aligned with the thing that you want to happen.

603
00:38:27,032 --> 00:38:29,864
So don't allow the malicious activity ever.

604
00:38:29,864 --> 00:38:34,342
Like it's not like, sometimes a malicious activity could be, you know, for a justifiable
reason.

605
00:38:34,342 --> 00:38:36,626
we're actually seeing that a lot with the product pricing as well.

606
00:38:36,626 --> 00:38:40,002
Pricing where like all SaaS products before were priced are like per seat.

607
00:38:40,002 --> 00:38:43,519
And seat pricing, you could like I guess proxy that to value from a product.

608
00:38:43,519 --> 00:38:45,602
You're like X peop people are using the product, they're

609
00:38:45,602 --> 00:38:47,736
Well maybe and that's and that's the point.

610
00:38:47,736 --> 00:38:48,968
For us, we don't care about seats.

611
00:38:48,968 --> 00:38:51,151
Like we we you can have one seat or a thousand seats.

612
00:38:51,151 --> 00:38:52,092
don't really care.

613
00:38:52,092 --> 00:38:54,095
Like for us it's like, okay, how much are you using the product?

614
00:38:54,095 --> 00:38:56,979
Like how much how much automation is actually happening, right?

615
00:38:56,979 --> 00:38:59,128
And I'm finding a lot of AI tools are kinda going down that path.

616
00:38:59,128 --> 00:39:01,940
There's a whole conversation here, which which I I I find is very interesting.

617
00:39:01,940 --> 00:39:09,167
I don't know if we have time for it, but what I'll say is that the closer to
infrastructure you are, the more the usage based pricing makes sense and it assigns the

618
00:39:09,167 --> 00:39:11,549
usage based off of the value that you're offering.

619
00:39:11,549 --> 00:39:14,552
It's like one X is the value you're getting out of it.

620
00:39:14,552 --> 00:39:18,675
And so you should be charging by one of those things or N of them or whatever.

621
00:39:18,675 --> 00:39:22,196
The usage-based pricing being converted to seat-based pricing.

622
00:39:22,196 --> 00:39:29,621
Is sort of like this legacy financial accounting thing that a lot of companies do, which
is like, well, we want to be able to predict our usage and we only know the number of, you

623
00:39:29,621 --> 00:39:30,622
know, engineers we have.

624
00:39:30,622 --> 00:39:38,126
And so being able to multiply the per seat, you know, times engineers or non-technical
people, whatever your tool is, it makes it easy for them.

625
00:39:38,126 --> 00:39:40,728
But you have to realize that there's a lot of waste in in there.

626
00:39:40,728 --> 00:39:46,246
And so there is this challenge of like a a pricing guide that's easy to understand and
also the

627
00:39:46,264 --> 00:39:48,245
like straightforward value association.

628
00:39:48,245 --> 00:39:49,747
Early on, we made the same thing.

629
00:39:49,747 --> 00:39:56,122
Like a lot of our competitors were charging like the number of different monthly active
users they had for for logging into a portal.

630
00:39:56,122 --> 00:39:57,413
And we're like, well, that doesn't make sense.

631
00:39:57,413 --> 00:40:00,805
Some users log in a hundred thousand times, other users log in once.

632
00:40:00,805 --> 00:40:05,269
And also our customers don't care about the number of users logging in.

633
00:40:05,269 --> 00:40:10,673
They care about the number of of these events that they have to do, the the complexity of
this or the amount of data.

634
00:40:10,673 --> 00:40:11,854
And that that made more sense.

635
00:40:11,854 --> 00:40:12,546
So

636
00:40:12,546 --> 00:40:17,892
That's a different story, though, than what a lot of companies are doing now, which is
they don't understand what their users care about.

637
00:40:17,892 --> 00:40:22,336
And so they're just like, well, our infrastructure costs N dollars per token.

638
00:40:22,336 --> 00:40:27,441
So we're also going to just charge like N plus one dollars per per this thing that sort of
maps the tokens.

639
00:40:27,441 --> 00:40:31,485
And so we aren't going to, you know, have a uh like a finance problem.

640
00:40:31,485 --> 00:40:34,708
We're not going to run out of money because our revenue is too low.

641
00:40:34,708 --> 00:40:35,389
But that's different.

642
00:40:35,389 --> 00:40:36,770
You know, what is the value you're getting out?

643
00:40:36,770 --> 00:40:39,894
We get that question a lot when we sell our products, like why can't I just pay for the
tokens?

644
00:40:39,894 --> 00:40:43,498
Their desire to capitalize on the cost savings that are coming in the future.

645
00:40:43,498 --> 00:40:45,601
Like everybody knows AI is expensive today.

646
00:40:45,601 --> 00:40:53,310
What they kind of are implicitly saying when they want to pay like token pricing or like
some some abstract pricing away is they wanna realize the benefit of like churning from

647
00:40:53,310 --> 00:40:54,452
Claude to beep seal.

648
00:40:54,452 --> 00:40:56,610
Like they wanna they wanna feel the benefits of that.

649
00:40:56,610 --> 00:41:04,984
Basically what you're saying is that they know that the infrastructure you're running on
is going to be cheaper over time and they want they they want reduced c price, basically,

650
00:41:04,984 --> 00:41:06,544
because they believe that to be true.

651
00:41:06,544 --> 00:41:10,456
But this whole mindset is like obviously wrong in a way.

652
00:41:10,456 --> 00:41:12,817
I mean, because you're not you're not paying for the infrastructure anyway, right?

653
00:41:12,817 --> 00:41:14,678
You're paying for the value you're getting out of the product.

654
00:41:14,678 --> 00:41:15,394
So

655
00:41:15,394 --> 00:41:17,806
You know, believing that you'll have a reduction there just doesn't make any sense.

656
00:41:17,806 --> 00:41:25,563
It's like going to any sort of SaaS provider and being like, I want a contract that
reduces the cost by ten percent per year over year because you're using AWS and AWS also

657
00:41:25,563 --> 00:41:27,244
reduces their their costs over time.

658
00:41:27,244 --> 00:41:30,667
So you should you should, you know, pass along that cost savings to us.

659
00:41:30,667 --> 00:41:34,771
And my answer is like, Yes, of course we will if it happens.

660
00:41:34,771 --> 00:41:39,905
Like our we will charge you less if we can charge you less, uh, if that makes sense.

661
00:41:39,905 --> 00:41:43,518
But also we're not charging you based off of the infrastructure that we're providing.

662
00:41:43,518 --> 00:41:43,832
We're

663
00:41:43,832 --> 00:41:48,046
charging you based off of, you know, what makes sense for this value and what we can
compete at.

664
00:41:48,046 --> 00:41:48,276
Yeah.

665
00:41:48,276 --> 00:41:48,687
Yeah.

666
00:41:48,687 --> 00:41:52,211
Well, I mean the mistake is believing that you're charging based off what your costs are.

667
00:41:52,211 --> 00:41:55,794
And most companies are not doing that and it wouldn't make sense to do

668
00:41:55,794 --> 00:41:59,588
Signals a potential feature where like software actually becomes a commodity, right?

669
00:41:59,588 --> 00:42:01,179
That's actually what it kinda implies.

670
00:42:01,179 --> 00:42:05,683
And in commodity markets the margins just getting pissed away to zero and everything
becomes cheaper everybody.

671
00:42:05,683 --> 00:42:08,235
And on a big scale, like people are like, is there a b A I bubble?

672
00:42:08,235 --> 00:42:13,038
you know, it it's unclear whether some companies will be worth a lot of money or all if
all companies will just shrink in value.

673
00:42:13,038 --> 00:42:20,722
If you are competing on price with your competitors, well, in a perfect capitalist
society, the marginal value from anything goes to zero.

674
00:42:20,722 --> 00:42:27,816
Because if the marginal value is greater than zero, someone else will come up with a
product and charge a smaller price for that same marginal value until it goes to zero.

675
00:42:27,816 --> 00:42:37,182
So if the price of generating software goes to zero and all software is like plastic, a
waste product of something else and it's disposable, then that also goes to zero.

676
00:42:37,182 --> 00:42:41,678
And then we basically say there's all all value of virtual products essentially goes to
zero.

677
00:42:41,678 --> 00:42:42,720
Outside of monopolies, right?

678
00:42:42,720 --> 00:42:45,363
Like all products will trend towards like their cost basis.

679
00:42:45,944 --> 00:42:47,106
So it might actually happen, right?

680
00:42:47,106 --> 00:42:50,971
Like maybe maybe the difficulty of creating software is what was present preventing that
from happening.

681
00:42:50,971 --> 00:42:56,300
And for a lot of businesses, the cost of labor slash software like what keeps the price
high, which who knows.

682
00:42:56,300 --> 00:42:59,932
With any software, there has to like you have to put something else in it, right?

683
00:42:59,932 --> 00:43:01,202
And this is true for anything, right?

684
00:43:01,202 --> 00:43:07,715
You have the raw material, and then on top of that, you have whatever refinements you make
and you're paying for the cost of labor there.

685
00:43:07,715 --> 00:43:10,036
So cost of goods plus cost cost of labor.

686
00:43:10,036 --> 00:43:14,799
Unless you're you're instilling some value into that product that you're making.

687
00:43:14,799 --> 00:43:16,629
And the question is, well, will that change?

688
00:43:16,629 --> 00:43:22,178
But I think the finer details of how capitalism works may be a topic for another episode.

689
00:43:22,178 --> 00:43:23,345
Yeah for the

690
00:43:23,892 --> 00:43:26,254
Z so I I mean that's not this podcast.

691
00:43:26,254 --> 00:43:28,066
I can recommend a few financial ones.

692
00:43:28,066 --> 00:43:32,270
none that I like, but for sure an option out there.

693
00:43:32,270 --> 00:43:34,562
Um at this point I think we should switch over to picks.

694
00:43:34,562 --> 00:43:37,164
So maybe I can ask, what have you brought for the audience today?

695
00:43:37,164 --> 00:43:39,956
My pick is a little video game called RuneScape.

696
00:43:39,956 --> 00:43:46,262
And my little plug for it is um I was really excited when the Jev like decision models
were created.

697
00:43:46,262 --> 00:43:50,954
The way I tested it out to see if they were good or not was I got it to play RuneScape.

698
00:43:50,954 --> 00:43:51,360
Okay.

699
00:43:51,360 --> 00:43:59,492
And actually like I just like set up this library that would go in like go and play the
game as if it was a player and see how far it could get.

700
00:43:59,492 --> 00:44:01,434
And the further I can get, the smarter the model.

701
00:44:01,434 --> 00:44:03,896
And that was kind of my benchmark that I created for it, which is really fun.

702
00:44:03,896 --> 00:44:06,776
But uh I grew up playing RuneScape and I learned how to program.

703
00:44:06,776 --> 00:44:09,148
playing RuneScape I should create bots for it back in the old days.

704
00:44:09,148 --> 00:44:10,807
That was always my number one pick.

705
00:44:10,807 --> 00:44:11,362
Yeah.

706
00:44:11,362 --> 00:44:15,014
Maybe this reveals too much about myself, but there was a time that I played a lot of
World of Warcraft.

707
00:44:15,014 --> 00:44:22,139
And I had just found out that one of the reasons I had quit is a lot of the real life
people that I knew who had played and stopped playing.

708
00:44:22,139 --> 00:44:25,150
And I felt like I didn't want to play with a bunch of NPCs.

709
00:44:25,411 --> 00:44:34,797
And what I had found out later is that the guild leader actually had transitioned to still
quote unquote playing, but their version of playing the game was basically on

710
00:44:34,797 --> 00:44:35,307
spreadsheets.

711
00:44:35,307 --> 00:44:37,046
They manage raids and

712
00:44:37,046 --> 00:44:41,052
and dungeon encounters and whatnot and the people, but never logs into the game ever.

713
00:44:41,052 --> 00:44:43,457
Like they don't actually play physically the game.

714
00:44:43,457 --> 00:44:44,999
And at that point I'm like, what is the game?

715
00:44:44,999 --> 00:44:49,927
Like what what are you you watching it watching a video of of an L L doing stuff?

716
00:44:49,927 --> 00:44:51,429
Is that is that enjoyable?

717
00:44:51,429 --> 00:44:52,110
I don't know.

718
00:44:52,110 --> 00:44:54,141
I mean, do you watch y YouTube videos of World of Warcraft?

719
00:44:54,141 --> 00:44:55,422
'Cause that's how I keep up with Runescape.

720
00:44:55,422 --> 00:44:57,290
I just watch other people play.

721
00:44:57,290 --> 00:45:02,494
So I don't have to invest the hundreds of hours they do, but I still get the dopamine hits
of hitting the levels, you know?

722
00:45:02,494 --> 00:45:02,974
It's

723
00:45:02,974 --> 00:45:06,105
I I was never one of those people that watch other people play things.

724
00:45:06,105 --> 00:45:06,955
That's that's not my thing.

725
00:45:06,955 --> 00:45:10,297
I I mean unless there's like some novel thing about what it's doing.

726
00:45:10,297 --> 00:45:11,898
So maybe it's a speed run.

727
00:45:11,898 --> 00:45:18,710
Like I'm I it's a game I used to play and I'm interested in how did people speed run it,
but there's always sort of something that I'm interested in learning.

728
00:45:18,710 --> 00:45:22,956
I usually don't watch YouTube's for like pure entertainment value unless it's like a
comedian.

729
00:45:22,956 --> 00:45:25,441
Runescape might be a little bit special in that in that regard.

730
00:45:25,441 --> 00:45:28,837
a lot of their popular YouTubers get more views than like the number of players they have.

731
00:45:28,837 --> 00:45:33,646
It just like s like there's this like huge audience of people that used to play that still
are into the game but don't don't have time to play anymore.

732
00:45:33,646 --> 00:45:35,820
They just like get their hit by watching the YouTube video.

733
00:45:35,820 --> 00:45:36,691
It's nostalgia, right?

734
00:45:36,691 --> 00:45:41,728
They're like, I I I want I want to relive how great it was, but I don't want to s you know
spend the effort.

735
00:45:41,728 --> 00:45:43,420
I'll just watch someone do the things, right?

736
00:45:43,420 --> 00:45:45,322
And you don't have your own recordings.

737
00:45:45,423 --> 00:45:46,534
And this is like the next level.

738
00:45:46,534 --> 00:45:51,090
Why would I watch a human do it when I can watch when I can pay an L LM to to play the
game for me?

739
00:45:51,451 --> 00:45:54,028
In those cases, are you running the the LLM locally?

740
00:45:54,028 --> 00:45:57,052
For this test that I ran, I generally don't run local LMs.

741
00:45:57,052 --> 00:45:58,269
I'm into the frontier right now.

742
00:45:58,269 --> 00:45:59,100
Okay.

743
00:45:59,246 --> 00:46:01,698
yeah, so so RuneScape, that's definitely a a callback.

744
00:46:01,698 --> 00:46:07,491
I have I've never played myself, but I know a lot of people that dedicate a lot dedicate
time to playing.

745
00:46:07,491 --> 00:46:12,504
And they just had their yearly convention in it was in UK, not in London though, some
somewhere else.

746
00:46:12,504 --> 00:46:15,326
I think this one was Birmingham for for whatever reason.

747
00:46:15,326 --> 00:46:16,736
Yeah, what did I bring?

748
00:46:16,736 --> 00:46:21,519
All right, my pick is the Martingtop 75 liter backpack.

749
00:46:21,519 --> 00:46:26,678
So this thing is like a total beast, and I actually don't have it in this room, but I
travel

750
00:46:26,678 --> 00:46:33,483
a fair amount, not like I won't say a lot, but having basically a duffel bag with backpack
straps on it, I can highly recommend.

751
00:46:33,483 --> 00:46:40,588
So it's not like one of those normal backpacking backpacks where there's only an opening
on the top and it you have to crawl all the way down to the bottom or use one of these

752
00:46:40,588 --> 00:46:45,021
really uncomfortable side zippers that like everything falls out when you when you use it.

753
00:46:45,021 --> 00:46:47,653
You can like put it on its side and open the whole thing up.

754
00:46:47,653 --> 00:46:49,224
And this has been a game changer for me.

755
00:46:49,224 --> 00:46:53,216
It's just so much better than any of those other things with the extra straps.

756
00:46:53,216 --> 00:46:55,896
I recommend anyone who does any traveling use it.

757
00:46:55,896 --> 00:46:59,149
Funny enough, people are like, I'm going to Europe for two weeks.

758
00:46:59,350 --> 00:47:00,671
what size duffel or travel thing?

759
00:47:00,671 --> 00:47:03,044
And I'm like, Don't don't get a duffel.

760
00:47:03,044 --> 00:47:05,437
Don't get don't get like a four wheeled thing.

761
00:47:05,437 --> 00:47:08,460
You want a backpack for for vacation traveling for sure.

762
00:47:08,460 --> 00:47:11,173
And this thing it was so cheap, I couldn't even believe it.

763
00:47:11,173 --> 00:47:12,655
But it's been it's been great, honestly.

764
00:47:12,655 --> 00:47:14,198
Are you you're a traveler?

765
00:47:14,198 --> 00:47:16,780
I I I am a t traveller, but I I'm a light traveller.

766
00:47:16,780 --> 00:47:17,903
I just bring a backpack.

767
00:47:17,903 --> 00:47:21,128
I'm based in Canada and I go to SF a lot for for my job.

768
00:47:21,128 --> 00:47:24,142
Every time I'm I'm at the airport somebody'll be like, No carry on, huh?

769
00:47:24,142 --> 00:47:26,616
And I'm like, Nope, uh just the backpack.

770
00:47:26,616 --> 00:47:28,406
Just one set of clothes and we're good to go.

771
00:47:28,406 --> 00:47:29,486
It's pretty sad at this point.

772
00:47:29,486 --> 00:47:31,927
I like I if I'm going for like one day, it's one backpack.

773
00:47:31,927 --> 00:47:34,828
If it's three days, it's like or to five days, it's a different one.

774
00:47:34,828 --> 00:47:43,071
And then it's like depending on the length of stay, like I have a dedicated pack just for
that, which I I know sounds terrible, but after you're on a plane, I don't know, twelve,

775
00:47:43,071 --> 00:47:49,033
twenty four times a year, you really start to think, Okay, how do I make this way more
enjoyable in every regard?

776
00:47:49,033 --> 00:47:53,324
And especially 'cause I vacation a lot in Southeast Asia, I feel like you're you're on a

777
00:47:53,324 --> 00:47:59,630
Multiple planes and then uh a car ride and then likely a cruise too till you get to your
final location.

778
00:47:59,630 --> 00:48:04,785
And having to wheel something around on like dirt roads, do not do not recommend.

779
00:48:04,785 --> 00:48:09,490
Well, Sucan, thank you for for coming on and discussing all the browser stuff.

780
00:48:09,490 --> 00:48:18,254
I I know we got a little bit away into the the Ethereum philosophy of uh LLMs and token
usage and product pricing, but I have enjoyed every moment of it.

781
00:48:18,254 --> 00:48:18,688
It was fun.

782
00:48:18,688 --> 00:48:20,157
Thank you for having me on today.

783
00:48:20,398 --> 00:48:21,170
Yeah, absolutely.

784
00:48:21,170 --> 00:48:26,116
And thanks for all the viewers for tuning into this episode and hopefully we'll see you
all back again next

