speaker-0 (00:07.992) Welcome back to Adventures in DevOps. We've been talking a lot about browsers lately, reliability, testing, browser automation. And so to fit with that theme, I figured we'd try taking one step further and getting a feeling about how companies decide, good or bad, to hook an LLM directly into their browser for some sort of automation. We brought on Tukaten Singh, founder and CEO at Skyvern, previously ML and Search. Welcome to the show. speaker-1 (00:33.134) Good to here, Warren. Thanks for the thanks for having me on. speaker-0 (00:35.394) So maybe let's just get to it. two weeks ago on this podcast, we talked a lot about Web MCP and providing better capabilities to make it easier to potentially use LLMs to interact with web pages and not just the browser directly. Why do this at all? Like, why not design, say, technical processes to interact directly with like we've had APIs forever, right? So if you have some sort of business process that you're running in your company, why not try to go to your you know, third party vendors or or providers or the source technology and try to interact with the APIs that are available there. speaker-1 (01:08.184) There's a couple really interesting reasons why that that's not likely to happen. How many websites exist in the world? It's about 1.1 billion. But there's only about 900,000 APIs that exist. You know, 1.01 billion shortfall between like number of websites to APIs. And and if you look at why that's the case, all the w websites that like benefit by making the customers' lives easier have APIs. So think like technical products, think like, you know, e commerce even to some degree has APIs. Basically, with a profit center for a company, they'll have an API. And if it's cost center, Like think of any government website or your cell phone bill. You gotta log into like Comcast to fetch it. Like those guys just have the portals because legalities comply them force them to have it, or there's some regulation that requires open data from the government. Those guys don't have any APIs and they have no incentives to create it. And so it's very unlikely that cohort of companies will adopt something like WebIn MCP, where it requires the website owner to make a change. And as long as that's true, it's easier for agents to just behave like humans and access the internet the same way that people do. speaker-0 (02:06.754) first thought that comes to my mind is like a list of all of the restaurants that you could potentially go to and you maybe want to make a reservation or even just see what the menus are. And that's a that's a challenge from a consumer standpoint because obviously the tools available are limited there. And I think it was Google, a long time ago, was suggesting, they can automate the reservation process for us by by doing something by sidestepping this and inserting an agent there. I think it was a duplex, I think is what they had originally called it. But there's still this human element and there's a lot more than just calling a phone number. And those companies that are running a restaurant, basically, they they may not get any benefit from having an API. Who's gonna order food from them programmatically in a way, or maybe reservations? but then I I do see the flip side though. Whereas like Domino's got a lot of publicity early on, I think it was like almost 30 years ago, by having a like a order dominoes command inside of EverQuest. And you do see restaurant reservation platforms pop up all the time or ordering platforms that want to interact with restaurants to be able to order stuff. And there was a big one that just got a lot of slack recently because they were AI generating the food images for their platform where no images existed. And so like I think about all these things and I and I still see like I'm still praying for this better world where it's cheaper and more reliable and you get server-side form validation. But on the flip side, you're right, the incentive may not be there, right? You know, maybe restaurants say, we only care about this local area. We already are producing as much food as we can at a particular price point. We don't need more customers. Being accessible on a particular platform or doing reservations doesn't help us. How do you make the decision on whether or not like do you even consider like whether it's a good thing? to automate some particular accessing or of a website or an action before going and and taking it or is just like, okay, customers have a need, they have a business process and we have to replicate that as best as possible. speaker-1 (04:00.14) do definitely do care about the ethical side of the question. Just to preface that answer, like we are an open source company. You can use that product, you can like clone it, run it. But with our cloud platform we actually get a bunch of antibot behavior as well. And we did we chose actually actively not to open source that. very intentionally because we got a lot of requests to do harmful things on the internet. Somebody came into our Discord one day and they're like how can I use Skyvern to create ten thousand Reddit accounts to promote my thing. Like obviously like like we're like, okay, don't do that. Like, you know, go go maybe go to our competitors or something and do that, you know. Not our but but like, you know, social media spamming, like it's there's a lot of incentive and value to be created for people by doing that, but we don't wanna be the the facilitators of that kind of bad behavior, right? So there's definitely like an open question on the ethics side of it, like like what kind of behavior do we want to support? If somebody comes in, they're like, Hey, I want I wanna like fetch my telecom bill. every month and like the telecom website is like cloud player protected. Like is that unethical? And it's like definitely not. Right? That's in the realm of like it's okay to go do that. We do try to screen for that. Like just last week somebody like tried to like create like ten thousand anthropic accounts through our product. We already banned that kind of behavior, but like but to like remind the users to not do do nefarious things like that. Could you share speaker-0 (05:10.646) a little bit about the primary use case for Skyburn and how customers end up using it and then once we have that contact maybe we'll we can dive into some of the more interesting scenarios. speaker-1 (05:20.428) Yeah, so Skyvern is a like a vibe coding platform for non technical people to create their own browser automations. j we work with a lot of enterprises that use our product to automatically generate insurance quotes so they can provide the lowest like surface the lowest cost thing for their customers. they use us to fetch invoices for like regional locations, each having their own telecom portal. they'll use us for things like healthcare automations, like every time you hire a doctor, you have to like prove they are allowed to practice in the area they're allowed practice. By the regulation, so they need to like go to a bunch of websites to collect that data and to make sure they get paid by the insurance carriers and so on. so we help with a lot of like back office use cases like that. usually it's like companies that'll like start and start scaling by having like people do that work, where it's like literally just like go to the same website and fill out the same like five fields and grab a screenshot and save it over and over again. And then they want to scale their business, they're sort of looking for automation to do that. speaker-0 (06:10.616) Do you find that a lot of the scenarios where there may be a customer doing or wanting to do something that you would maybe flavor as unethical, are they r real customers? Are are they people that are doing something within their business domain? Or is it like a completely separate group of users that are potentially doing something illegal or unethical specifically and you don't really you didn't really design the product for them, but they they feel like they have an opportunity because it's open source to somehow utilize it. speaker-1 (06:38.926) We d we didn't want to limit like our UX for certain good or bad behaviors. However, it's pretty clear that some behavior is not good for the world, even though like people individually might get value out of it. From the open source side, we don't really restrict it, but we use playwright under the hood in in our open source to run things. And if you run stuff with Playwright, basically broadcast the internet. It's like I'm a bot, I'm a bot, I'm a bot. Like, you know, stop like just FYI, like I'm a bot running the stuff. and so that kind of takes care of itself. On the cloud side, we actually do have some some better stealth, if that makes sense. speaker-0 (07:07.82) Maybe an ethical question, but if customer is using your platform and doing something that will get them banned or by association or proxy their account through Skyvern, then it's it's a real risk for you to even execute those those actions in that way and have that behavior associated with your your cloud accounts. Like if you're running on prem, then your IP addresses could be would be at risk potentially. And if you're running in a cloud, then your cloud accounts. speaker-1 (07:32.782) Those are actually surprisingly easy because w if you run stuff from like AWS, like AWS has their own data center IP range. So if you run automation there, like every website can be like this coming from an AWS data center. speaker-0 (07:43.81) Do you find that blocking that sort of behavior is because you consider it unethical to do that in the first place? Or there is some real ramification if you sort of allowed it? And on the flip side, I'd be like, well, if anthropic or Reddit had a concern with allowing that, wouldn't they already need to have a mechanism in place to block that sort of behavior in the first place? And so it wouldn't really be your responsibility or your company's responsibility or the or the product's responsibility to take an extra step and do something about it. speaker-1 (08:10.158) I think the legalities there are quite ambig ambiguous. There's not not a lot of case law that goes the one way or the other because our customers direct the AI to do what it needs to do. And so they're f they actually take liability because they direct it. But it is still our job to make sure that they are behaving well. To our ethics. I want to feel like I'm doing good in the world. Like I didn't start a company to be evil, but I do take some pride in blocking the bad stuff. speaker-0 (08:30.414) Do you find that there's like direct and expected like categories of particular unethical or illegal behavior that you're often dealing with and you you're just like, okay, it's like one of three these three buckets and we're good? Or is every sort of scenario a potential like, well, we don't know upfront in how to categorize this stuff. Everything seems like a little snowflake and we have to evaluate them like on a case by case basis. speaker-1 (08:53.944) So I think there's some cases where it's obvious. Like the the three examples I gave. LinkedIn is maybe okay, but we block it just because of the potential of abuse there. Reddit, I I haven't seen any good use case of people wanting to go to Reddit. That that that that isn't already it doesn't already have a publicly sanctioned way of doing it. Yeah. Like if you wanna know what people are talking about, you they have RSS feeds you can subscribe to, you don't need to go through use our product. Right. s same with like all the social media websites. Like they have they have business APIs you can use to actually access them legitimately, so there's no reason for them to go through our product. and then there's a set of ambiguous use cases where it's like somebody wants to apply to a job. Mm-hmm. Like, is it bad? Is it good? You know, it's kind of ambiguous. Another one is like they want to go to Amazon, but they want to like manage their transactions for their business. Like Amazon doesn't have great APIs for that and like is that bad? Or is that good? Like, you know speaker-0 (09:44.344) Just saying like going to Amazon isn't necessarily a problem. But then like what are they doing when they get there? Are they trying to go to every single web page and download all the product details and whatnot from their competitors to do you know dynamic pricing for their own items? Or is it about actually just trying to automate the buying of particular items or the management of a shopping cart or you know history of supplies? whatever you got vendored out so that you can manage your resource planning, you know, in your own ERP system and just the buy through Amazon specifically without having to deploy your own Amazon partner web app that interacts with the APIs directly, because that may not even exist to support you and your needs. I see like this weird duality where it seems like a lot of the use cases which could help a a business automate particular internal processes or change the user experience that the user is getting. So instead of having to go through a bunch of different platforms or different websites to list out all of the, I don't know, products that they need to purchase. Like if you're maybe if you're building a data center and you want some sort of Ethernet cables and and the jacks and and wall outlets and whatever. And obviously not probably buying this stuff from Amazon, but even if you were, you may have that in a spreadsheet somewhere. And then you have to go and you know go and actually buy each of those things and jumping over that gap is is useful. I I I want to say that it does seem like the business use cases for process automation make a lot of sense. And feel more ethical than the ones that are on the end consumer side. Like I I feel like the consumer things are all like, I don't know. That's the gray arrow. speaker-1 (11:10.86) Well, w the other the other thing when you deal with businesses is that y often they'll you'll have to sign a contract with them and in the contract you can just say that you're not allowed to do anything illegal. Right. And and like like a an actual business has a lot of incentive to follow the contract. Like obviously like a startup like a very small startup or like a one person shop, those those companies they can just go declare bankruptcy and move on. But like an established business with like, you know, fifty employees, the probability they want to survive for the next ten years is extremely high. So when you put something like that in the contract, you can just generally trust that they're gonna do reasonable things. speaker-0 (11:40.59) Do you have like a complex like KYC or know your customer process setup? speaker-1 (11:44.792) Yeah, we don't we don't enable our anti bot stuff unless we have a contract in place with our customers. a and at which point we would we would put in the in the MSA the like the master service agreement that they're only allowed to use us for legally sanctioned stuff. speaker-0 (11:55.744) In order for a customer to come on the platform and and really get the value out of it, it will always be the case that you'll have to know who they are and what business that they're trying to automate. Or do you see in the future this shifting closer to the self service onboarding side? speaker-1 (12:07.212) I have not yet seen where somebody wants to spend more than five hundred bucks a month with their product and they don't want a contract in place. Will that happen in the future? Either the company wants discounts or they need some like protections in place like healthcare companies like need like HIPAA compliance and so on. That naturally forces to go down this path. speaker-0 (12:22.606) So as I mentioned, there's like WebMCP that has come out, but you obviously rightly identified that this actually requires platforms to actually implement the mechanisms and basically write a whole new API just for interacting through the website. And if they're going to do that anyway and they're technical, why are they just not offering an MCP server or offering a real API to interact with? speaker-1 (12:42.26) I think it like a fifth customer, was this company in Ireland and they wanted to go automate some like tack data retrieval from like the I Irish government and their like VP of operations emailed me. It's like, Why would anybody use your product? This tax authority has APIs we can use. That's great. Like, that's great. If there's an API use it, don't our product. Great. Can the API do this operation? And he's like, no, it d it that has a gap. Can't do it And like, what about this other one? He's like, no, can't do that one either and I'm like You see the problem? Like even if they have an API or an MCP, what if the tool call is not available? We use this product called Hex for data analytics and the MCP doesn't let you schedule reports. Like like you're just kinda screwed. You know, like if all my agents just schedule reports, I gotta go through the browser now, there's no other option. And it's just like a tech company. You it's like not a website with zero incentive to actually create a good MCP. Even if there isn't a API, it might not have the coverage it the website needs to actually be useful for people to the fullest degree. Or speaker-0 (13:30.67) Worse, they think about it and then they decide that they specifically don't want to solve the problem for that particular customer set. So a few weeks back, we had an architect from the Wikimedia Foundation come on where you know they're powering Wikipedia, they're they're writing the product open source and everything. And you get a lot of LLM usage, a lot of customers coming in and trying to answer questions whose answers are on the wiki somewhere. And it's a lot of use. The thought is like, well, you could be one of those companies that's like, we're not going to offer an API for These users. And I just feel like, but you're losing so much value. You're like, you're basically saying we don't want users just because of the tool they're using. It's like there used to be a thing where, they're using the wrong browser, so they can't access our website, or you know, that you need to have flash installed. And at some point you realize, wait, these are real users. we we should do something for them. And it's really surprising when tech companies don't have the APIs available to actually even do the integration. And I think over time, it has to happen that we that every single company is offering. Endpoints that solve whatever their users want. Now, if you don't have any users that use LLMs, interact with your your APIs, then whatever. You don't need to automate in that way. The mind-boggling part for me is as a company, I want to power that. So we offer like a very technical product. Our UI is like administrative and it's not like a pristine users use it eight hours a day, every single day as part of their job. They almost never have to go there. If they're like, we want to build our own administrative portal that uses your APIs, I'm like, Yes, please go do that. You know, because then you can get the value out that you exactly need and not have to worry about whatever new features we're putting in or having the exact graphs or data that's available. And just seems like a no-brainer. And yet you see lots of companies that don't offer a way to do this out of the box. speaker-1 (15:10.85) And it's not hard anymore. Like like if we were having this conversation a year ago, we'd be like, okay, we've got to like allocate like an engineer for two weeks. And it's like, okay, what's the opportunity cost for that? Like, what are we gonna give up on the roadmap to add this feature that the small percentage people need? But now it's like now I can just be like, hey, like go go go build an MCP for this. Like, you know, like I can literally just like automate that and comp company still don't do it. There's some PM in the company that's like deciding the roadmap with opportunity costs associated with like each each line item, right? And it's just not important to them. For government websites, they might not even have a PM, right? They might just not like put this on like a zombie, zombie budget, right? Where there's like zero like zero like maintenance. Like we had a customer that's so funny. they run an automation with us that goes to like this government portal to like look at the inventory accounts or something. And their IT team reached out to them and they're like, Can you stop running this automation? 'Cause it's running at the same time every day and it's crashing our servers. And instead of like making it sc instead of making it scale, they're like, actually, can you just like stagger it so we can stop having our servers crash? when this thing runs and it it was like one run a day. It wasn't anything substantial. It's not even like the quality of engineering. It's really like the attitude that the organization has towards our web offering that that defines this lack of functionality. speaker-0 (16:18.982) It's it's surprising how often we see challenges show up in our own technology or like workarounds that we have to build because our providers didn't build it in the optimal way. I think the number one that always shows up for us is lack of idempotency on APIs. So like y something is gonna crash at some point for sure. And you're like, I have to retry this. And you go to retry it and you're like, Well, wait a second. This is like a dangerous operation that we don't ever want to do twice. And yet this API doesn't support. A way to actually deduplicate it. We had we have we have these all the time. We interact with email, which SMTP doesn't support item potency really. I mean, you can use a message ID sort of, but most email senders like it's so ridiculous to me. Most email senders don't let you specify in an API like what the message ID is that's generated by the provider after they've sent the email. And like that's the number one absurd thing. But even like Slack and Discord don't do this. There's no way to deduplicate sending messages there. Stripe for the longest time didn't support this. Like you want to send a charge and you can sort of send a message to deduplicate it, but they only support like specific IDs and only within five minutes. And they didn't do that always. And I'm I'm just like, I don't want to double charge this. And so you you write some ridiculous extra automation. And like these are companies that that are so fundamentally technical. Like they're offering like first class APIs to do this thing, and they don't even have the necessary functionality. So it makes a lot of sense to me that we're getting to the point where. You don't even trust a company to write the API correctly. If the the UI exists, you know, that becomes the interface that everyone's using. speaker-1 (17:53.26) Like item points is like an advanced topic, right, for engineers. Like like the stuff I'm talking about is like you you wanna you go on a to like register a company and you you're like shit it's one AM. I wanna go to the IRS website and get an EIN number so I can do so I can do payroll. It's it's it's offline from one AM to s s nine AM every day. You can't access the website. You know, like I'm talking like like stuff that, you know, was common I guess in like the two thousands, early two thousands and like late nineties, but still there. That's the kind of websites our customers send out to send to. It's different world. It's like Blows my mind every day. speaker-0 (18:24.558) Do you find that a certain kind of company is focused more on the process automation than than others? speaker-1 (18:30.23) Actually, we solve like a very specific problem in the world. There's this like entire industry called the RPA industry. it stands for robotic process automation. The the category exists to like automate repetitive work within large businesses. And the way the in in industry worked is there's these like people called RPA engineers who are good engineers and they learn how to use these like archaic tools like UiPath. What they do is they like go around the company and they like interview these like business leaders and business team members to figure out what work they do all the time. Write that down and then automate it. The person who actually knows the work to automate is like some non technical person who is doing the actually doing the work every day. They actually know it inside out. They all the edge cases that could happen. if the website changes in a small way, they can like they they exactly what to do to fix it. Where they'd have like they know everything, they have to write it down, forget like half the stuff that they know in their head, hand it off to this guy to go program, which is honestly like not super interesting programming work anyways. When it would be deployed, they'd run into a bunch of edge cases and they'd have to go back and forth. It would take weeks to automate. Like with our tool, this person can now actually just go build their own automation. Like we we we just go after companies are already kind of know the problem that they want to automate and they just want like a shorter feedback cycle. speaker-0 (19:37.72) That's sort of the interesting learning that a lot of companies will face is that it's like the amount that you spend on automation actually increases over time if the cost of the automation decreases, right? So when the cost of the automation decreases, the ROI increases. And so like the amount that you're willing, the number of things that you're willing to automate actually increases over time because you're now getting a positive ROI on on more stuff. And so companies actually end up spending more over time now that automation is cheaper. Which is sort of this like confusing paradox. but I I I think it makes a lot of sense. Programming stuff for the browser environment is just a huge headache from my standpoint. And in the browser episode that where we talked about this, there was like there's a lot of things that just seem like they go wrong all the time, or not built well, just a challenge to get around. Are are you fundamentally trusting Playwright to do most of the management of the integration with the browser? speaker-1 (20:26.586) Remember iframes? Remember Shadow DOM roots? they're not over. Like all this crap. Like I remember when we first open sourced our product, the first PR we got was somebody adding ice i frames before Discovered. They're like, I need this shit to work. You know? and it's like everybody's got these battle scars from like when when when they're building automations, but the Chrome browser. It's like an unintelligible protocol. So playwright just makes it like legible for people to use. speaker-0 (20:53.25) They're being replayed against Chrome primarily, or is the browser selection part of the automation that you're creating? speaker-1 (20:59.622) this is actually a very common feature request we get is to support other browsers. I if I listen to our customers who want to do QA, they care a lot about the different set of browsers. But if I listen to our customers who want to do automation, they don't get they only want Chrome. Chrome or like Microsoft Edge. Like like that's all they care about. Which is also Chrome. Yeah, exactly. speaker-0 (21:11.96) They're both Chromium. So I think I think I'm gonna get some emails about this. I will say I keep trying to like Firefox, and every couple of years, they always do something where I'm I'm like, I'm using Chrome. I mean, I'm I'm not using Chrome, I'm using Brave, but I'll say Chromium for now. Everyone can just plug in whatever version of Chromium they're using. And I'm like, you know what? This is just getting worse. I really wish I could switch to Firefox. And like then I go and I'm like, wait. Firefox last year just had another whatever terrible, you know, and then insert whatever scandal thing. And it just happens over and over again. Like I think the biggest one that just happened this year was a change to their their terms and services about like training data, like training on they they they made a change to that like they're allowed to potentially train on the gestures that you're using or wh whatever your history is. And I'm just like, okay, I guess I'm not switching this year either. And it was really sad because early on I was definitely a Firefox user. speaker-1 (22:04.888) Same. speaker-0 (22:05.346) I ended switching for one of those reasons and I keep wanting to go back because I hear about great things and then I hear about wait, they don't really support pass keys in every scenario, or wait, they don't they don't support this security thing or that thing and it's just it's just over. I do get it from an automation standpoint. The user shouldn't care or know about what where the automation is running. That's they don't they don't care about that. They care more that it's happening and the right thing is happening and a a tool that's dedicated to a single set of browsers. That does the right thing more reliably makes a lot more sense. In an episode where we talked about test automation, we were reviewing the challenges with doing any sort of testing with a browser. And one of the things that came up was there was a company that had done an amazing thing where they allowed you to write prompts to execute QA, you know, automation tests against like your website running in the browser. And I'm just like, why would you do that? Did all the tools exist to do this deterministically? You don't want test. To run through a prompt and they're like, yeah, well, we did it, we did it because we could. Just like the we we've had solutions to this for for so long. it's not like we needed LLMs to start doing unit tests against the browser. And if anything, we with products like Skyburn, we definitely see the market moving more towards forgetting about the browser, forgetting about the website. speaker-1 (23:23.896) Google published like a really interesting paper on this in twenty in twenty twenty one, so about three years before LMs well four two years before LMs got really big. And it was called semantic locators. Basically, can you use natural language to identify which selectors to interact with on a website? No. And and QA was actually the primary topic of that because if somebody updates like the location of that button, should should that test case break? Like unit tests, another spicy topic, like unit tests are mostly garbage. Like you know, like no sorry, sorry for all the numbers, you know, like they're mostly garbage. speaker-0 (23:52.694) Let's say you've got a search bar on your website and you want to test that searching for a particular product in your e-commerce catalog is actually returned and someone can add it to their card. The way you do this is you may write some automation, which first finds the search bar, it then enters some characters in there, then it clicks the search button and then finds the list of product results and clicks on the right one and goes to the rest of the flow. The challenge is like, where is the search bar on the screen? Like, how do you even find that? Now, the strategy has historically has been well, there is an input. element in in HTML where it's probably that, but there's lots of other input elements on the screen. So you can add a class or an ID to that element. And so you can easily find that if you're escraping the DOM. Now the problem is that you may not have done that. the class may have changed. Or if you're using an LLM to sort of generate the test, you're like, yeah, go to the search bar and enter the text. There may it may not say search. Like if you've done any sort of LinkedIn automation, you'll note that like all the classes and all the things that are in the DOM, they don't they don't reference anything that's there. You have no idea what the text is. So you could try to find the word search on a button somewhere and hopefully that that's the the search button, but it may not say search, it may say something else. And the text of that may change over time, especially if you're using any sort of locale specific stuff. The button may not actually say search in any sort of QA environment, it may say something else. And also king off of the text in an element seems like it's it's risky. So when I see these tests, yeah, the the whole goal of automation is obviously the where we want to get to, but it really does require the website or the the development of the website to be implemented in a way that still allows it. And so I I think over time, one direction that we will see that going is more specific elements that pop up, right? Like instead of just a search an input for for search, it will be a search input which has you know, additional expectations around it. And I think browsers are very slow to get there, but I think it is a requirement to make the next step, just like we've added a lot of things for accessibility, like paragraphs and summaries and descriptions that actually have first class meanings and can be used in this way, rather than, yes, find the fourth paragraph object in the third div that's in there and the one that has the span is the one we want to, you know, you know, look at for the actual value because there's no class or ID speaker-1 (26:04.034) I see more things moving towards like real time rendering generation, client side in information being generated generated and so on. speaker-0 (26:10.564) let's let's fight about this. so I I I'm actually doing a talk this month, the challenges of of getting a dashboard spun up for a reasonable amount of money looking at some business metrics. And that that that's not really where I'm going, but part of this was an experiment that that we did using what what is commonly coined as generative UI now, which is basically dynamically generate a UI to render some specific data that you care about. So maybe you have the database of data and you're like, I don't want to. have a static dashboard. I want a dashboard that shows me the most relevant thing. Well, what's most relevant? How do you figure that out? Well, you can look at some data, perform some queries, and then feed it into an LLM and get back a result. But every single step along the way where we were doing this is still so early that we pretty much had to define every possible thing you could possibly imagine. So like the colors that the graphs have to be, just so that you can see it like And it's from an accessibility standpoint or user experience standpoint. Like what is even a graph? Like the difference between when something should be a line graph and when something should be a bar graph or when it should be a table or a pie graph. Like you had to explain to it what those things are. And so like I'm sitting there and I'm going over the whole development that we've done. And it's like we pretty much just explained how to build a dashboard product in natural language so it could be passed to the system prompt of the model so it could generate this stuff. And we also had to take a lot of extra annoying steps. Like we had to find a graphing framework and then we had to proxy all of the elements and components that we would use that to put it in a form that the that we could pass into the system prompt so that the LM would understand like what to actually generate specifically. And then we had it generate SQL queries to fetch the data dynamically, but we had to explain what SQL was, like what type of queries make sense, like what is a what what belongs in a like a like statement and what you know what's valid, what's not valuable. And then we still weren't done. We we're like, well, we need that we need the prompt to actually generate the right graphs. But if you say show me what's actually relevant, it still was doing the wrong thing. And so now we have a bunch of prompts where it's like, okay, this is prompt A, where it has like, I want show me four graphs where the first graph is a line graph and the x-axis is a date and the y-axis is a list of counts, and you know, here are the lines that they should be, and the second graph is and so on and so forth. And so like after we're done, I'm just like, why did we even do this? We pretty much did all of the software development and then converted into national language. speaker-0 (28:24.246) natural language just so an LLM could generate it for us after the fact. That was definitely not the right solution. speaker-1 (28:29.838) Well what what like the question I'll ask is like what model? Because like when we started Skyburn, this is GPT four era of models and people would be like, go apply to this job. And it would fill out the form and click click submit and then maybe there's like a question missing and it'd be like I'm done. I did it successfully. Like and we had to like we had to do write so much code around the model to make sure it was actually gonna do the right thing. but with the latest generation of models, like I I did we did a full redesign of our product a few months ago and We the b basically the redesign was just deleting all the shit we wrote like last year like the previous year 'cause the new models just did it implicitly. Like the new models just like did it. Like they they they they didn't need to be instructed to be that specific. They they took reasonable I mean, they made reasonable assumptions. Like we still have I would say we still have too many instructions in there, but like the the cohort of instructions and post processing like maybe this long and now it's like this long. And a lot of it was deletion. So I wonder I wonder if that's gonna speaker-0 (29:25.932) It it could be coming. like I yeah, I literally so I literally wrote this prompt like a week ago. yeah, so don't get me wrong. It for sure is much better since there has been more documentation on basically the semantic layer for writing queries. So the TLDR there is a way to inform the model like what sort of data you're saving and like what different columns actually mean. And before that it was definitely atrocious, like no question. Good good luck even getting speaker-1 (29:32.588) Okay, maybe yeah, okay. speaker-0 (29:55.478) something that matched the system prompt or understanding that we are trying to generate a dashboard. But I I just at this point with a system prompt and with a with the actual user prompts, we are getting the right things to show up there. It's not what we want. Like it's not intelligent about about what should actually show up or what is important in a particular use case. And that's the part where it would feel good to automate. Like let's say you have an incident and there's some data around the incident. Maybe there's like some null reference exceptions or some customer reports. Or some five hundreds or whatever have you, and you want a bunch of dashboards that are automatically generated based off of the data that you have. I don't think it's I don't think we're at the point where it's going to know what dashboards to show you, like what to even figure out what the incident is. speaker-1 (30:37.238) Yeah, or or worse, like it might show you insufficient information. I find I find outlines do that a lot. Like particularly like if you use the open ad models, like they're they're lazy. And and they'll be like, Here here's three of the ten things you might care about. Let me know if you want more. And I'm like, God, I just need I I'm having it to it right now, give me everything. What are you doing? Yeah. speaker-0 (30:57.312) One of the huge critical failings of the current LLM technology though, what's based up, you know, from the BERT paper that came out of Google, it's basically a transformer architecture. It can't generate new things. It just takes whatever the input is and transforms it. So realistically, if you don't tell it I want 10, you know, graphs, it it's just gonna make up what it thinks is the right number of graphs. If you don't tell it you want a particular thing, it's not gonna know to include that. You can sort of try to get it to prompt in the real stuff, like have an output that contains some. new and novel ideas, but you still had to sort of be like, make a breakthrough. And as far as you know what it's gonna actually generate. And it's just unless you're very specific, I I just find it's never really gonna get to that level. Like obviously the graphs it generates will be maybe it'll be better accessible because we'll have the right tools, or maybe it will include more relevant data. that would be better. But it's like it's always gonna have a problem in some area. speaker-1 (31:48.344) Yeah. But this is the dimension that I'm like really excited about improving 'cause like I I write really short prompts and I I get pissed when they don't follow. I'll be like F you like come on, like you don't be an idiot. Right. But I found myself doing that less with Fable. so I wonder how true that'll be in six months. speaker-0 (32:04.152) will still be at the point where given a prompt and getting a response, I'll still feel the n need to share expletives back to the L L based off of my current speaker-1 (32:13.58) Emotional. Those are legal now. Dorbike changing their terms and servers we can't swear swear at them anymore. So it's speaker-0 (32:19.79) I I can't like if I swear, will they terminate my account? Because because I I was already planning on leaving. I I know a lot of our company has already transitioned off of Claude, but yeah, I mean if they close my account because I do that, then yeah, I'll I'll go I'm going somewhere else anyway. I get really angry. I I definitely do. So stuff not going my way, especially if I'm tell it to, you know, interact with the browser in a particular way and it doesn't do that. but I'm not there yet. I I don't feel like I want to automate my social media stuff. speaker-1 (32:53.644) No, but I I I I set up a personal agent and I swear a lot of mine to the point where like our one of our interns like started forwarding all my like expletives to like this like highlights channel and I like, Okay, I gotta stop stop doing that because like these are being memorialized whenever I'm like, my god, don't be an idiot speaker-0 (33:09.548) Yeah, when when you become the meme you've you know you've gone too far. Week to week, month to month, when new releases come out, there must be some breaking changes. speaker-1 (33:12.812) Yeah, exactly. speaker-1 (33:18.574) Know if you see many captchas as you browse, but I went to my neighbor's house one day and his friend had convinced him to start using some like stretchy VPN for his privacy. He's a plumber, he doesn't really know. And every time he clicked anything, he would see like these like nasty captures where it would be like it it would basically be like, you can figure out which button is flashing, it's like this cadence and other things are happening. And to be honest, those were like easier for machines to solve than people. Yeah. Done. Like some of them are getting really crazy. Like I speaker-0 (33:40.632) For sure. Yeah. No, that that's all we've done. speaker-1 (33:44.994) Like there was one where I saw the other day where it was like basically build a platformer. Like you have to like move these things around to get like the character to go across and like reach the flag at the end. To sell the captains basically like build a Mario level. speaker-0 (33:55.502) Captures are just so wrong. They're just targeted the wrong way. Like, sure, they may have a problem with too many account signups, right? And historically, the way to do that is sort of user identity verification. No, know your customer. Now, obviously, that's challenging. There are very few countries in the world that have a giant program set up to make sure that your online identity matches a you know real real identity. And so as long as you allow virtual people, you know, virtual accounts set up, you're gonna end up with this problem somewhere. You're absolutely right. As the agility, you know, goes forward, the complexity of the captures will continue to change. And the only time that they were nice for humans was never. Like they've always been a challenge for us to solve. This is like proof of work for humans, right? speaker-1 (34:37.217) PayPal invented captchry, but they're the ones that invented it to fight like Russian mob rigs that like scamming money off their thing. PayPal like had this like huge fraud problem and they were trying to find ways to figure out like which users are are automated, which ones are not. because they kept getting fraudulent accounts that like siphoning money from their users. They invented captchas to solve that problem. It was the captchas, you the one you where it's like slightly tinted words. back then, like the none of the machine learning algorithms could actually solve it and basically felt their fraud problem overnight. And so if you want to thank a company for captives or or or blame them. You know, that's that's kind of the root of that. So before nineteen ninety nine, you know, you could rouse the web, no problem. Like the good old days, as they say. speaker-0 (35:13.26) I find that this whole sort of thing is just a a good approximation of how bad we are at long term strategy as humans, because realistically we know that on one hand, we believe that LLMs will handle more things for us. And on the other hand, we're like, but we can have captures to stop them. I'm like, well, yes, but then we'll have the captches solved by the LLMs. And then they're only getting to a point where the problem is so challenging to solve, as you pointed out, you have to create a perfectly working platformer game and then you know also run that in order to prove that you are a human. And I think we're gonna very quickly get to the philosophical problem, which is like, what is a human? speaker-1 (35:48.29) I think the line is deturing test is I think we're well past that point now. speaker-0 (35:51.692) The captchas are a Turing test, right? Yeah. This is how bad we are. Companies believe that the appropriate Turing test to have is a captcha. And clearly it's not working, right? I will say we found a much better strategy. So we have login and access control as a product. And there are some interactions that we want to basically prevent malicious attackers from utilizing. The one thing that a lot of people forget is that attackers don't have an infinite source of resources and money. So they will only spend so much on doing something. And so for us, we just have a proof of work. hidden from the user running in the browser on every single dangerous interaction. So if it is a malicious automation that's trying to either do credential stuffing to guess accounts or email addresses, they can do that all they want, but they're gonna have to spend a lot of money to make this happen. And it it increases over time. Like if you want to do two requests or three requests, you have to spend more and more users never find out about this. They go once, they enter their username and hopefully they have a passkey or unfortunately a password. And if they They get it right, then they're fine. And if they get it wrong, they're well, then they suffer a little penalty. But with automation, they just hit the wall, you know, almost instantaneously. And then all the attackers drop off there. And like this feels way more appropriate than making people play a little mini game that they're not good at. speaker-1 (37:03.554) Part of it's also just finding the malice, right? Like maybe not all automation deserves to be stopped and now we put a credit wall in. And agents can actually submit the credit wall and keep going if they want. I'm I'm happy with that. speaker-0 (37:13.326) That's a good point, right? The block is in the wrong place because fundamentally you want to block a particular kind of action and not some step leading up to it. It's the wrong solution, right? maybe if we put an obstacle in this completely unrelated other part, you know, the malicious activity won't actually happen. It's like, well, it happens in this other part, so it's they're totally unrelated to each other. You know, people will find stuff. And then you end up with a story where, well, now they solve that capture and you're like your whole strategy for blocking malicious activity is totally gone overnight. But if you focus on it differently, Then you can solve it. Automation is not the issue. You should understand when users want to use your platform, whether it's through automation or by manual intervention with a with a screen and a UI or a browser and do the right thing. And one of the things I see a lot from startups is that they don't understand why users come in, they take the free trial, then they drop off and make a new account, a new free trial, and go through this cycle to get value added with their product. And I think this happens when the thing you're charging for isn't well aligned with the value you're getting out. Right. If you give someone a free trial for the value they get out, then they're going to keep getting free trials to get the value out. There's no reason for them to make a long lived account. If you want them to do the right thing, then you need to organize or orchestrate the product or website that's all that's aligned with the thing that you want to happen. So don't allow the malicious activity ever. Like it's not like, sometimes a malicious activity could be, you know, for a justifiable reason. speaker-1 (38:34.342) we're actually seeing that a lot with the product pricing as well. Pricing where like all SaaS products before were priced are like per seat. And seat pricing, you could like I guess proxy that to value from a product. You're like X peop people are using the product, they're speaker-0 (38:45.602) Well maybe and that's and that's the point. speaker-1 (38:47.736) For us, we don't care about seats. Like we we you can have one seat or a thousand seats. don't really care. Like for us it's like, okay, how much are you using the product? Like how much how much automation is actually happening, right? And I'm finding a lot of AI tools are kinda going down that path. speaker-0 (38:59.128) There's a whole conversation here, which which I I I find is very interesting. I don't know if we have time for it, but what I'll say is that the closer to infrastructure you are, the more the usage based pricing makes sense and it assigns the usage based off of the value that you're offering. It's like one X is the value you're getting out of it. And so you should be charging by one of those things or N of them or whatever. The usage-based pricing being converted to seat-based pricing. Is sort of like this legacy financial accounting thing that a lot of companies do, which is like, well, we want to be able to predict our usage and we only know the number of, you know, engineers we have. And so being able to multiply the per seat, you know, times engineers or non-technical people, whatever your tool is, it makes it easy for them. But you have to realize that there's a lot of waste in in there. And so there is this challenge of like a a pricing guide that's easy to understand and also the like straightforward value association. Early on, we made the same thing. Like a lot of our competitors were charging like the number of different monthly active users they had for for logging into a portal. And we're like, well, that doesn't make sense. Some users log in a hundred thousand times, other users log in once. And also our customers don't care about the number of users logging in. They care about the number of of these events that they have to do, the the complexity of this or the amount of data. And that that made more sense. So That's a different story, though, than what a lot of companies are doing now, which is they don't understand what their users care about. And so they're just like, well, our infrastructure costs N dollars per token. So we're also going to just charge like N plus one dollars per per this thing that sort of maps the tokens. And so we aren't going to, you know, have a like a finance problem. We're not going to run out of money because our revenue is too low. But that's different. You know, what is the value you're getting out? speaker-1 (40:36.77) We get that question a lot when we sell our products, like why can't I just pay for the tokens? Their desire to capitalize on the cost savings that are coming in the future. Like everybody knows AI is expensive today. What they kind of are implicitly saying when they want to pay like token pricing or like some some abstract pricing away is they wanna realize the benefit of like churning from Claude to beep seal. Like they wanna they wanna feel the benefits of that. speaker-0 (40:56.61) Basically what you're saying is that they know that the infrastructure you're running on is going to be cheaper over time and they want they they want reduced c price, basically, because they believe that to be true. But this whole mindset is like obviously wrong in a way. I mean, because you're not you're not paying for the infrastructure anyway, right? You're paying for the value you're getting out of the product. So speaker-1 (41:06.796) Yeah, and those are the buyers of our product. speaker-0 (41:15.394) You know, believing that you'll have a reduction there just doesn't make any sense. It's like going to any sort of SaaS provider and being like, I want a contract that reduces the cost by ten percent per year over year because you're using AWS and AWS also reduces their their costs over time. So you should you should, you know, pass along that cost savings to us. And my answer is like, Yes, of course we will if it happens. Like our we will charge you less if we can charge you less, if that makes sense. But also we're not charging you based off of the infrastructure that we're providing. We're charging you based off of, you know, what makes sense for this value and what we can compete at. Yeah. Yeah. Well, I mean the mistake is believing that you're charging based off what your costs are. And most companies are not doing that and it wouldn't make sense to do speaker-1 (41:55.794) Signals a potential feature where like software actually becomes a commodity, right? That's actually what it kinda implies. And in commodity markets the margins just getting pissed away to zero and everything becomes cheaper everybody. And on a big scale, like people are like, is there a b A I bubble? you know, it it's unclear whether some companies will be worth a lot of money or all if all companies will just shrink in value. speaker-0 (42:13.038) If you are competing on price with your competitors, well, in a perfect capitalist society, the marginal value from anything goes to zero. Because if the marginal value is greater than zero, someone else will come up with a product and charge a smaller price for that same marginal value until it goes to zero. So if the price of generating software goes to zero and all software is like plastic, a waste product of something else and it's disposable, then that also goes to zero. And then we basically say there's all all value of virtual products essentially goes to zero. speaker-1 (42:41.358) zero. Outside of monopolies, right? Like all products will trend towards like their cost basis. So it might actually happen, right? Like maybe maybe the difficulty of creating software is what was present preventing that from happening. And for a lot of businesses, the cost of labor slash software like what keeps the price high, which who knows. speaker-0 (42:44.802) Yeah, yeah, absolutely. They to. speaker-0 (42:56.3) With any software, there has to like you have to put something else in it, right? And this is true for anything, right? You have the raw material, and then on top of that, you have whatever refinements you make and you're paying for the cost of labor there. So cost of goods plus cost cost of labor. Unless you're you're instilling some value into that product that you're making. And the question is, well, will that change? But I think the finer details of how capitalism works may be a topic for another episode. speaker-1 (43:22.178) Yeah for the speaker-0 (43:23.892) Z so I I mean that's not this podcast. I can recommend a few financial ones. none that I like, but for sure an option out there. at this point I think we should switch over to picks. So maybe I can ask, what have you brought for the audience today? speaker-1 (43:37.164) My pick is a little video game called RuneScape. And my little plug for it is I was really excited when the Jev like decision models were created. The way I tested it out to see if they were good or not was I got it to play RuneScape. Okay. And actually like I just like set up this library that would go in like go and play the game as if it was a player and see how far it could get. And the further I can get, the smarter the model. And that was kind of my benchmark that I created for it, which is really fun. But I grew up playing RuneScape and I learned how to program. playing RuneScape I should create bots for it back in the old days. That was always my number one pick. Yeah. speaker-0 (44:11.362) Maybe this reveals too much about myself, but there was a time that I played a lot of World of Warcraft. And I had just found out that one of the reasons I had quit is a lot of the real life people that I knew who had played and stopped playing. And I felt like I didn't want to play with a bunch of NPCs. And what I had found out later is that the guild leader actually had transitioned to still quote unquote playing, but their version of playing the game was basically on spreadsheets. They manage raids and and dungeon encounters and whatnot and the people, but never logs into the game ever. Like they don't actually play physically the game. And at that point I'm like, what is the game? Like what what are you you watching it watching a video of of an L L doing stuff? Is that is that enjoyable? I don't know. speaker-1 (44:52.11) I mean, do you watch y YouTube videos of World of Warcraft? 'Cause that's how I keep up with Runescape. I just watch other people play. So I don't have to invest the hundreds of hours they do, but I still get the dopamine hits of hitting the levels, you know? It's speaker-0 (45:02.974) I I was never one of those people that watch other people play things. That's that's not my thing. I I mean unless there's like some novel thing about what it's doing. So maybe it's a speed run. Like I'm I it's a game I used to play and I'm interested in how did people speed run it, but there's always sort of something that I'm interested in learning. I usually don't watch YouTube's for like pure entertainment value unless it's like a comedian. speaker-1 (45:22.956) Runescape might be a little bit special in that in that regard. a lot of their popular YouTubers get more views than like the number of players they have. It just like s like there's this like huge audience of people that used to play that still are into the game but don't don't have time to play anymore. They just like get their hit by watching the YouTube video. speaker-0 (45:35.82) It's nostalgia, right? They're like, I I I want I want to relive how great it was, but I don't want to s you know spend the effort. I'll just watch someone do the things, right? And you don't have your own recordings. And this is like the next level. Why would I watch a human do it when I can watch when I can pay an L LM to to play the game for me? In those cases, are you running the the LLM locally? speaker-1 (45:50.956) Exactly. For this test that I ran, I generally don't run local LMs. I'm into the frontier right now. Okay. speaker-0 (45:58.766) yeah, so so RuneScape, that's definitely a a callback. I have I've never played myself, but I know a lot of people that dedicate a lot dedicate time to playing. And they just had their yearly convention in it was in UK, not in London though, some somewhere else. I think this one was Birmingham for for whatever reason. Yeah, what did I bring? All right, my pick is the Martingtop 75 liter backpack. So this thing is like a total beast, and I actually don't have it in this room, but I travel a fair amount, not like I won't say a lot, but having basically a duffel bag with backpack straps on it, I can highly recommend. So it's not like one of those normal backpacking backpacks where there's only an opening on the top and it you have to crawl all the way down to the bottom or use one of these really uncomfortable side zippers that like everything falls out when you when you use it. You can like put it on its side and open the whole thing up. And this has been a game changer for me. It's just so much better than any of those other things with the extra straps. I recommend anyone who does any traveling use it. Funny enough, people are like, I'm going to Europe for two weeks. what size duffel or travel thing? And I'm like, Don't don't get a duffel. Don't get don't get like a four wheeled thing. You want a backpack for for vacation traveling for sure. And this thing it was so cheap, I couldn't even believe it. But it's been it's been great, honestly. Are you you're a traveler? speaker-1 (47:14.198) I I I am a t traveller, but I I'm a light traveller. I just bring a backpack. I'm based in Canada and I go to SF a lot for for my job. Every time I'm I'm at the airport somebody'll be like, No carry on, huh? And I'm like, Nope, just the backpack. Just one set of clothes and we're good to go. speaker-0 (47:28.046) It's pretty sad at this point. I like I if I'm going for like one day, it's one backpack. If it's three days, it's like or to five days, it's a different one. And then it's like depending on the length of stay, like I have a dedicated pack just for that, which I I know sounds terrible, but after you're on a plane, I don't know, twelve, twenty four times a year, you really start to think, Okay, how do I make this way more enjoyable in every regard? And especially 'cause I vacation a lot in Southeast Asia, I feel like you're you're on a Multiple planes and then a car ride and then likely a cruise too till you get to your final location. And having to wheel something around on like dirt roads, do not do not recommend. Well, Sucan, thank you for for coming on and discussing all the browser stuff. I I know we got a little bit away into the the Ethereum philosophy of LLMs and token usage and product pricing, but I have enjoyed every moment of it. speaker-1 (48:18.254) It was fun. Thank you for having me on today. speaker-0 (48:20.398) Yeah, absolutely. And thanks for all the viewers for tuning into this episode and hopefully we'll see you all back again next