And we're live with another episode of Adventures in DevOps. Almost blew the podcast name right there, Warren, Well, you know, thanks. Catching, I've only done I've villain done a few hundred of these at this point. It's it's still a learning curve. So how are you, Warren, to bring us a security fact today? Well, you know, I think again. I had brought one last time, and I thought, you know, this time for sure our fellow co host was going to be here at share, the long awaited one, Matt. I had been promised at it, you know, coincidence not here, so I don't have anything today. She's pausing for dramatic effect, like a really really dramatic effect. It's gonna be the best fact ever. I swear it. When it actually drops, it is it is right well. Also joining us Daniel Loretto from Jedifhy. Daniel, Welcome to the show man. Thank you so much. Will happy to be here. Dude, I'm excited to have you here. So you are with Jedify. Now tell us a little bit about your background. Yes, I'm a software engineered by trade. You know, I've worked at a lot of different tech companies of all different sizes, so I can talk about, you know, different experiences. But I worked at Google, Airbnb, Twitter, I worked at you know, tiny startups that you've never heard of. I worked at Verta Health, which is a telemedicine company for the ABDIS, where I ran engineering, data science, it info SEC kind of like all the tech stuff. And now I'm founder and CEO of Jedify. We're a company kind of trying to make it easier and faster to turn your ideas into working software. And anyways, today we've been playing with AI and agents and how they can help with the software develop process. I mean, you you've got that opening so well nailed down. I can tell this isn't the first time you've. Had yourself think you. Got the elevator pitch nailed well. I mean, it's sort of funny because like, I meet with a lot of people, and a lot of them feel the need to sort of introduce themselves and say a lot about their background, even if it has nothing to do with the conversation at all, And some of them just go on and on for minutes, and I'm like, you're mentioning other people's names and I don't know who any of those people are, and you're talking like they're the most important people in the world, and like I for sure, like I don't care. I mean, at least you start to company name, so like, you know, that's great on you. And I think a lot of people know most of those companies. I think most of them still exist by their original name, so you know, that's what's important. I was raised in a Midwest Protestant community, just really deep for it's just not saying anything. You're gonna mute me, aren't you. I got, I got. I actually got sucked for a second because I spent not short period of time in the Midwest after I graduated from the university because it was the only place that I could get hired as a job. I wasn't actually a software engineer by trade, and so I ended up there and I was like, do I have a connection for this, Like can I talk about the religious values? I'm like, not really. I actually I actually don't know a lot there, Like I've never really been a huge into the religious scene, so I can't say anything. And so I just was like, I'm just gonna keep my mouth shut. I was trying. I was thinking like in the second Austin Powers movie, whenever doctor Evil goes to therapy with his son Scott and they're like, tell us about yourself, and doctor Evil gets up and he does like this long rant about how he was raised by his father and it goes really really deep. But I couldn't remember enough of it to try to work it into it as a joke. Here, So don't don't worry. You'll have plenty of opportunity to go look up that for for next time. So today chick for the next S episode. I don't spoil it now. I really like the idea of getting back on topic of dev boxes, environment machines, Like I feel like this is a huge challenge that still exists. And you know, it's interesting because I work personally and like over the years and a lot of different ideas and a lot of different software languages, and I've always hated it, Like there was never a time where I'm like, you know what, working in this language, this is the best. And I feel like, you know, if you're in the space, you've got some horror stories. You've got some like like how did you even get into this? You know? Is why is this where you went? Yeah? So man, So I mean I guess a lot of these stories start with kind of scratching your own edge and trying to solve a problem that you have. So, you know, just having worked at different software companies, I've seen those stories where things work on your computer and they don't work on your teammates computer, or there's a bug in production where you can't reproduce it locally, and often that's tied to the environment. And so when we started edifying, we were like, you know what, from the beginning, let's just set up some reproducible the environments. And we decided to set up some Docker files for every kind of piece of software that we were working on. But very quickly we run into some pains because you know, Docker uses a lot of resources on your computer, and so when you're developing locally, like shipping to production great, but developing locally like I don't know, the fan spinning on your laptop, you do a little change and you want to see the results, and oh, it was the wrong line in the doctor file, so you busted the entire cash and now you're like rebuilding the entire container. And so all of that felt really, really painful. And so we had heard about this technological nicks. We wanted to try it. We kind of felt two things at once. One it was like, this is awesome. It kind of creates reproducible environments. And then we were like, oh my god, the developer experience and UX for this thing is not what we wanted to be. So anyways, we ended up building an open source tool called debox that kind of wraps nicks and lets you create those environments locally. And you know, I just posted it on Hacker News before. We knew it. It was like number one on Hacker News that day, and so kind of told those oh, people are having this problem that they want some solution, and that's yeah, that's how we started with the developer environment work. I love the story of like pain and suffering through development process because it's such an unusual I mean, it's such a common strategy for innovation, right, Like I as a software engineer, you can fix your own tools in a way, and so realistically, like when that happens, it's like, well, I'm going to make something better that doesn't it isn't as bad as the thing we're using. It's interesting you brought up docer. One of the things I've seen a lot is engineering teams attempting to get the development experience on their local machine working through Docker. And this is something I've never understood, and maybe this is something you could jump into if you have some understanding about it, Like why do that? Like, if I have the source code on my machine that I've cloned from get repository somewhere, why why not just install the couple of tools I need and you know, hit F five and run it. What does doctor really get me there? Yeah? So I mean I think the issue forget doctor for a second, like why isn't it enough, you know doctor or no doctor? Why isn't it enough just to check out the code, install the tools, and color to day. What happens is that in many cases, you don't just need the source code. You need the specific versions of the tools that you're working with. So it's not like I need I don't know Go or Python? Is I need Python two point seven point eleven? Right? Hopefully not anymore? Yeah? Ussion but just well but but see, I mean you say that. But like what happens a lot is is you're working at a large company, uh, and there's this project that everybody depends on, but it's not staffed well enough because all the other things became. More important, and you got to keep that thing running. Right every day everybody depends on it, and nobody's willing to update exactly there is. I think we should just pause for a moment here, because I know a non trivial amount of our audience members are like suffering through some sort of PTSD withdrawal right now. Having said that, like, or you know, you're working in some environment and you know some code that hasn't been touched in a while is not working, Like I feel like that's like ten percent of your job, whatever it is. You know, you have no idea what's going on there, and it's not working, and you didn't write the code. You may be lucky if you knew the person who did write it. Because so one of one of my first jobs I had, there was an engineer who if I saw his name on a commit message, I know that there was gonna be a bug with that code that was going to cause me to lose the rest of my best I like, if I'm debugging around and I see a line of code by bye by by Craig, uh, that's it, Like I'm sorry, Like I know it's gone. So but but anyway. I mean, yeah, that's the point, Like you often have to work with these projects and they often need specific versions of software, and if you don't use those versions of those tools, things. Bring, right. And so I guess in a in a project where you haven't set up too much around developer environments, you might just have some read me file saying install Python. Uh, and so everybody is installing Python, but somebody install Python. You know a year ago, somebody installed Python six months ago, somebody in Stoll python today. They all got different versions of Python, right, and you compound that by all the tools you might need in a project, right, so it's not just Python. But maybe let's say you're using protocol buffers, you have a compiler, you know, and so on, and so. Yeah, I think that it's rude, like that's. That mismatch of versions is kind of what people are trying to solve that is not solved by just to read missaying install these tools and start working on the project. Yeah, I definitely agree. I mean there's a huge aspect here, which I think has to do with the languages themselves and their package managers are part of that. I got it working once. Let's not go back and touch it by even the language maintainers themselves. You know, it's very limited experience there, and so they go off they you know, the packaging manager for whatever language you're using, like it's working, and so there are some that are definitely not as great as others. And then you just compile that on and put that usage as a dependency for it literally everyone who uses that language and framework. And I still think there are languages today that no one is happy with the package manager for that language, and yet somehow they're getting over it. And I mean there's languages where you have five package managers. So I have a feeling that you are throwing the JavaScript environment under the bus there a little bit. But I have to say that JavaScript is one of no JS specifically, is one of the best development communities compared to the other ones. Like yes, there's en package managers, but all of them like work at least ninety nine percent of the way, And there are problems with them for sure, individually, you know, they're not so totally perfectly robust, but you know, compared to some other ones, like I don't I don't struggle to pull out a new jobascript package manager. I do struggle to figure out how to make Maven work, even though I've gotten it to work like a hundred times before, and. I haven't touched the GEMA world in such a long time. I have no comments. Yeah, uh, just had a little bile come up in the back road. You know. The thing is like I remember seeing sharp like new Get was not a good package repository. The tool wasn't great. Even dot Net improved a lot of things there, but they had a diamond dependency resolution problem where like you depend on two dependencies who require different versions, and and no j has like figured out how to get around that. NPM solved it, yarn't solved it, PMPM, they all have solutions there. So yeah, I mean the individual tools aren't great, and it's not great that there's lots of them, but at least I feel like they're not getting in my way. But yeah, I like, if I'm programming in C plus plus or something old school with it for like a large company, then not only are there dependencies that are probably commit to a repository, like they're not in some package manager somewhere, they're like the binary dependencies are like in the source code and they require some Microsoft reddistributable C plus plus library from twenty years ago that I have to go and download from the internet from some suspicious website in order to even you know, build my project to get it to run somewhere. Yeah, agreed, So yeah. Anyways, I think it's all those paper cards that just slow you down in your development process, and all of a sudden, you are, I don't know, you wanted to work on a feature, right, you wanted to work on something related to the project that you're interested in, and you find yourself independency hell and just going deep into package managers when all you wanted to do was your working on an actual feature. Well, I think that's I think another aspect of that is onboarding new developers on your team, right, Like, how long does it take to get them where they're actually working on the code that you hired them to work on versus setting up an environment to actually see what they're actually supposed to be building. That sounds like a question to air our dirty laundry on the show, So I guess I'll go first. You know, it's like a couple of weeks to a month, and then I think we know that the long tail turnaround is like ninety days to get to peak performance and then it's like six months to do effective reviews at that point. Before that, I think there is a very long time where there's still a lot of training and education that goes in not but for us, it's mostly domain specific, like if we don't hire a domain expert for our company, we deal a lot in the authentication authorization space, So like how Samuel works, how oh off two works, how open id works, how SCIM works, how audit trails work. Like, there's a lot of things that you don't want to get wrong in a way, and so there are some tools. So but it's a long period of time. Will you're smiling there, so I feel like, you know you're going to tell me that you know your new hires that I believe you were interested in interviewing and getting into your team. They're like a couple of weeks max, and they're already a performance. So this is definitely an outlier. So I won't say that this is the state of our environment by any means. But the last guy I hired about a month ago, he was pushing production commits by the end of the week, which is good or bad, right, Okay, he was pushing functional production commits that actually did beneficial things. Wow, that's amazing. Yeah, but it was like it was a very unique situation, you know what. We hired him and he came in with a very specific set of skills that we needed. So he looked at our environment and was like, oh, yeah, here's where it's wrong and immediately started contributing to that. And then there's like the rest of the company, like the Web three space that we live in, where he still has a lot of onboarding to do before he fully understands that. So it was cool to see and I paraded it around the company just because that's the kind of person I am. But you know, and like the big picture of things, you know, he couldn't have done a domain specific commit in that first week. Yeah, are you using something like some sort of thermal development environment or like how was he able to get acclimated so quickly? Yeah, it was over. It was on our Kubernetes clusters, and so he had a deep Kubernetes experience and just looked at the way that ours were configured and was like, oh, yeah, here's here's something that's going to help you out. And then he also had a lot of OI d C experience, so he removed we were maintaining a lot of permissions and roles via pull requests and get repository and so he refactored that to use OI DC so that you just get it inherited when you authenticate. Yeah. I mean, I think the okay, you guys presented two different like onboarding experiences. But I think regardless for me, you want to spend the time creating the main specific knowledge. You want to spend the time kind of teaching and mentoring and having your your employees grow. I think you want to take away there. I'm just fighting with the tools and this should be a solve problem for sure. Yeah, I still have to deal with this now for I don't know, to days, just to get my environment set up before I can even start, you know, doing a little co change and testing it. Yeah, I totally agree. Like, are none of our customers benefit from the time and effort we spend into working on Kubernetes. They only benefit from the time and effort we spend building a better product. That they that they use. And I think that's what you're alluding to and warned to catch on a point you made earlier there, you know, dealing with ephemeral environments. We recently refactored a bunch of ours because we had this one VM instance with I don't know, it had hundreds of gigs of RAM on it, and it was a bunch of Doctor composed files and when you needed a new integration environment, you would copy and paste one of the existing Doctor composed directories and then go modify the stuff that you needed in there. And so we refactored that to use Helm charts. So now the launching of a new environment, you just add a new Helm Helme values file for the environment that you want and then let Helm build that out for you. Yeah. So I will say that NIXT is something that's always scared me personally, Like I feel like I never wasn't an opportunity where it was like, you know what, I'm going to reach for that new tool. So like, you know, congratulations for you know, seeing that as an opportunity and going at it and then building the UI on top of it to make it happen. What do you feel like the turnaround time is for getting new engineers acclimated to using the dev boxes through through NICKS, so they come into a team, they have no Knicks experience, no experience with maybe Thermerald dev environments. You know, they're all doing software development on their local machine. What's the turnaround time for them understanding and being able to execute effectively? Yeah, I mean, assuming we're talking about an environment where the tools needed are kind of available in the NIXT package store and kind of the versions are well represented, I think it's very very fast because I mean dev Box, the tool we built, it presents an interface that's very similar to package managers from different languages that you're already familiar with. Right, So you know, in you know we're talking about JavaScript earlier, you have that package the adjacent file. Well in that book, you have that books the adjacent file, and it lists a bunch of packages that you need for your project. And you might just say, you know Python and the version, and it might say the product of filer and the version. And it has a command to add a new package, to search for a new package, to remove a new package. And so what we try to do is kind of present that Nick's functionality but under aux that people are already familiar with, and because people are already familiar with it from other tools. I think it's a quick learning curve. How does this affect sort of software development cycle? So I make a change in my code, we're talking about impacting the Docker built cash and forcing it to rebuild. Like what's the corollary for dev blox? Do I do the software development locally? And then it uses some sort of remote server somewhere where the dev blox is being spin up? Like you know, yeah, so that box is our open source project. It lets you declare your environment in this Jason file and then the idea is that you can take the adjacent file and reproduce that environment. In many places, we we offer kind of on our paid solutions, ways to spin those environments on the cloud. But just to quarify, like that box is just going to be the open source tool and let you kind of do it locally. And the way it works is it is, like I said, it uses next behind the scenes. In practice, there's no container locally. I mean you can you can choose to use that box inside a container if you're building a container, but you can also choose to do it outside of a container if you don't need one. And what's really happening is that Nick's kind of installs all the versions of all the software that you need, and what's known as the next store. Think of this as some path in your local drive where every binary or piece of software that's installed is stored under a content hash, so that the path for it is unique. So even if you start installed five versions of Python, they will each have its own content hash and therefore will live in a different place in that store. And so anyways like that, it's able to install all sorts of versions of software, even if they would be technically conflicting. And then when you start an environment, it kind of sets the paths to point to the appropriate versions of things. And now you're working locally. So is it changing the environment variables and the alternatives that are set on my operating system to make the environment match what's necessary for that? Yes? Yes, so it will change, for example, change your path environment variable. And so it might you might say, Okay, in this project, we need in this version of Python, this version of the product of compiler. Oh, I know those live over here in the next store. Let me set up the path such that you're going to have access to those binaries and not the other ones. Okay, sure that makes sense. Is that done like system wide or is that done on like a project per pect. Just one a project basis. So so the analogy I make if we want to your reference existing tools for particular languages, it's kind of like on Python you have. Virtual type of environments. Imagine that, but now kind of like. At the system level for the tools and compilers that you're using for a project. So done correctly is what I'm hearing. We think, So for in your own opinion. I mean, I think virtual m is like a quintessential example of it doesn't get much worse than that. You know, maybe maybe someone can point me in the right direction there, but I don't remember. Like usually things automatically get committed to memory for me if they're easy to understand, you know, like I think everyone has this. You know, you see something, it's like all I can remember what that is. There's a le of things. Three months later you come back and you're. Right, that's the thing. Is like, I don't touch it enough. So the only time I actually touch virtual end at all is when I'm generating new h cert box certificates from let's encrypt on one of my remote servers, And that happens once every ninety days and every ninety days I cannot remember how to do that. So you know there's Celsius something is wrong with that technology for sure. So you know it's great when there's an easy way that makes sense to set it up and exit the machine. And that of course, it has to be processed specific and local so that you can potentially run multiple the same time, so that you can run multiple pieces of like different services that have to communicate with each. Other, right, right, makes sense? How do I how do I know that? Early on in my career it was always like, I don't need that garbage. I'm just going to do it myself. And later in my career I've definitely gotten to the point of I should definitely listen to those things and not think that I could do it with myself. So like, what's the advice here? You know, how do you know you're in a situation where whatever you're doing needs to be upgraded to using next through dev box like Quintessential. Yeah, I mean, I think let me maybe start with the cases where I don't think you necessarily need it. I think those tend to be you you're working on a team with this pretty standardized stack, probably have one particular programming language that you're focused on, and for the most part, all you need are the tools for that library, and you're not bringing into many external tools, right. I think as soon as you start having more kind of system level dependencies, additional tools that you're bringing in, maybe multiple languages across the company, then tools, you know, things to control the environment start making more sense. You just to grab an example. Let's see you're coding in Python. If you've standardized a version of Python and you're building a pure Python libraries, I don't know, you probably don't need the tool like this, but as the range al these days, and you decide to do some mL stuff, right, and you need some Python libraries that really depend on native libraries, and so now all those native libraries are going to get pulled and you're doing some image processing and so you need like image magic installed in the container, and so it depends on a particular version of image Magic. And so now your environment shifting from kind of pure Python to kind of like these extra dependencies. And I think that's where this makes sense, because you can now treat those extra dependencies. That's something that you track and version and pin as opposed to I don't know, just will West and you know, pull whatever is available today, and yeah it might be different yesterday from tomorrow. I give you. You know, some examples I give earlier were only on the development phase. But like imagine you you're doing image processing in production and the container requires a particular version of image Magic installed. Like now we're talking about production environments too, that if you're not careful, are just kind of changing nearly willing How does. The conversion from using next through dev box to get your local development environment. I mean, I feel like you open the can of worms here, so uh, when you do that locally, how does that invert to using getting those same dependencies ready for deployment production? Yes, to be fair, I do think that's an area we're continuing to improved of early days on the shift from using the box from development to production. But high level, I mean, you can craft a doctor file and a container in many different ways and really just kind of running any command you want right and setting up the environment inside the container. So what we think is that. In general, you can have one definition of the next packages that you need, and that one definition can be used to set up your developer environment. But it can also be used to install those same packages in the production container. Uh. Of course, you have to differentiate between things that are needed in both production and development versus things that are needed just for development. Now, I said it is work in progress. I think something that I feel Knicks packages haven't been optimized too much for in the past, and I think it is one of the challenges with what I'm saying. Is that. I don't think Knicks packages are optimized for size, and so sometimes they pull in a lot of things into the container and the container grows kind of bigger than you want it to be. So I think that's something that. Still needs to be improved within, you know, in terms of using Knicks for creating these types of production containers. Yeah, but I mean, I think it's a huge step in the right direction because the dependency management through Docker has always been oh yeah, just install like use Pseudo and then install a huge list of packages that are required to run here, and like what version those are, whether or not they work with the actual Linux os that you're actually running there compared to the one on your development machine or in your you know, in your depth box or whatever could be totally different or what's available production because you're not actually running the exact container. There are some differences there. And then there's also your CICD pipeline which probably have a different version of Linux on. Right, And this is exactly this is. Just if you're using Linux, right, I mean, then you throw in ARM and other operating systems and it's it's for sure a huge mass. So I mean, I definitely see that. Persistence is another aspect like Nix doesn't. You're focused on using it for dev boxes in sort of ephemeral way, right, Like you're not making permanent changes to the operating system, but in production you almost want the permanence there. So there is this sort of disconnect, and you're achieving the connection there through using doctor or plugging into persisting it in the doctoring container build file assume. Yeah, exactly, exactly right, Yeah, And I mean I think you touched on this, but just to emphasize it, I think sometimes people don't think about it. A doctor file is not necessarily reproducible. In fact, I argue it's probably not reproducible. People sometimes think of doctor as reproduce in the sense that if you previously built a particular image and you reuse that image, then you know you're getting you're getting what you already packaged before. Right. But if you have a doctor file with instructions and you build it six months ago and you build it today, the output, the resulting image will likely be different because again, I mean it's the same scenarios talking about with with your engineers just installing the environment. Probably your doctor file just says install Python, and six months ago the version of Python, you know, was X. Today the version of Python is Y. And so you build a container six months ago versus today, you get different images as a result. I think code dependencies definitely fall. Code dependencies fall in this abyss here right there. They're not quite infrastructure as code, which has been solved as far as deploying what we have to our cloud environment, and they're not the source code, which is stored so elegantly in there is this middle area where I need to configure that container or heaven forbid, you're using a virtual machine directly or bear metal in a very specific way, and it's like, well it's not I see, and it's not my source code, and then what I'm end up with is some sort of script in Bash. A lot of times to explain how to actually install that thing. And so it's like, well, why aren't we doing declarative programming here like we are everywhere else. It's like the white area that still is procedural and still prone to a lot of issues, And like, that's what I'm hearing is like, Okay, this is totally solved now. You know, if you need more than one dependency, how can you not be using this solution? Right? Yeah, exactly. I think you said in a way that I agree, Like it'd be great to just have your developer environment defined in a declarative way. And then that declaration, you know, we can use to build the environment correctly, and you can use it to build the environment in many different places. You can build it locally, you can build it inside a container, you can build it on the cloud, right, and yeah, take the environment with you where you need it. Will's processing all of this, I can say, I am, but I think I failed to press the turbo button on the CPU this morning because it's processing slow. I'm just trying to slow this morning. That's okay, it happens to all of us, right. So with the when it comes back to the production dependencies, is that declarative in the manifest file this dependency is a production one or in this one's a development dependency. Right now, if you want to talk about like the implementation we have in the in the open source school, you would have like two different books adjacent files, one for production, one for development. But we have talked about you know, potentially UNI find that in one file. Actually we were making. The analogy to package the jacent before and in the package that Jason, you have the dependencies and dependencies that are for both. So yeah, we're we're considering cannot unifying that, but yeah, I mean put aside the exact implementation we have today, the exact version. I think the idea is, can we declare our environments by specifying the packages we want on the versions that we want, and then specifying which packages belong in both development and production and which ones belong only in development, and then take away that very procedural like Bash scripts that you're doing to install every little thing, and then let a system that can reproducibly create the environment. From that declaration you do the work. I think you hit on really well the scenario where you may not need to do something like this, like you don't need the I hesitate to call it complexity of using NEXT based configuration, but it's just it's one more tool, and so. No I would call it like, I mean, it is an extra tool. Yeah, I agree with you, Like. Yeah, you know, compared to I mean, swap that in for something else. I mean, it's pretty much a very simple bootstrap and clear configuration files that are very understandable of what they're doing. I'm curious if you see a particular vertical or industry market segment where it becomes super critical. Still, Like maybe if we say web servers, web development, maybe that's really simple. It's not too complicated here. But if you're building a game, I'm just making stuff up now, like you know, this is where you can jump in and say, you know, this is exactly it, like game or web three. Like, maybe there's a lot more value in or doing something with LMMEL, a lot more value in next configurations. Yeah, I think, I mean, I think one of them is when you have a lot of when you need to use native dependencies that are not native to the language that you're using. Right, So I think Python and mL is a good example. But you know, it's not just the male. Like you know, there's several cases where you might need to install Python libraries that depend on kind of underlying C library. I think those tend. To not work as well if you don't have a tool like like that book. I think the other. Case is when you rely i mean made similar but maybe it's not a library in the scenario, but you're you're relying on a binary that needs to be present in the environment because you want to call, you know, exect that binary or do something with it. How much of a mess is mL development from a dependency management today? Oh say no more think I think it can get complicated. Yeah, because and then yeah, but by the way, you know, I'm talking about that box and mL, but there's even cases where we are not able to help fully in that vertical because then kind of drivers come in, uh, you know, for the different GPUs and whatnot, and and there's some things that NICKS can't fully control there unless you were using next os. Right, But we're remember we're doing this in a scenario where you keep your current computer, current operating system and just use that as added virtual environment. But yeah, you can you can get real. Messy, not just with the library dependencies, but kind of like all the related drivers and how they match the corresponding architecture that you're developing. Again, so you wouldn't recommend doing mL development today quite yet. I mean we wait a couple more years though. Yeah, yeah, I mean I think it's unimproved. Maybe I'm saying I think it's unimprovement, but it's not all the way then. Well, I do remember, and I'll date myself here when I was trying to extract bitcoins from the bitcoin faucet when I was in the university, and that included having to basically getting my operating system to a state where I would just ended up with a black screen with a terminal, you know, and it's like, Okay, you know, I really mess something up here, And like that was the state of the software at the time to just do anything related in the cryptocurrency space in cryptography in general, really good luck using those tools. So like that's definitely improved. Now you can use Python or JavaScript to interact with any of the chains that are available out there ethereum and all the derivatives of it without a problem for the most part. I mean, there are some educases for sure. If you want a stake or mine, you're going to have a problem. And I think that's where IML is right. It used me much worse, and I think it's slightly better. But I do feel like maybe we're going so quickly and we've monetized whatever little amount of value we've been able to create as a society through model development, that we haven't really took the opportunity to innovate with the development pipeline. Yeah, and you know, I'm sure this. I do think there's probably plenty of people working in solving this, even you know, the AI wave and the whole interest in. You. I mean, it's I think it's I think it's optimistic, but I also I also think that may be a little bit realistic because with the deep state model that had come out, I mean deep seek right, Uh that. It I thought we were in a different podcast. Yeah, I mean it's a joke that you know, I frequently go around with some of my colleagues about you know the impact of these models. But the fact that you are there's now competition requires doing things more efficiently. You can't bring a new engineer on board and have them waste months and months not on on tool set, like not even on the domain. So that's what drives innovation there a lot when you're in competition to succeed. And I think that's going to be a really important qualifier going forward in the mL space that we continue to have sufficient competition to drive innovation there. So this is an area where I haven't spent any time or effort, but Daniel, you have, right because you have been working on some mL work to do QAT in testing. Right, So what's the what's the barrier to entry to get into that space? I meaning, like, what's the barrier for. Engineers to start creating agents themselves and kind of what knowledge and tools might they need? Yeah? For sure? Yeah. So yeah, I mean, like, like you said, we are developing an agent that we call test Pilot. It's meant to be an ai QA engineer take away all that boring, repetitive manual clicking on UIs that people often do. Yeah, and we've been learning a lot. I mean, I think I think agents are relatively like big investments in agents are relatively new, and so I think the barrier of entry is something you can catch up on, Like, like I would actually encourage if you're interested in creating agents. I think it's a good time. I think you can learn. I think you can do it, I would say, I mean to me, one of the most important things you need to do with developing AI applications is set up a. Evaluation framework for the work that you're doing. So in non AI non you know, kind of traditional software development lets, you say, you can often write a function that you can reason about fairly well, and then you can write tests that are fairly deterministic to know whether your the behavior is what you expect right as soon as you involve the LLLM. Well, now you have non probabilistic like not sorry, non deterministic behavior, right, And so you might call it once and it might tell you something. You might call it again in a second, and i I'll tell you a completely different thing. And often because you're dealing with very general cases, you might be feeding in lots of different examples and you don't fully grasp how it's behaving. Across all those different examples. So to me, kind of like one of the first things you need to do is set up a bunch of examples with kind of like the expected behavior and then be able to measure kind of the changes that you're doing and whether they're improving that evaluation. That you have in mind. All right, and in this case, it's going to be not purely past fail. Is going to be like, Okay, today you're at fifty percent of being able to handle the cases that you want, and then you do a few changes and okay, now you're at fifty five percent, and then you're at sixty percent and so on. So yeah, anyways, I don't know, I can probably think of more. I mean best practices, but that's the first one that comes to mind. I would even maybe take a step back before that. I mean, that's really great insight, like someone who hasn't spent a lot of time in it. Is there a particular site or foundational model that is where you would suggest people start going for first? I mean probably, I think from my own experience, I know that building models is super expensive, so they're probably not going to do that. But maybe there's a SCE out there that you say, you know, go pull this model and try to fine tune it or use it out of the box. Like any recommendations on that. Yeah, I mean I do think you need to start with the with the pre existing in general model, I recommend using probably a kind of multimodel provider, something like open router that allows you to easily swap models, because what's happening is I mean you guys see on the on the tech news all the time, right, every other month, each AI vendor like outdoes each other, and it's like, oh, we have this new new model and it performs better on this benchmark. And so you don't want to tie yourself so strongly to one model that when those innovations are happening, you can't test the other one, right, So I would pick some provider like, like I said, open Router is one of them. I think Rock. I think cloud Flaer has something these days too, But anyways, a provider that lets you spat which model you want to use and then kind of build the agent on top of that. I think the other part of building an agent is not just the underlying model that helps you reason, but there's kind of how you prompt it, right, and kind of how you tell it to do what you want it to do. And then the last. Piece is kind of like the tools that you give it so that it can take action. Right. So a high level in my mind, like an agent is you're grabbing the LLLM for the reasoning. But you need to kind of give it hands and eyes, if you want to think about it that way, so that it can manipulate an environment that you're giving it control of. Right. Is that something that you've got a significant portion of your team dedicated to or is it something that you can iterate on quickly with just a few people, like how big of a how big of a man power? Yeah, I mean most of our team is working on on test pilots. I mean. Something we've found is so we're a Go shop. Like we use Go as our primary programming language. We've been using it for for a while and so we wanted to continue using it. And what we're seeing is that a lot of the tools that are readily available in this space are Python based, and you know, we can use Python for stuff that's not production, but for our production stuff we want to continue and go and so we're having to build some of them ourselves and go. We do plan to open source them later on and kind of. Give back to the community and hopefully kind of people that like Go, you know, let let them have some readily available tools for building AI and agents. But yeah, that stuff that's getting built right now, and often we look for it and it's like, oh, I guess nobody has you know, done this type of library yet in goal for AI, So it is early days. Yeah, I mean I think that's a really good point actually, Like it's usually a mistake to rely on Python for high reliability software development and production, and so if the tools aren't there to support your production workloads in a different language, then that really tells you how early it is, because the people that are utilizing those tools don't care about like sustainable code or reliable code because it doesn't offer the core features to guarantee safety or quality of your value of your service that you're building. Yeah, like I I mean there's many different ways of building a company, but just I've had to deal with like Groovy production systems, Python production systems, all that stuff, and I'm kind of like in this case where we're making different language and kind of moving forward with that. Yeah. No, I'm totally with you. I used to be like early on in my career. I'm just like every language is the same, right, you know, there's you can swap between one or another if you've had a sufficient years of experience. But then over time you start to realize the fringe benefits of individual languages or negatives that are associated with them, So things like Go or other well typed languages like c sharp, the community or the packages that are available to help drive forward. Having the right execution of the right stuff in production is like super important, not which is when you're building something quick and hacky, but when you actually care about it working continuously, and like if you're at a company and one of the core things you're building is reliable builds, I feel like you know that goes hand in hand or using other I mean, it's sort of like a different mindset, right, Like if you're hiring an engineer and they care about using scrappy tools that ones that potentially break from day to day, the nightly build version of that tool is broken by default, they're probably not in the right mindset to be working on something where you need high reliability or reproducible builds or you know these other technologies. It's just a complete other side of the spectrum, right. Yeah, I totally agree. And yeah, I mean what we've been doing maybe where we kind of allow Python to come in is you know, I mentioned we have this kind of evail process. We're able to try ideas, and so I'm like, okay, if we're prototyping a direction and there happens to be kind of like a Python library that does that off the bat, and we just want to check if that's gonna improve our performance, you know, the performance of our agent. Like you prototype something quickly with Python, pull that library in, run it against our evil framework. Let's see if this direction has legs. Oh, it does have lengths. It is promising. Okay, how are we going to build it now for production usage? And you handle the low we want to handle, and take care of all the edge cases we want to take care of and so on, and then we at that point in time, then we spend the time developing the the go software and infrastructure. Makes a lot of sense. So that's one of the things I want to ask you. You know, you talked about open sourcing some of the tools that you're making and contributing back to the community. That's a huge commitment and it's a it's a conscious decision by companies because a lot of companies are like, yeah, we invested this, it's a core part of our business. We're using it for the business. But then you're saying, I want to I want to give it back to the community. How do you how do you tie that back into your business model and and like justify or account for the time that and the salaries that you're spending on people working on an open source product. Yeah, it's a great question, and you know, I'm sure everybody will kind of make different judgments there, But for us, the way we think about it is, Okay, we're developing tests. I love this kind of q A A I q A engineer. You can think of the software that makes up test Pilot as being into different buckets list in my hand. Right, one is like the truly domain specific logic and kind of proprietary aspects of it that make it behave like a QA engineer, right, and that we're not open sourcing. But then there's the tools that we wish we had had that that we had to develop for ourselves that are applicable to developing all sorts of AI agents and all sorts of kind of. AI related products. And you know, we're a startup, We are trying to get more eyeballs. We're trying to kind of become more prominent, get people to hear about us, and so I actually think of open sourcing almost as a marketing uh exercise. Uh, we don't have marketers, we're all you know, we're very technical teams basically engineers. But sometimes I think you can look as open sources as a way to kind of bring in that audience. And because our audience are developers, right, I think if we're in a different vertical, you're serving different customers, maybe open sourcing doesn't work that way. But our audience is developers, and if we can create a reputation of Wow, those guys that genify really know what they're doing in terms of AI agents and have you checked you know, have you seen the tools they've made and the libraries they've made. I think that all just kind of comes back to the type of business that we're building. Right, I totally agree. I mean we went down the same path a while ago with some of the things that we've opened source. It is like unfortunate to say it is like a huge marketing story. It's marketing story for brand awareness. It's a marketing story for hiring engineers, so they know about your company or what you do before that you know, they show up the end interview. They can research it and see that you have something technical rather than a website with a couple of links on it and a knowledge base like that's not a lot, right, you know, this gives them more ability to have insight in what you're doing. I mean, there is a pessimistic side of me that says, like, no one cares what you put on the internet, Like, no one's that special, right, your thing doesn't matter. I remember like a bunch of times like Netflix went and open source a bunch of huge technologies, and I'm just like, I don't care about those things. But now some people maybe are using that in production. I guess could be potentially because I think they wrote them for Kubernetes or something, and you know, so that's great, you know, you get that experience. I think the one thing we try to be really careful about that I think is important here is that the thing that you're building needs to be really close to your value, your actual product, because even if it's like one or two hops away, then it's the people that are utilizing it. They don't care what that thing is that you're actually doing, which is super unfortunate. But I will say that rather than a marketing tool. There's a second step here, which is if you're going to open source something, I feel like you feel like it's more important to do it well. And things that are done well will live longer and people will see and start to utilize more. And that give brings us sort of like a your own usage levels it up because you actually invested in doing that thing well. And so like, even if you weren't a deb box, wasn't sort of out there as a cloud thing that you could utilize and pay for the fact that you've opened source it means that that tool is now serving you, hopefully better for your mL development, for your own agent. Right, yeah, absolutely, And I actually we're about like different benefits that open source my brain. I'll give you a third one, at least in my view, which is, you know, when you're a startup, I think one of the most important things is be talking to users, right, be talking to users and learning from them. And we've found that when we open source things, we often get to talk to more users, but we also take get to talk to users that normally would not have talked to us. So you take that box. I don't know if we can share the specific companies. But there's some big ish companies using that box that I think if we had just kind of pure proprietary solution and or this little startup, they never would have looked at it. I think what gave them confidence to look at it is like, oh, there is this open source project kind of you know, worst case scenario, the startup goes nowhere, we can rely on that open source project and we can kind of continue improving it and whatnot. And so that's given us at least a channel to communicate with those types of potential customers and users, and it brings in a ton of learning for us in terms of Okay, what features are important, what is that class of user or customer asking us about and how should that I affect our product roadmap and whatnot. Oh that's huge, like in my opinion, like one of the like the primary objective of every startup is to figure out the difference between what you built and what you what your customers thought you were building. And so open sourcing gives you a way to have that conversation without trying to force them down through a sales pipeline, right, exactly right. It's scary though, because you know when you put it out there, it's like, well, people start will start to see my actual bad code, or it is scary my commitment messages you know that maybe I should have worded differently. And I mean it is tricky too, and like, you know, I gave you our kind of like ar heuristic for figuring out what parts of the spilot are open source and which ones aren't. But sometimes you're in some grade lines and you're kind of like should I open source this or not? Like is it Do I get more benefit from opening it up to the community and getting the audience and maybe getting some faster adoption, or do I get more benefit from keeping this kind of like the secret sounds of the company. And you know, there's all those stories like Elastic Search and AUS and. Everybody gets big and starts changing their licenses. So like, son, how I don't know how right I am here. We've done some initial research on the topic, and it does actually seem like customers that fall into a particular bucket never leave that bucket and go to a different one. Like a free user never converts to your lowest tier paid plan, and the average business plan doesn't escalate to enterprise. It just doesn't happen. People self identify with whatever bucket they're in, and so whether or not you have an open source solution really only functions as advertisement. The people who are using it aren't going to be more likely to get to like start to pay you. And one of the things we recognize, which is one of the reasons why our product, authoress is not open source for ninety percent of it, is because those free users actually generate ninety nine percent of the support cases that you need to deal with. So there's like a huge cost investment that goes into it, and you don't get any reward coming back from that. I mean, of course there's some gray areas where there's highly coupled technology and where it's being utilized effectively, but often the one the customers that are going to pay you the most money or be your best customers, like, they don't particularly care about running it themselves in most cases, unless they're like a government entity or you know, they have some regulations they have to deal with. Yeah, I mean usually if they really carry some compliance requirement, but that might just be pointing to like you need to solve that compliance requirement as well, right. Like, oh yeah, I mean, compliance doesn't mean security, so like that's a completely different story. Yeah. Absolutely, it's an additional chex right on. That was that was intense. A good way. Yeah, yeah, yeah, definitely in a good way. Definitely a good way. I mean, I know we jumped through a lot of different topics, Daniel. If you feel like there's something, you know, one last thing you want to share regarding either dev box or the QA copilot test pilot, you know. Have about it, let me think famous last words, go. Oh man, a lot of pressure. I mean, I don't know. I guess I'll just said maybe this. Is less about the specific products and just kind of things I'm excited about. But I am bullish on on AI agents. I mean, obviously I'm building one, but I just think there's more and more parts of software development that could be taking care of through agents, and I think it can be done in a way where we take away that kind of repetitive work and we still leave a lot of the fun and creativity to the humans. I know there's more doomsday scenarios that people talk about right where it's like, oh, we're all out of jobs and whatnot, But I'm actually hoping it's more like, now we can all achieve more because those less interesting parts of the cycle are taking care of automatically, and now you can focus more on system level thinking, on product level thinking and so on. So yeah, maybe I am an optimist like you guys were saying before. I'm just gonna say to that, no comment, I'm gonna agree with you, Daniel, like, regardless regardless of what technology does. I know, there's always going to be problems, and I've I've made a career out of solving problems, so I'm not too worried about it. Okay, I'm super worried about it. Should I share? Yeah? Absolutely, So you. Can worried about it, but like, let's make the world we want. Well. I feel like it's not a technology that's right for being democratized, for access for everyone to utilize equivalently. The people with the most money and the most power will have the most powerful agents at their disposal, and that just entrenches their own power and prevents them from being ousted, and it doesn't leave a lot of opportunity for those that are already being disadvantaged to rise up and fight against that oppression. There's a documentary on this called Terminator check it out. I think I've heard it. I think it has a well known narrator, right, yeah, yeah, No, I mean that's that for me, is that's optimism, that documentary scenario. All right, Now that we've all made our way onto an FBI watch list, let's move on to picks. Yeah, sure, so I'll go first today. My pick is going to be The Martian, the book by Andy Weir. He has a whole bunch of books. I don't know if I'll say this one is the best, but it's definitely really good. They're all science fiction adjacent. I mean, they're not the reality played out in a hypothetical future where well it's new scenario, and they're great. I think it's really well written. One of the first things I ever read from Andy where was actually something called The Egg, which is a very short story which I find also really good. And I and many many years later, I read The Martian and I didn't realize that it was the same author. And it's quite interesting to have read two pieces by the same author and be like, wait, it was this other really good thing that was written in a similar way, and like going back and actually seeing it's like I actually read that thing. So if you haven't read it, I haven't seen the movie yet actually, but if you haven't read the book, I highly recommend it. Right, and I have done both. I've read the book and seen the movie. But I'm happy to hear you say that that his other books are better, because that's yeah, yeah. So I really liked the short story. He has a drawn comic with like an overlap between Alice in Wonderland and Dorothy from Wizard of Oz, and I forget who the third character is. That's okay. The Martians great. There's two other books. I think there's a fourth one coming out. They're not related at all. The one Luna I think for the Moon. It's not as good but still interesting. And then there is one with a Deep Space, which is also really fantastic. Right on? Cool, Daniel, what'd you bring for pick? Awesome? So I have a book as well. It's Wild Robot. So I think I mentioned earlier. I have two young kids, an eight year old and a five year old, and so I do a lot of bedtime routine and reading with them, and so while Robot is one of the books that I'm reading with one of my kids by Peter Brown, there's a movie as well, kind of similar to the example you gave, But yeah, I just find it really interesting to read with my kids, your high level. It's about a robot that founds itself in an island, doesn't know where it came from, and eventually needs to take care of an animal, and it kind of makes you reflect on all these questions, like, you know, where his parenthood is this robot taking care of this animal? You know, is that a parent? What's the relationship between that animal and the robot? And I don't know. I guess I get a little sentimental because I'm reading that. With my kids and I have just I've just found it like really interesting for both of us to go through the story. That's cool, that's super cool. All right, Well, we're gonna make it a three peat. I'm changing my pick and I'm going with a book as well, because Warren you said something about, you know, recognizing the writing style. A while back, I read a book and really enjoyed it, and I was like, Wow, this guy writes so much like Stephen King. It's just insane. And the author's name is Joe Hill. Turns out it's Stephen King's son, so oh, wow. Yeah. Yeah. The book the book I read was Heart Shaped Box. And if you like Stephen King's writing style, Joe Hill just seems to have picked up on that style and continued running along with it. Is it also like thriller suspense stuff or is it a different. Genre, same same stuff, same like sick twisted thing where you're like, oh wow, I did not see that one coming. Yeah, good stuff. Do you ever find it a challenge to like pick up a book that you know is going to be like that and go through it and be like, I'm I don't know if I can keep going, like I'm really afraid of what I'm going to read next. No, I don't. I don't think so. Yeah, He's like, I love it. That's what I look forward to. Right, Yeah, I mean that. I guess. I guess the rest of that conversation is for me and my therapist. But well, don't worry, we'll shut the camera off here in a moment and conversation. Or it'll be like no, no, I promise I stopped recording. Tell us all about it. Well, you see this little red light on the top left hand corner. It's just a U I glitch. Don't worry about it, and thank you so much for joining us. This has been a blast, that's fantastic. I've enjoyed as well. Thank you guys for having me. All right, and to all of our listeners, hopefully you enjoyed it as well. Thank you for listening, and we'll see everyone next week.