Ah, Warren, you're looking kind of sharp there in that jacket. You look kind of like a NASCAR driver with all those corporate logos. What's going on? Man's fun It's funny you should mention that, because I have something interesting for today's episode. One of the recurring themes on our podcast I feel like is incident management. It's something like lots of people want to talk about, and quite a few guests that come on to discuss their stressful, traumatic experiences with on call and whatnot. One of these guests has stepped up and wanted to be today's sponsor for the episode, and that's a pager Duty. So I've actually been a fan of page Duty in the past when I've reached for an incident management tool. And what's nice is, compared to the competitors that we've heard from, it's clear that they're actually listening to feedback, unlike you know, other enterprise companies utilizing your internal messaging platform like Slack to interact with incidents. Especially for US, I feel like it's like a baseline requirement for communication and collaboration, and they've actually opened up their Slack integration to everyone, not just the customers who have shelled out money for their enterprise plan, so it's really nice to see compared to competitors where it feels like you need to pay an enterprise tax even if you aren't an enterprise company. I'm particularly like their automatic channel creation when there is an incident. You know, if you have like lots of incidents, like one every single day, that gets pretty tedious. So, uh, thank you Page Duty for sponsoring this episode. Yeah, that's super cool. Definitely thank you to Page of Duty. And I'm a big Page of Duty fan. It's one of those tools which I think there's not a lot in this category. I could be wrong, but it's just one of those like when you are trying to do a certain thing, you know, incident response, like Page of Duty is just the name that comes to mind. You a sake of you know, people thinking like it's really easy to do, and then they go out and do it themselves. And like I've been in a bunch of companies where this has been a pattern of trying to do it yourself. And the best part is when you hook up your incident management or on call reporting you know, monitoring solution to your production systems and you're running on the same infrastructure, and then you have a production incident. Well you know what else is down at that exact same moment, So I can I can highly recommend not building this yourself. That's my own sort of traumatic experience. I think that's a rite of passage, building your own monitoring system. And then you do it and you learn that lesson that you just brought up and like, wow, that was stupid, and then you go to patre Duty's website and move on with your life. So I've I've shilled out for the episode. So maybe we'll talk about something interesting now right on. Well, yeah, I think today's guest might have some interesting topics. O Mair, how are you, Bud? Yes? Good, how are you? Thank you for having me here? Dude, welcome back. I can't believe we determined it was three years ago, right that you were last time? Yeah? Yeah, that's just nuts. Yeah, Like I don't even want to have this conversation with myself about what I've done in those last three years that I also don't remember. So the listeners are are missing out on that pre recorded conversations that were having about you know, how long how long has it been? Actually? Because Will you were telling us, you know, last three years may or may not have happened for you. Yeah, And I did mention before we started recording that I bucket things into events that happened between before nineteen ninety and things that happened between nineteen ninety and yesterday. And I really can't get any more granular than that. So Omer was just here yesterday, I know exactly. And you know what's weird. I noticed that you and I are wearing the same T shirt today, and I've got to go back and look further recording because I think we were in the same T shirt last time you were on as well. Here's a fun fact. I only have one kind of T shirt. I have like thirty of these differing colors. It's the same T shirt. That's very possible. Yeah, my wardrobe is very much like that as well. My wife bought me one of these T shirts once and I was like, oh, that's super cool. And now that's all I. Have, just that one T shirt. Just the one one copy. Yeah. I mean Omar's over there flex and saying that he has like them in different colors and multiple shirts. I have just this one. Every day is laundry day for you. Yeah, sure, yeah, yeah, yeah, yeah, we'll go with cool. So we were going to talk about Kubernetes and ll ms, right, yeah, give us a rundown on that. Over the past years things have happened for me as well. So I'm an already architect a zest, which means that I lead our Kubernetes products. So we're building stuff to help you optimize kubernets clusters. And recently it seems like, well everybody have noticed the one hype that drags the world to direction. It's all around AI now and people are starting to focus more on kubernets and AI at the same time, which is something I never expected. I actually thought the word goes specifically for DevOps. I thought it was always going towards serverless and kind of not worrying about infrastructure and managing your own stuff or not caring about resources, which in some way did happen. And you did see like different platforms like both serverless from Awus, azure GCPAA, but also platforms like versaill, Heroku, fly Io, things like that helped you just deploy your app and move on with your life. But then that's some I don't know what changed the tide but it seems like companies are pushing towards Kubernetes. The last time last Cubicon, I think it was in London, they said seventy something between seventy and seventy five percent of corporates in the world are either already using kubernets or migrating. So that was mind blowing to me. I never saw that coming. And over time we've started seeing companies naturally go to AI, either using AI throughout their products or trying to train lms, or actually adopting them after they've been trained, but trying to deploy them on their own. This is kind of things that we can focus on. We can also speak about the unrecorded part that has to do with AI hype and how people are vibe going their way to production. I think we're definitely going to get there. But you mentioned a lot of interesting things because I was on the same path for you, like Servilis for me started like twenty twenty fourteen, even like before Kubernetes was a thing, and there was ideas of this coming into a lot of companies, And I feel like there is this question of like why did not the better technology become more popular, the more extreme like push things aside, focus only on the business value. And you basically you just said it really got me thinking like why did And I'm sure I'm going to get some angry emails for this. Why did a worse solution become more popular? Why did you get better adopted? And I think it's because there's this natural tendency for humanity to take step changes for things rather than giant leaps. And there's actually a core concept for this in mathematics. It's like you found a local optima and you're just making small little jumps, and in order to find a larger jump out larger maximal, you have to make giant leaps. And in Japanese it's called kaikaku rather than kaisen if you're familiar with manufacturing lean terms. And I feel like it's really uncomfortable for people to throw away everything they have and make a huge leap, and I feel like serverless is a huge leap, and Kubernetes is people can just keep doing what they're doing today and delude themselves into thinking they're making a real change. Do you feel like it's a step back. I don't think it's better than what the Open container initiative Docker containers could have been, but given the companies that were backing it, which was pretty much Docker, I think that really answered the question of why it didn't get further. My opinion of this is that the one great thing that Kubernetes did is growing insane, like an insane community around it, which goes to open source projects under C and CF, but also companies either building on top of these open source projects or just pushing themselves into C and CF, which is incredible because from a very raw product that you had to do so much to just get to production, you can now get your well you deploy Kubernetes. You can either do it on your own, but you probably won't. You use a service, but then things like helm and Customize and other things around it can just help you deploy things. And then companies started building on top of that operators, so you can get you know, elastic logging, monitoring databases, cash instances, you can put whatever you want. With an operator, you just help install something and then you have everything you need. It's not from the start, but it kind of gives you control over everything. And then people said, okay, then now you have to manage infrastructure, have to manage notes and what's going to be with auto skating and things like that. But then you have projects like Carpenter, which is AWS is pushing, but Azure is jumping on that wagon. So you can kind of have the best of both throws, right, You keep your control, you don't pay as much, which is debate of but we can talk about that. It's Kubernetes. It's something that deploys containers. That's it. I think there's a nerd aspect of it as well, because it's something that's just fun to totally nerd out on. It's so configurable and so flexible. I know quite a few people who are part of the Kubernetes at Home project or the Kubernetes Home Lab project, and the amount of money and level of work these people have put into their home Kubernetes lab for doing who knows what, you know, like telling the refrigerator when it's time to defrost or whatever, and and they just get completely passionate about it. And I think that's a big allure to Kubernetes because in our industry were people who like to just nerd out on stuff and tinker with stuff, and with servilists, you don't get that option. You can deploy your container and it works, and if it doesn't work, you can deploy your container again and then it'll work. Yeah. I mean, I think the whole lad thing is interesting. You got experience doing something that you liked more than whatever your company was doing because your company was doing something horrific, and you can go down that route. I mean, I do see people who are even running stuff at home that would prefer servile lists, but they feel like it's too much of a burden to convince their organization to make the switch. Fundamentally, it does feel like it has to be a switch, you know, And I think that cognitive burden or political burden is just too much for people to deal with a lot of ways. What I'm actually interested in is something you said earlier, and that's you're surprised there's a marrying between AI and Kubernetes because, as we'll pointed out, running locally in a home lab, like what are you going to run? Like, you're not going to run dockers Form and you're definitely not going to run Nomad After everything that Hashi Corpus done, so like what are you left with? You're you open open stack or coro os, like you're talking about spinning up operating systems everywhere. I do feel like Kubernetes is an answer there. But for AI it also is like one of the few things that I have actually recommended, especially if you need to spin up lots of models or configure the parameters for running those models, for doing infernts that are different per user or per customer. Having independent models scales well with different namespaces in Kubernetes, whereas the other container orchestrators don't really have this concept. You just spin up the same container over and over again with the same parameters, and it's not really fundamentally controllable. So I feel like the models a little bit different there. The beautiful thing about Kuberneties is that it's very extendable. You can basically do whatever you want. Right It's a set of APIs that you have on the basic binary. But if you want to build on top of that, we mentioned operators earlier, we're building operators at ST. You just make up your own APIs, you deploy them to the cluster. They work, so you can kind of change whatever you want. You mentioned name spaces, you can if you want it. You can build your own notion of namespaces that would fit whatever you're trying to do, and it would just work. You would have to install it with other applications as a service provider or as a stand alone project, but anything is possible. That's how they This is something I really appreciate about kubernets. First of all, I totally agree with Will that it's totally something to nerd on about and just see things moving around and having your containers shift around different notes. It's cool, it's fun to play with. Here's a hot take. It created job security in a sense, so people are kind of motivated. You mentioned home labs, right, both of you people were deployed. Who would in their right mind with deploy Kubernets in their home lab. It's just it's it's so many layers of complexity. You can solve it with a fifteen other open source project, but you're going to kubernatis because you already know the beast and you feel like you've tamed this. So now you're going to deploy everywhere, including your own home, and that's how you want to see things progress. Because you moved people would literally only only answer to recruiting emails that have Kubernetis in this job description, right, So that's part of it, and I think it created like this ecosystem of companies are building for Kubernetes. People only want to work with Kubernetis. It's be they build a fomo around it. Everybody wants to be there. It's that somehow keeps being the next cool technology thing that always builds up and progresses for partially for a good reason. You ask what's good to use in your home lab. Well, you can use Kubernetes. There's k three S, which is the smaller sibling, the lighter weight thing Kubernetes. There's also we mentioned Fly Versail, all these commercial companies, there are open source alternatives to all of these, like Qualify is one that comes to mind. This is an open source Versail, slash Heroku, slash Fly that you can deploy in your home lab, which integrates you with anything. So if you wanted LMS, for example, you can. It has a list of you can make up your own plugins. It's just containers. But you can't choose an LLLM or speak to open AI from your application, or put your own LM there to run next to your application. And it works. So I'm curious about the LLLM aspect of it. You know, using lllms with Kubernetes, what kind of what do you actually gain from that? Right? So, I think there's a lot of use cases. The two main ones are either I'm an AI company and there's you know who is not an AI company these days. So great anyone who wants funding. Is every company's name has changed over the past twelve months to something AI. First of all, there are these companies that are either they own the LLM or they have to train it right, so they need a fleet of large and powerful machines to train their lms. They need big discs, YadA ya. And then the other part of it is these services are starting to cost a ton of money, especially if that's your core business. You're starting to pay like your cloud bill is now the second problem in the organization, not the first. So some companies find it that running your own l M or you know, there's a ton of open source lms. You can go to hugging face and get whatever you want, maybe lighter ones, smaller llms that can work even quicker, maybe tailor made to whatever you're doing, and then you can run it alongside your application, which means reduced latency not as high costs. But then this brings the complexity complexity story of infrastructure right because you have to right size things. Speaking about Kubernetes, Kubernets lets you dictate how much memory and CPU you're going to use. But once that set, and we'll talk about the last upgrade of kubernets. But once that's set, it's set. That's it. It's there. Over time, you probably want to change these requests and limits to fit whatever the application is doing. Either it's consuming too or it requested too much and it's not actually utilizing everything, or it needs more and now there's no more memory to serve, so you want to change these things. There's a few ways to do that, but it's really really hard to do automatically. That's one aspect of things. The other is these llms are usually just large, large language models. They also consume They consume a lot of disk, and with disk space it's exactly the same. You can create a PVC on kubernets, but once it's there, it's really hard to change. You can extend it. Sometimes if you're working with the right cloud provider, the right c sign. I'm not sure if you're aware. Kubernet Is one point three to three was just released a week or two weeks ago, and you can now go to your requests on a specific pod and change them and you don't have to restart the pod. They can just change and it will inflate inside the node to whatever it needs to consume, which is a really really cool change and that helps us a lot. You don't have to restart pods anymore if you want to scale up the resources that you consume. So that's a really big release and that's only been out for a week or two. I mean, I hear things like that and I think, wait, why wasn't it doing that all along? That seems like a required fundamental piece of the infrastructure. And you mentioned earlier the APIs have been figured out. You build a lot of things around the APIs to make adjustment. So my question is, like you actually like the api is that Kubernetes provides, because if you're building operators all the time to sort of adjust and change the abstraction layer to interact with your provider containerization level schedule, or your orchestrator, then I feel like you're getting closer to exactly the promise that Servilus has been offering all along. That's a philosophical question right there. Oh, definitely, that's what you're trying to do, right You're trying to abstract things by building them yourself because you don't want to mess it with infra, but you want to control it, which is kind of the conflict we all have because sometimes a lot of times we need the control. Right, we as a company have to have access to the notes, so we can't work with serverllus. We literally have to make changes on Linux. By the way, the recent change why would why can't you change a container? A running container? Why can't you change? The resource had to be developed in order for us to get the functionality. So we're always trying to make changes so that it's easier to deploy things. Production is more stable, our life are easier. But we still want the overall control, and there's still the ability to make changes to APIs and new operators and you know, change this beast. However we want to sell new products. I'm gonna keep going there, Like, how many companies actually need this level of control? It depends what you're calling this level. I mean, not everyone needs access to the noes, but some do in a way, not every You mentioned the scheduler earlier, By the way, big thing now in Kubernetis, don't know if you know, you can't really access the scheduler, which is just another open source component within kubernets. You cannot use it in most cloud providers as it was intended to. Meaning since when you're using Kubernettis through a cloud provider again Aws, Azure, GCP, or any other flavor, the scheduler is part of the control plane, you cannot access it. You can use it if you're instantiated a new pod the scheduler, we'll take that pod and then we'll schedule it wherever is available. But if you want to extend the scheduler, which is something you can do in kubernets, for example, you can plug in your own extender and create custom logic scheduling, you can't do it. So that's another place where Kubernetis was built to serve everyone so that they can do anything. But then the cloud providers take bits of it and say, okay, no, no, no, that's serverless. Now you're not touching that. We take care of that, which is okay until it's not. I don't know. It's the ever going conflict of who manages what and who has access to it. By the way, AWS can run you know, fargate on AWS. Probably other services other cloud providers have the same, But fargate you can run Kubernetes without nodes. I think to some extent it works. But what happens when you do need the access I don't know. I think most most companies actually don't use it. Oh, I mean, so fargate it's sort of a special part of AWS that gives you the serveralless aspects of container management without having to go deep into understanding the complexities of the node management or scheduling. And it was like I'm going to say it was recent, but it may have been three years ago still that you couldn't actually run fargate effectively with an EKS on AWS due to some of the limitations that came along with it. Like you could do it, but then for whatever reason, you wouldn't be able to get Internet access or i AM wasn't working correctly, like you know, permissions didn't really work out of the box for accessing other services, and so you know, people obviously were trying to do that, but it was sort of a joke from that standpoint. If you want Kubenetes, you're only going to get the horr mode version of it. I think now some of those have been or most of them have been fixed, so there's, you know, back to very little excuses not to use that. Although the canonical reason not to use our gate or server list has been access to say GPU, the if you are building models or doing any sort of video rendering, et cetera, et cetera, you aren't going to be able to use fargate. I want to say, it's been a while since I looked at it, but it used to be the case that you need to actually get virtual machines that have access to GPUs or GPU optimized machines in order to run your cluster. Yeah, that's a great point, but if you don't have that requirement, I would urge you to test what was farget was built originally for ECS, which is the alternative, a WS alternative to Kubernetes, which works great, really a great orchestrator. If you don't have anything complex, you don't need operators, things like that. You are going to be married to a WS naturally, but you get everything out of the box, just as you would expect with any other platform. Right, you get autoscaling and routing and the firewalls and naturally cloud which connects to for every monitoring need you have. Again, you're going to have to pay something for AWS to run all of that, but it's going to be usually it's going to be cheaper than running Kubernetes, something I really like doing. Running ECS with Farge is great. Yeah, no, it's one of the best things. Ever, I'm surprised that more companies don't find opportunities to utilize that. Like, if you're I usually start out to con station of that. You need to prove why you can't use that as a solution before you decide to just hop over to EKS or running kubernets on top of your own easy too. And we talked about the community right around kubernets. There was an ever going rumor I think there still is that AWS are going to ditch ECS in favor of EKS forever for like a decade really, and every time you ask them they would officially unofficially that we tell you no, we're building it. It's core business. Said that. I think Netflix was running a large part of it once, it's not going anywhere, but there still is a rumor. So many manages to maintain that rumor going on and just you know, convince people to ditch whatever they're using and go over to kubernets. It's probably started by the Amazon eks team. Kudos to them. Right cool. So back to your point earlier, Warren about who needs this level of control, I think there's a it's like one of those right time. Right place things. You know, you hit a certain level of scale where if you're going to manage your costs, you do need that level of granularity or at least visibility into it. And with a lot of the serverless type providers, you know, you just get this huge bill because it provisioned whatever you told it to and then your finance team is like, hey, dude, you need to figure something out here. And that's whenever you start wanting to get more control over it. But I think it's it's not something that most people initially need. It's something that you find you need after you've got everything else up and running, you know. I mean it's interesting you bring that up because I always like big the model contrarian here. We don't use Scubernetes, and not only do we try to embrace over lest wherever possible, we actually try to use edge workers. So in out front that's lambdad edge or cloud from Functions, or in cloud Flare it's web workers. I don't actually know if Azure and GCP have something, which is the fact they never heard of. It encourages me to say, yeah, they don't have something, but I'm not going to be caught recorded saying that's you know, on the record for sure, which is really interesting. And honestly, even with that, even if you multiply our compute costs two or three times more, it's still nowhere near the top of the biggest cost concern in our organization or even in the cloud. I have a question to you. Maybe that would be a segue to the other point we talked about earlier with vibe coding and how everything is going around AI. But it feels, it feels, at least from scrolling doom scrolling LinkedIn that everyone's building something right. Everyone can now spend a weekend and build an MVP or even a product and push that production. And I wonder where they're starting something, starting a narrative where people on the Internet can just build their own products mainly with AI. By vibe coding not vibe quoding whatever, you can pretty quickly get to something working and then deploy that to one of the platforms we mentioned earlier, which is mostly completely serverless. Right, you pay a monthly subscription based on how much you use, and you don't have to worry about anything. So do you think that would change the tide a little bit in how many organizations are using Kubernetes and how many people most of the people we know, I think, are not actually running businesses on their own. They're part of a team in a company, a large company that uses whatever one of the cloud platforms in Kubernetes. Do you think that will shift something? Because so many people that are building products and trying to build their own businesses and using so much serverless because they don't code, and they don't know how to manage infra or at least that's not their core business and that's not what they want to deal with. I mean, that's I think what I heard there is technically lms are generat like when we use vibe coding, the result is actually a serverless solution. So everyone who uses LMS, the vibe code or generate solutions using AI in anyway is actually saying Kubernetes is wrong. Serverless is the right answer. Yeah, I mean sounds good. I like that argument. I mean you're on you're left with that option only right, because it generates code. Most people who I think most people who use it don't actually tell it here's the architecture we're going to use. This is how you're going to separate things. It's not working like that. They actually they describe the business logic or how they want things to look and work, and that's all they care about. They don't care about how it's built, whether it's the most efficient solution ever. They just care about something that does the business logic they care about and for it to be accessible to other people on the internet, right, which basically means serverless. Like you said, so, I think from my research and what I've read through like door reports, and we actually interviewed a whole bunch of product managers, what we did find is that quality goes down, but the throughput on delivering solutions goes up. Speed for development in a way, when you're using vibe coding or ELMS in any way, so I think the question is, you know, are you willing to make the trade off of less quality for delivering a solution faster. Then the ones that were most effective in this mode were the ones that could basically give an LM a spec of their solution, so not just the architecture, but literally how it's supposed to work, and those interactions followed from what product managers do in some way. Now, if product managers today are giving specs to their development teams on what they should be building, I mean they're probably not doing a very good job because I don't know any humans that like taking you know, hard coded specifications unless you're a consulting company or contracting company who's doing software development doing value based work getting paid by the spec But most most teams are not, and so you transition their abilities to working with LMS, and they're very effective with churning stuff out. Tim goes with like deep challenging hard tech. If you're an engineer and you're working with some specification released by standards body, converting that into actually something working using an LM is way more effective because it is very much consume this data and transform it. Transformations are very effective. So I have a little bit of my own hot take here, which is that my theory is that the more engineers nerd out about a topic, the less value it offers the organization. I tend to agree. Yeah, that one's going to be hard to argue against. I wanted to ask you something. You started by saying, based on what you measured, that quality goes down, like throughput goes out. Yeah. And what I hear between then and correct me if I'm wrong here, But what I hear between the lines is we moved the problem to our future. Set right, because it's out there. You see there, you go, it's in production, you can see and use it. The functionality is there. However, the moment things need to scale or you know, stability. If quality goes down, it's not as stable. There's more bugs to fix. These things tend to grow exponentially. So don't you feel that's just pushing the problem either elsewhere or to the future. Yeah, and in a critical way. I think this is one of the pasts that will cause that, Like the downfall of humanity doesn't come from you know, robotic AI terminators that are that are you know, impacting us. It comes from very subtle things that we've already accepted. I was reading some paper and I don't remember what it was, and I may have a link later, but it was said, we're comparing often humans capabilities, like how well we do versus how well LMS can do, And what we should be comparing is our weaknesses versus their strengths. It's very part of war some sup perspective here, what problems are we causing for ourselves that lms are you know, falling into and are going to cause this problems in the future. So yeah, for sure, it's a huge issue in a way. However, I think this goes into the perspective of like, what do you actually need in your company? You can sacrifice quality in some way and deliver your product because your end users don't care about it, then yeah, for sure. You know, increasing throughput on delivery, increasing your delivery rate, you know, is a thing that you should do. But if you care about performance and reliability and architecture, you know, something that my company cares about. I think a lot of companies secretly care about this. If you look longer term, you can't be using LMS in this way to be long term effective. It's going to be a critical problem for your company, sooner rather than later. I think it goes even beyond that what I've seen in one of my projects and one of two three more developers. But it's basically just me and I figured using so much it's curser now. But I've used a bunch of them. Sometimes it would build the feature and the feature works, and then when I code review, it removed a bunch of other lines totally irrelevant to what it was trying to do, which is I could not figure out why or how. And this made me think, if I deploy this to production, my throughput goes up. Right if it's a GERRA ticket, that Gerra ticket is now done. I've finished my task. I can move on. But if I don't have automated QA or the right CI pipeline, nobody knows about this thing until someone needs this feature a month from now, which again begs the question is throughput goes up does not mean that everything is done correctly, And if people don't actually code review what's going on? Is it real? In a way? And that made me think that maybe developers are kind of moving away from being the ones that write most of the code to the ones that have to review most of the code. So we know that that's going to be a failure though too, because in order to effectively review stuff, you need to be able to have the whole context of what's going on. And for a human you're like, I already forget things, you know, I'm sure everyone forgets things in a solution that has millions and millions of them lines of code, and so especially code that you wrote yourself. I think there's tons of jokes out there is like, who is the idiot that programmed this? Oh? Oh that was me? Actually so and now that idiot is is going to be an LM and also produced ten times as much or one hundred times as much code that you've never seen before. And so that's not a realistic solution to expect people to actually review that. They're gonna, you know, looks good to me and approve it. And the counter argument has been for a while, oh well, they'll just also create automated tests and you'll review those for the business cases and validate your solution against it. However, the problem is that the context window, the context window is fixed sized, so the thing that the input tokens into every LM will never go to infinity will never be able to contain all of the relevant information that is necessary because it just it's a computational model. Even if it gets more and more, you still have to provide it that context way. Maybe you hope that providing it all of the source code on MPM, if you're using JavaScript, and also your source code, and also your GEO tickets, hopefully you're not using Gira, using linear or something else, and you know your GitHub or get lab repositories and every email that was ever sent in your company, and every Slack or discord message or some better chat tool, like everything the company has ever done. Maybe you have enough context there, maybe you'll get to that point. The interesting thing though, is that humans aren't computational models, So the value we're providing into the system includes some non computable box black box that is an input to this buffwer development into the business development, and where if we're taking humans out of the loop there, we're actually by nature removing something that an LM will never be able to replace. That's super interesting in my Again, I might be it might be disrespectful to elms, but I feel when I'm working on a project for a few months or a few years. I have a deep sense of familiarity, and like you said, maybe one day this context window grows enough to replace me. But at the moment, it feels like even if things work and it's not deleting lines, that it shouldn't they work because it's just created a bunch of additional code that is already there. And maybe it's doing things. I have a ready sketch instance that works for my application, and it's just built another layer of interaction with Reddish, just because it wanted to extract something, and there's a library specifically for that called Reddish. It just couldn't find it and just did something on its own. There's a great paper that compares the Linux operating system versus I think it's ECOALI and how the DNA structure represents the source code and how these two things compare to each other, and you find that Linux is sort of this upside down pyramid where there are some root modules that are fundamentally critical and used by everything, and then there's leaf nodes that you know, depend on composite things that end up depending on the route, an upside down binary tree, if you will, and whereas e COLI is like a right side up pyramid. The most critical functions are highly replicated throughout the DNA, because if one of them becomes corrupted, you don't end up with a single point of failure or catastrophic failure for the organism. It can still continue on and you know, not leak all your customers data to the internet. I mean not, you know, just go through apoptosis and die as an organism. So you're saying it's a good thing the way it operates, well. Yes, and no. I think there's an intentionality behind the evolution where you can say, well, for reliability, it needs to be this way. But the LM isn't doing it based off reliability to preventing its mutations, right, I mean it's not going in that direction. The other thing it's not it doesn't care about is maintainability. Right. If you'd have a feature to build and it's just added a bunch of additional code, it might not be all that critical to anyone, not even the resources. Fine, a few more lines of codes, just a few more bites that are stored, especially if you're come finding it. But it's not mainteenable, and that begs the question should it or if we're taking if elms are going to take over everything, it should not really be maintainable. However, something that's not maintainable is not really reviewable. Right If there's one thousand lines of codes added to everything, every little feature you develop, because just how lms work, it's not really maintainable, scalable, reviewable. You just kind of shift humans away out of the process. I think there's a huge mistake where we're generating things from l lambs and committing that as the relevant artifact or other humans to review. So in the case of generating source code, having humans review that, or even writing tests and then reviewing that, or you know, generating I think emails or blog posts written by l lambs and outputting that, Like, that's not the value. Isn't the output In this case, the value was the prompt. It was the human input here or what however you generate flipping or flipping a coin or asking the l lump to generate, Like, it doesn't really matter. You have a prompt, that's the thing which was valuable. It's like when someone says, hey, I used an LM to completely generate this blog post, I'm like, cancel. Just tell me what prompt you use to generate the blog post, because then I can do it myself and interrogate the result. I don't need your blog post. You didn't apply any original thought there. You just copied what someone else created. If you use claude, you copied what anthropic thought. If you use CHGPT, you just copied what OpenAI has for data. So just get get rid of all that. And from a source code standpoint, it means committing these prompts and trusting the underlying models in some way, or doing some sort of model validation separately, and then using the prompts as the mechanism and so on. Every build of your project's solution architecture, you rerun all the prompts against the model, generate a new output validated against some historical data from what your users use, for instance, and go from there. And I think that's a much more mature understanding of how lms can be effective. So I heard someone doing that. But in order for just throwing out prompts and expecting some results and then doing something, his prompt is always I'm going to ask for something. Don't do anything yet, just give me the plan and build it in the best best practices in mind, blob about mcp another buzzword we can throw in there, but if you have the right mcps, you can actually grab the best practices from whatever you're building and then give me the plan. Let's talk about it, let's go every over everything, and then once we're done and I approve, then you start building, which he says reduces the number of errors by like fifty percent and he can maintain the same output but without so many errors. I think that's. There's a video I was just watching yesterday from Anthropic mastering cloud code in thirty minutes, and it's from the guy I don't know his name, the guy who created the claud cli, and that was that was his fundamental approach in the talk is first is just have a chat with the AI about what you're trying to do, have it, throw out some suggestions on ways to approach it, and then talk those through and really just having a much more interactive conversation with it before you ever let it start doing anything. And then I think to touch back on something you guys brought up a few minutes ago, like, I think that's the real role, the long term role of software engineers with AI. It's not reviewing the code, and it's not having AI replace you. It's about giving AI a clear set of instructions and scope so that when it goes off to do a task that it doesn't you know, build a completely new library instead of using the one that's already there. Or one of the cases I had early on, I asked it to write some tests and then went and checked the word work that it did, and it it was trying to like install Postgress inside of my doctor container so that it had a database to to use during the tests, And I'm like, no, no, we're. Not doing that. How about you just mock the database? Call okay, can we do that? But it comes down to like to like, you know, giving a clear set of instructions. You know, had I told it up front, that it would have gotten to the result faster. And so I think that's probably the downfall of like vibe coding and letting it take on large chunks of work, is it's going to make bad decisions and then you end up with the problems that we've talked about already, with something that's not maintainable, largely inaccurate. And then tying back to the original conversation, it's probably going to be a cost hog when you try to run it on some serverlist platform. I feel like this is the the quintessentially example of our user experience. Right, it's as a user, don't do the thing you want, instead, you need to be trained to use the tool. And I feel like I'm dystopian perspective at this moment where it's like we're being trained on how to interact with the robots that we've created, rather than changing the models in a way to respond to how individually we work. And I mean I say that it's like sort of really ridiculous, but you know, we're now in a way beholden to our AI overlords who have already decided what's right and wrong, and only if we interact with them in the correct way where we actually get a valid response and we're what we're looking for. So do you think it's a valid analogy then to say I shouldn't have to learn how to drive a car. I just want to get in it and go. Yeah, And I think that has improved cars over time. Right, automatic seat belts, automatic braking, automatic air bags, cruise control. Right, I mean these things are like, right, We're bad at all of these things we should provide capabilities and improvement. And you see, like maybe UI products for companies that do care about the user experience are improving them. And I know you meant that as a joke and I took it too serious. No, No, it was a serious question because I wanted to see how the analogy compared. It's actually a great analogy, right, It's something that we use technology to. It's people's work, right, driving cars, driving taxes, driving whatever. If you move that over to ropaths, then these people need to change their line of work, which I'm trying to think of in my line of work. Is it going to be redundant? Is it going to be able to be done by AI solely? You don't need to review anything, you don't need to All you have to do is prompt the right things and it works. And I wonder if this is going to happen and when and in what way? Because throughout history, every time there was a technological advancement, people thought, Okay, that's the end of the world. Everyone is everyone's going to be out of work, and the opposite happen. Instead of it improving our lives and making us work less hours. Is just the other way around. Okay, great, more profit like more throughput more profit, work more, produce more. I like that you brought up this example because I actually feel like it's a counter example to the argument if we if you look at books like Sapiens, which was released not too long ago, we see that the goal for improving or automation has never been to improve individuals lives. That's exactly where I put this from. By the way, allow allows society to support additional humans, even if it means subjugating even larger portion of those humans or entities organisms to you know, a below poverty line, or you know, sacrificing even more for them. So it's not that those technologies exist to make humanity better, it's humanity exists to be able to make the technologies better so that, you know, other we can increase our population size. So you know, there's a question. Of who's controlling who's controlling who? Yeah, and how many humans can be supported on this planet. But the argument means that we as humans support the technology, right, so we produce technology to improve the chology. That's basically what we're doing. That then reproducing that one. Deep AI may take away our capability to reproduce in the future. That's that's where you wanted us to get too, well. Right, I heard an interesting argument that said that humans are the sex organs of machine it's going to take over, but where it's we're the reproduction capability. I mean obviously not taken to be literal, right, I don't know. No, there's there's got to be a movie or a book that's that that covers that topic. Like that's that's some pure sci fi gold, right there. I mean, in a way, it's sort of a how viruses work, like AI in a way is a virus and we and viruses work by getting into your cells and if there RNA viruses, replacing your DNA or inserting in your DNA their own set of DNA sequences for those amino acids, so that your body, you're the cells automatically produce the virus itself. And it wants to reproduce and infect others, right, which is great at we're using everybody's using My parents use AI now to ask whatever they want and it works and then they tell their friends and that infects someone else. So it's exactly like a very good virus, to be. Honest, effective virus effective. Yes, a virus. Virus is virus is canonically by scientists all over the world have said to not be alive. So I like this analogy. If humans are the cancer, then the virus the AI is definitely the virus. This lives me with some thoughts, Well, bring them on. We got to we gotta keep going with this episode till we all get canceled. I don't know how to bring this back to devovs and code about it feels like we're installing viruses. And I think it's useful for people to take a deeper look at the technology that they're utilizing and how it's being deployed within their company and what changes that they're making over time to make that more effective but for them and both their future jobs but also the long term in the company. Wow, well done, Warren, getting us right back on topic. That was a pro move. Thank you. Yeah, and I got nothing to follow up. I have so many thoughts I need to process. What's the What's the most important one for you? I think would be the question. So you know, you brought up the topic of not just Kubernetes and how ZESSI is utilizing it, but also the impact of building it to support lllms, both internally and from third party companies. Have you seen hands on specific challenges other than what we've already talked about, not really. If I'm trying to connect that into both the philosophical aspect of it and the technical parts. It's mainly focusing on improving, right, which is everything we talked about is improving either the technology or whatever drives the technology. So people are trying to companies are trying to improve the way they run lms, the lms themselves, the infrastructure that surrounds it, and honestly their cloud bill at the end of the month, which has to do with everything. And it's also a funny aspect of AI because it consumes so much energy resources not in the form of computers, and chifts in the form of energy, right, and the costs are to other professionalism. What they're saying is that it's not maintainable and it's not scalable, and it's going to hit a wall at some point, and I'm wondering whether that wall is going to be more companies trying to run more tailor made lean llms that only serve one purpose or general purpose solutions like that are going to be more consumed by AI companies, much like we're consuming cloud resources from cloud providers and shifting away. So I don't know exactly where it's going, but I seems to be a very very expensive resource at the moment, So I don't know. You've pulled out the optimistic perspective. There's a good principle on this, and I just I the name eludes me at the moment. But even if you make it cheaper, you'll end up with this actually contradiction where the result is more usage, not less. And I think the biggest problem is that we're already seeing we talked about this a little bit in one of the previous episodes, that companies will just continue to use additional energy and rather than care about trying to make it cheaper, will keep on trying to figure out how to build more and create more energy. Yeah, and so this means open reopening coal and gas mines. Now, the question could be, do you think there's an opportunity for good here where there will be companies will start trying to invest in figuring out how to get fusion reactors so that we can get a step up in energy creation over because we're never going to get there with sore or wind or water. I mean, we have seen some situations where I believe it's trying to trying to beam energy from outside the atmosphere back down to Earth. You know how in Google flights, when you're searching for a flight, it would tell you how much how much pollution it creates or whatever in terms of forgot always the measurement. So at one point we were trying to do the same because we're in the business of making infrastructure more efficient and effective and in a way allows you to reduce both your costs but how much infrastructure you're using, which you can follow up to aws not using as much. No, so maybe we're you know, supporting the environment by reducing the energy companies use, and at some point we're trying to give you how much you've saved, but also how much you've helped the environment by saving on resources, which is an interesting analogy. I don't know if it would work, but okay. So I'll be pressed by you know, the contrarian perspective. So here it is, it's the only way this could be effective is if we paid companies for having a low spend but compared to what how do you actually do this. We know carbon credits didn't work for reducing carbon dioxide emissions into the atmosphere, so yeah, so I mean that fundamentally is the problem. But there's actually another issue here. If reducing your spend or using comparative technologies, let's say the cloud providers using AWO said, you know, here's A and here's B use B, we'll actually reduce the cost more. You know, this is actually cheaper, not because it's technology technologically cheaper or requires less energy, but because it is better for the environment. The problem is that you'll start to see companies pop up that abuse option B and reselling that, you know, at a cheaper way to other companies, so like competing with aws but increasing the price and then taking a cut of it. So there are companies out there that just deal with carbon credit resells. They buy credits to resell them, or even the worst case, the worst polluters in the world just buy tons of credits from these intermediaries, and so it doesn't help at all in any way, and you're just making another company rich in the process who is just abusing that gamification. Model can fight human nature, you can with AI. Maybe no GCP has that there's different different regions you can choose, and some of them like display their carbon emissions or carbon offsets that you gain by using resources in that particular region. Yeah, I mean, I guess you'd have to be fined for how much you're utilizing. But you know, from a human perspective, if you get a fine, then you say, oh, no, it's okay that I'm doing this. The government is, you know, extracting their reward for their their you know, return for that. So I don't know how you can really think about this in a way that makes sense, Like maybe there's some way. I just haven't thought about it enough. But it seems like there's not a lot of good options like how much should it be? How many? How many carbon credits like or usage should I have as a company? Like one five is five a lot. I don't know what the appropriate numbers are. I think it's like one one kilogram per like international flight. I think it is the number. I don't know if that's right. I'm just gonna go with that kilogram of carbon dioxide per international flight. Yeah, I think it's something like that. I don't know if that's right. And what does not even mean, okay, let's say that that's what does it mean in terms of pollution in terms of that effect on the atmosphere, well. I mean actually figuring out what the direct effect is is an impossible problem to solve. So you know, you just take what the pollutant is and you measure it, right, like the amount of whatever poison that you jump into the river. How much poison is it matters for human beings? Well, that's sort of hard to describe. There's like a huge problem right now with the forever chemicals. Not the not like the teflon on your on your hand, but the products that care hate the teflon on your pant are dumped in the water by companies. And how much of that is bad? Well we can say that you know, one is work is worth like two is worse than one, But is how bad is two? How bad is one? Like? That's that's a really hard problem to answer. So I don't know what it is for carbon credits. I do know there are a bunch of companies out there that are investing in trying to expose as information and somehow utilize it. And there's lots of countries with grants available to create green products or projects, but they don't usually focus on like the carbon credits because it's like it's such a challenging thing to go off of. So instead they invest in things that they believe are sustainable for whatever definition is sustainable. You have like Kubernetes. This actually always made me think whether everybody hates the cloud platforms? Right? Everybody wants to. I mean there's a people want to manage their own infrastructure. They don't usually, but they like to hate on them. And I always thought, do I by using a cloud provide like AWS, RGCP or azure, is that better for the environment? Solely on that perspective, Is it better to use something central that has a lot of resources that are actually shared resources in a lot of ways, which means it's more efficient in a global level as opposed to a company or me just putting a server act here, which which could naturally consume a lot more than it actually should because you're you're buying in order to scale, So companies who do that probably buy lots more than what they actually need. Is it better for us to use? Again, solely on the perspective of efficiency utilization and how it affects the environment, is it better to to use a cloud provider than to run your own infra? Yeah, I mean that's sort of difficult. I think there's a couple of different parts to that equation. The first one is how bad is it for what you're doing if you're running on PREMI I think that the recycling of electronics waste is like one of the biggest waste recycling problems in the world, and it's like only getting worse by a huge fact. There's a magnitude in order to do the waste processing. It actually consumes a ton of energy, both humans and like physical energy, and no one plant can take care of it all. Usually it's like, well, we remove the plastic parts and then ship all the other parts to other people, and then we we take out the gold and then ship the rest of it, and then we take out the silver and ship the ret like we can't do anything else, and then at the end of the day it's all in the ocean. Yeah, that's right. So that's sort of a hard answer. I think, how long as an individual, both as a company and a individual, are you good at handling your technology waste? You get a new iPhone every every year, you know, that's probably bad. Although you know, here's the flip side. You know, what would you use the money for? Are you taking the money that your company saves by hypothetically not using the cloud provider and doing you know, using it for green purposes? You know, are the products you're creating and making the world better? Well, yes, okay, it's that's an absolute. Really, you have to compare it to the company you gave the money to. So is Amazon on if using aws taking your money and building green product projects to improve the world or what they're doing worse than what you would have done with the money that you had as a company. So sometimes paying less could be worse for the environment. Other times paying less is better because now you have more cash to do things in a better way. But that doesn't mean that you will. I always like to think that being in the optimization business, it's reducing waste at the end of the day, regardless of how it's reducing waste on the application level translates to the resource level, translates to energy papapa. But thinking about it further, that might not always be the case, which is super interesting to me. We optimize things because we like to. Builds. Might very well be a local optima that doesn't affect the chain. Right. Another thought, I'll be left with the night. It's busy work with a dopamine hit. Well, I feel like we've thoroughly covered the topic. We think should we do some picks? I think I think it's time. Where Yeah, we've been here before. Would you bring for a pick this time? Two things? One, if you're watching series TV series, I really liked Mobland. Did you hear about that? It's guy Ritchie Tom Hardy. It's really cool Mobland. That's one right and completely relevant. And the other one actually is a little bit relevant. Is a few. I think it's a few. It's probably one software developer from Google on their spare time start building kind of an alternative to GET, which is not really an alternative because they can work together. So I started using it in one of my projects. It's called jj jiu Jitsu. It's a really cool open source project that kind of lets you work with change management, but not as much hassel as GIT. So it's just at a chain of changes that you can just change at any time, move through history like it was nothing, where GIT makes everything a little bit more complicated. That's it. These are the two. It's like editing the object model graph asd or or GET as your direct mechanism. So I feel like, you know, people that want to spend more time with their source of control revision system but feel better about it. This sounds like the perfect tool. Yeah, yeah, it's it's really nice and they work together. Again, if you're working on a Git project and everybody else is working on GitHub with the git locally, you can still run jjly on your own machine, but then when you're done, kind of wrap it in a commit and then push it to a different branch which opens a PR and everything, so you can enjoy both worlds. I really liked it. Right, all right, Warren, what do you got? So I have a very controversial pick this time for our listeners. I'm going to say doing surveys that that's going to be my pick, now hear me out. I'm not talking about like doing surveys that like from to pay you money, because those are a waste of time. Although I did start like that as a person who thought that you could make some money doing that, I never did. What I'll say is that surveys are like I see as my opportunity to change the world in my favor. And by doing them and giving feedback means that I can change these companies how they're thinking, hopefully so that they start and actually listen to that and then make some changes. And so if I withhold my opinions, that means I basically say, I love the world the way it is right now, but I also don't care if it changes, but it definitely couldn't be better, and I don't think that's true. I like complaining about things. So now, lots of companies we know just completely ignore the surveys after they're done. You know, if you've ever taken like an EMPs survey to ask you if you love working for that company, we all know your executive team is completely ignoring whatever you wrote there. I'm sorry to tell you that. But on the flip side, if you fill out the survey for the Adventures and DevOps podcasts, I can guarantee you that you'll be entered into one of the four remaining twenty dollars AWS credits left that we have in store, So you definitely want to do that. I'll sign up for that. I just have to say, I'm I have a searching in the gym. Every time I come back home, I get an email with a survey and I filled it like one, two, three times. Nothing happened. It was it was I put in a lot of time. They didn't even they didn't even reply. So it feels like they are throwing it away. But I'll definitely do yours. The only reason that The only thing worse than doing the survey and feeling and not getting replied, I feel like they throw it away is doing the survey that's really long, getting to the end, and when you click submit, it says like, oops. It crashed, right, you have fifteen minutes. Yeah, please don't if you make a survey, please don't do that. That's what happens when you have vipe quarders building that oops. The other thing I don't like about surveys is when you get the email that says like how do we do and it's like a smiley or a frowni emoji, and so you just click the smiley emoji like cool, we're done. Oh wait, no, you're asking follow up questions. Okay, I'll do the follow up question, and then then there's another follow up question and then it's like. No, screw you. I'm not doing this. I was trying to be nice, but now f off. So everyone that's listening. If you're building a survey, remember that Will says that you can get him both with the email and then one more question after that, and maybe one more if you promised him something. That's that's that's the threshold. I mean, there is a there's like the sun cost fallacy. Right, exactly what I wanted to say. Yeah, so you know you've already committed to submitting your feedback still more. Before you got the reward, right, for sure, you got to. You gotta feel like you're unlocking something in each step of the survey. Yeah, it should definitely have a like anyone who build a survey platform s, I should definitely see every survey that I submitted, all the feedback so I can be like, I got five more points for this for filling out this feedback. I don't know what. That's absolutely nothing. I mean Google does it and it makes me feel good about leaving reviews for restaurants and other places. You have to getting your green Garden githubh. Oh yeah for sure? Right, yeah, so you just need to tie the survey to fake Internet points and everybody will be fighting to fill it out. Yeah, well, what'd you bring cross? My pick has to do with some changes for me. I have become the engineering manager for a new company called Katana, and so my pick is going to be Katana. If you want to go check out a website, it's Katana dot Network. It is a layer two blockchain. That specializes as a DeFi platform. So we're like really rethinking the way that decentralized finance works. And how to make it. More financially rewarding, but also a lower barrier to enter, so that if you've ever played with DeFi in the past, you know that you had to go and get like buy ethereum and then find someplace to convert that to wrapt ethereum and then find a bridge that would let you swap it into what you were really trying to invest in, and like every step of the way, you're going deeper and deeper into rabbit hole and not really sure if like this place that you're interacting with is just fixing to steal everything in your wallet or if that really is the right pass. So we're trying to eliminate all of that. So well, that's my pick. That's quite interesting. Honestly, I see you a get a new new opportunity there. I know whenever I try to do anything with crypto, I always ask an l ALM to decrypt my my wallet for me. Now there's a way to use energy efficiently. Well, I don't think. We're ready to get started on in the next podcast. Yeah, I just. Had chat GPT remember my seed phrase for me? Cool? Yeah, so there you go. If you're interested in that, checkout Katana dot Network. It's it's been pretty cool, like there's some smart dudes working on it and I'm excited. So we'll have a link below the podcast. But is there like something you're specifically looking for at the moment, like are you looking to hire? Are you looking for customers or users? Like what's what's the breakdown? So I am hiring a full stack role. So if you're interested in that, hit me up. But other than that, if you're just interested in DeFi check it out and I would love to have your feedback to see what we're getting right, what we're getting wrong. Yeah, do that survey right, It's so there's no survey. Just hit me up on X or email and say, dude, this is cool or dude, this sucks and that's the end of the survey. Uh oh, so you're gonna plaster your email over the all of the internet so people can respond to you. Well, it's not hard to find. I mean yeah, like, based on the number of recruiting emails I get, my email cannot be hard to find. Cool. Oh ma, thanks man, it's been fun having you back on the show. Yeah, thank you for having me. Good to see a boat Warren. As always, thank you appreciate everything you do here and we'll see here when next week