speaker-0 (00:07.896)
Welcome back to Adventures in DevOps. we brought in a real principal engineer, not just one only in name, as we get up to some things. Justin Abrams, principal engineer at Parlow. Welcome. Thank you so much for having me. And I see a large set of experiences, Google's, AWS's, Walmarts, and like a whole bunch of companies in between, as well as migration from like East Coast, US, West Coast, and to Europe. I I mean that's just like someone after my own heart.

Yeah, the way that I'd describe it is that I've worked for many Globocorps. and I've worked for tiny startups you've never heard of. My last two jobs have been the middlest companies I've ever worked at. I've mostly spent my time in in Globocorps. That's where I've am the most normal. I the I the structure I appreciate. And middle sized companies, it's a learning process.

I found middling size companies the most easy to jump into. They don't have such strong regulatory or policy driven approaches and they're not like we have no idea what we're doing as a startup. We'll just throw in random tools and just complain about how much everything costs.

Really like to have a scoreboard and I would like to win the game. Big companies know what the scoreboard is. Could you please develop us a plan to get this number three numbers bigger? You're like, Yes, I can absolutely come up with a plan on how to get this number three numbers bigger.

being able to go after something that's trackable, right? Rather than let's just do random things and and pray and maybe it will work out all in the end. How well does that translate to engineering though?

speaker-1 (01:26.926)
So during my time at eBay, my job was to like help make software delivery a competitive advantage. And that was meeting with people all all day. Like thirty five hours a week I was in meetings. I was hooked into the six hundred engineers that were in the payments group and my job was to like help them deploy more, help them have more effective software delivery, lower change failure rate, things like that.

through the course of that work I'd moved them for to have six times more deployments and seventy two percent of their applications were deploying automatically to production with zero human involvement after merge. So like that's an example of like number go up. Number go up, feel good.

I guess there was a question of so number went up. Did everyone feel good, you know, with those changes? Yeah.

Yeah, yeah. No, it was it was a great initiative. It was a lot of nervousness in the beginning of what could possibly go wrong. Answer everything. Okay, let's talk about what could go wrong. Okay, that could go wrong. Okay, great. How do we mitigate that? Okay, cool. Let's that that's a workout. Let's go do that. Okay, now what could go wrong? And kind of iteratively improving.

Especially when things benefit from having the metric, like tracking the, you know, the Dora stats, it's much easier to actually target that and and create an initiative around that rather than even having an argument of like, what is Dora? Should we follow it? Is it even relevant in the first place? Like is six hundred engineers for payments, that sounds like a lot of people. Is that the right number of people to have for for payments? I mean, maybe in my naive perspective, and I've never worked in any sort of payments solution, like that just seems like

speaker-1 (02:44.78)
It's a lot of people.

speaker-0 (02:54.048)
An absurd number of engineers to dedicate to a single area.

There are many payment providers. They're not just like accepting only PayPal and s and Stripe. Like that's not their thing. They will take your real credit card number and PCI comes with that. And now they have all whole fraud vectors they need to be aware of. there's information about how we not only receive money, but also it's because it's a two sided marketplace, you also have to pay people.

It's like one of those things where when you get to the details, it's like, well, yeah, each one of those payment options probably has a team dedicated to just doing that sort of integration. Maybe a team has like two integrations or three, but then you start going down the list. And yeah, the the platform aspect, true. that's definitely another one. Like actually having them pay out people is is something that a lot of platforms don't actually have to deal with. So most of them deal with just like receiving money, right? And so if you had six hundred engineers just on receiving money, that I don't know, maybe I'd challenge that. But you would think that getting paid.

Every company is good at this, but apparently not. Like if you want to give money to someone, I remember not too long ago there was a seller I was buying a book from who lived in Canada and there was an issue. I'm not gonna get into it, but the post was involved and they messed up like really bad. And I wanted to return the money, but I couldn't because like Canada only accepts like money transfers like internal to the country. there and you try to do this, it's like, I'm sorry, I have no way of actually giving you money.

To which point it like we resulted in just giving the money to a third party that was okay accepting it and they paid the other person. and so we have like extra Canadian dollars that one of my friends owes me and that's it. And it's like I I don't know if I'll ever get that back, but at this point, this is still an unresolved problem. Yeah.

speaker-1 (04:33.622)
No, interstate commerce is no joke. There was a thing I I learned about through this process of trying to get this book. But the book was Psychology of Software Teams from Cat Hicks. I want the ebook version and I can't really get it. That's not available for sale in a DRM free way. But I found that there's a service that you can purchase where you can ship them a physical book, they will slice its spine off, scan every page, OCR it, and give you a pe give you an EPUB. And so I'm like seriously considering that process.

With I mean, when you were working at eBay, it was a little while ago, and making those changes. Do you think that a similar transition to getting the organization to one of the the far corners of the door graph, which is like like performing or outperforming teams basically that are doing automated deployment on merge requests or solving one of the other metrics, like reducing bugs and and testing, et cetera? Do you feel like that there would have been a similar approach to solving those things if you were taking on that initiative today?

Yeah, I think it'd be the same. It's just a lot of talking with people and you know, I I so at my current job I'm doing kind of similar work, trying to help people be more effective in their work. And I often joke that like if we did all the things that I'm suggesting, we would be like a a really solid twenty ten company. 'Cause like this is all known. Like there's nothing about AI that makes continuous delivery and small feedback loops, small batch sizes. All of this stuff is known, has been known for a decade or more. Do the things that we as an industry know you should do. And then

you will get outcomes that we at the industry have have learned over time and have researched back up that this is important and it works. And so we can just follow the case studies of everyone else who's done it before and get very similar results. It's not easy, but it's simple.

I I think it's really, really a bit sad that a lot of companies still haven't made those transitions and coming in into them as some sort of senior leader where you're accountable or at least that's your core responsibility to to bridge this gap, like, hey, we saw our competitor, they're doing such a great job. You came over from them and now we want you to do the same thing here. And you're like, I just did that. The last five companies I was also at, how is it that there are still more companies that are are doing this and are are behind in this way? Why why is it that the whole industry hasn't transitioned to

speaker-0 (06:40.641)
Handling these aspects.

There's a handful of things. I'm not sure if you're familiar with the eternal September phenomenon, where it was the case in the before times, people who are on these internet servers would get a new class every September. And you would just get this most inane message board posts of people who were like, My this is my first computer. I don't really know the social norms. And then you train them, okay? And now we've returned to normal levels of discourse, and then a new freshman class shows up and they've never touched a computer before.

And so sep it was like this September. And then they realized that like when AOL happened, now you were in an eternal September. There were always new people showing up who had no clue of the social norms or how to write an email effectively, et cetera. While we've known for some time what it means to do effective software delivery, that's not necessarily evenly spread knowledge. And also new people join the industry all the time. A lot of my work has been this is what software development really means and this is how we do things. And we actually have like terrible on ramps for things like this. How do I do a sprint?

Like what is a sprint? How how do I make that go well? Answer Well, there's two months of certifications. Would you like that as the answer? Or would you like a single document that says what you should get out of each ceremony? Because we don't have that second one unless someone's written it.

And it's out of date and references things that, you know, aren't necessarily relevant anymore. And it's like, well, it turns out that I I love this one. there was a time where the bit the most interesting thing to do was to say, Hey, look, there's a thing called the Spotify model that Spotify came out with. And then like a couple months later, after after the entire world found out about it.

speaker-0 (08:07.522)
Which I the long story short is, you know, here's this sort of way to run a matrix organization where teams are cross functional and you can, you know, evaluate them on their success based on the success of the team and performance management is all rolled into one. it's not a org structure, whatever. It's like, well, turns out actually Spotify was pretty much like right after be like, actually we're not even doing that anymore because there's just fundamental issues with a matrix organization. Like we need to have component teams that are responsible for their own or accountable for their own outcomes and being able to

Go with that full steam without constantly being, you know, have weird reporting lines. But everyone's like, matrix organization, Spotify model. And this was like for I think at least a decade. Like I don't hear it as much anymore. I think the pandemic broke a lot of things. One of them was certain spreads of information, like what was working, what wasn't. But I I do think that companies being behind is part of it is like a lack of of documentation, a lack of how you even do things. I think we're even further behind than that. I don't know if I believe that there are people who know.

what they should be doing. Like we're doing this because it's good and it's not written down. It's like they don't even know that they should be doing this. Like, why are you having these meetings? Like, well, we always had this meeting. It's tradition, right? Lack of documentation, the onboarding ramp, or went to a university where they didn't focus on like, this is how you collaborate, which is not historically an academic thing, at least in the United States where I got educated. I had teachers that would forever say, you don't get accepted to companies in groups, like to jobs, to interviews. So you should be working by yourself. And I'm like

And now I'm like, actually, you are definitely accepted for your ability to collaborate. And if you if you had practiced at that, you would have been much better and much better off and being able to actually pass the interview than knowing some rote engineering activity.

I think the importance of collaboration changes a lot company to company. Like, so I went through the Google interview process and and worked at Google for a little while. And there they like kind of famously teach computer science fundamentals. There it makes a ton of sense. And the thing they don't tell you is there's no blueprint for the job here. Like you can't go look up how someone else did it because you're working in the make cool shit factory. You're the first one. There's no one who's done this before. So

speaker-1 (10:15.542)
There's one to learn from. You just have to kind of go based on the fundamentals you know. Most companies are like, I would like another crud app, please, with login. And you're like, Okay, great. I you don't need to know what a red black tree is for that. You need to know about a different suite of things. I

Find some of the engineers that have gone to like bear organizations with you know solving challenging problems, like they have no idea what these other problems are. Like I wouldn't call them simple. Like I worked at a bunch of enterprise organizations and I come out of them, I'm like, okay, we're gonna build login and access control for this e-commerce platform. And they're like, I don't know what those words are. Like, I don't know what LDAP is, I don't know how Microsoft even authenticates, you know, Windows computers or whatever. I'm like, you've been running a monolith software in Windows on

Windows servers for almost 20 years and you have no idea how the you like user login and how you secure an endpoint works. Like, no, it's like all taken care of. Like we don't we don't focus on that. We have no idea. but if you solve linear algebra problems, yeah, that's what we do all day. Like we have a very complicated have to figure out like the order to manufacture things. And that's really important for us. And so everyone everywhere in the company knows how to do that. But when it comes to how to integrate with some system to solve what every other company, especially like in the startup world,

'cause you of course you're making new apps all the time. They have no

Yeah. And I think this is a a reality for all of us, that there are things that we ignore even if we don't know that we're ignoring them. In kind of my practical day to day, I think a lot about how people deploy code and how people review code. I think we're kind of rapidly approaching a world where people don't look at the code as much as we historically have. And you may initially think, that's terrifying. And that's terrifying.

speaker-0 (11:44.969)
Mm, yes.

speaker-1 (12:01.398)
And then the reality is is that you should already be terrified because if you look at any note modules, you're shifting a hundred thousand lines of code you've never seen before. What's two thousand more?

Of course the joke is like the heaviest mass in the in the universe is is the node modules directory, right? When we think about what code makes sense, what what we are proving in a pull request, it was do we trust what is here is going to have the right long term implication for us? And long term is in itself vague and individual specific according to whatever my personal values are, how I see them. Like maybe I care about some aspect of the company or I care about what whether I'm going to be on call or I care about

Having kudos sent to me in the next all hands meeting where someone says my name on stage, like those those different motivators could change how I particular particularly review code. And I think when it comes to the dependencies, I think it was always based on trust. Like maybe I looked up the number of stars on GitHub or the number of downloads on a package manager. And I use that to help me make some informed decision here. Is that same decision still being made today with the dependencies that you're utilizing? I don't know if that's true.

And I do think it's very different than the code that's being potentially generated by an LLM, although maybe not. May maybe fundamentally where did the LLM get trained on? It was open source packages. So in in a way, it's literally no different than the code that is in packages. So that seems like a a weird hill for someone to have to die on.

Yeah, I think the way that you talk about reviewing code is like the best version. Twenty total engineers who's had that level of intention. And one of them literally wrote a book on code reviews. Most of it is kind of performative or I think I need to be looking for this stuff or I need to say something so that people think I'm doing my job. It's not an inquisitive thing. We're not trying to learn. I think that the way you talk about application code that we're writing, we do need to be kind of vigilant and

speaker-0 (13:32.494)
You know, right? Twenty

speaker-1 (13:55.842)
Focused on the long-term maintainability aspects of it. The inverse of that is we never think about that for the open source library. We're never saying, this thing from kicker 42 is like not a maintainable piece of code. And you're like, no, it's an abstraction. It's a box. It's labeled. It has an API. I don't really care what happens inside until it shows up on my trace graph as being too slow. And then I'm grumpy.

Mistake that's probably happening more today is it was always the case where you couldn't trust the library's semantic versioning to be correct. So they upgrade from like, you know, point zero one to point two and suddenly there's a critical breaking change there. And I think in the past we would have just pinned our version and now we're getting sp I hate to say AI, you know, SAS tool.

That you have to do static analysis is saying, hey, there's some sort of vulnerability in this package. You should upgrade to this version. You're like, well, semantic version looks fine. I'm just going to merge it in. And now you've created, I mean, at worst, it's a performance issue. It could be like a subtle, hidden, silent production bug that is there and affecting millions or billions of rows in the database. But maybe we were getting luckier before where there was less versions being rolled. But you did make a really interesting point, which is I don't recall ever being taught how to do a code review. You go to a company.

And they're like, okay, hey, Warren, can you go review this code? The at one company that didn't even exist. There was no code reviews. They also didn't have source control. But at another company, it was like, I'm going to show you the list of changes by file name in an SVN commit. And before I click commit, you're just gonna like look at them and say, like, does that look okay? You go through that a couple of times, it's like, what am I doing here? Like, why am I doing this work? Like, is this benefiting someone?

And that's just like the actual technology side. On the on the flip side, like from a business standpoint, you're like, is is this even the right way to look at this code? Like, no, I want to go sit at my desk. I wanna, you know, click out these files one by one. And I think what I settled on is how I do code reviews before LLMs. Like maybe we'll get to that in a moment. Is if I wrote this code from scratch with the ticket that was created, assuming I had all the context, how would I have written it? And after I'm done writing it, how does my version compare to what was written? Did I miss things? Where

speaker-0 (16:03.074)
Things included that I had that they didn't, you know, why is that?

was very lucky to work at Google when I did. One of the people I worked with is a guy named Trevor Foucher, who wrote the book Art of Readable Code. And it's a literally a book about how to do code reviews. And unsurprising, he was on a team who had excellent code reviews. And they were writing C for the ads world. And so very high frequency, low latency things. Getting a code review from him and that team was just perfect because they were like

we don't do things this way because that leads to this problem, which we don't want because of this reason. And it was a very educational process, a dedication and an attention to the to the artifact that I've not seen a lot since then. And I don't really think that the industry supports it in the same way. Like I think we're too interested in output over necessarily outcomes. There was a good post which was like about being addicted to speed, just the fact that you did it quickly and how it's like a dopamine kind of thing.

Yeah. So we actually had a whole episode on this podcast where we talked about like what is productivity and we came to the conclusion that it's it's a feeling and not a measurable thing. Because if it were measurable, you would be like, we're doing twice as many deployments as we were doing before, or we are releasing way fewer bugs, or there are way more tickets making it through our pipeline to completion. You would just say that we have more tickets completed, we have more deployments. But instead you say, No, we have higher productivity. And the fact you're using that word as a proxy for the real thing.

Tells you that you don't actually know what you're measuring. I do think that there is a focus that is different by company. Like some companies, realistically, they don't have the same needs, right? Their whole business model is we spin up our company, we release a product, it lasts five years and then it dies. A whole company goes bankrupt, but we we got some people paid, I made some money and I'll make the next company. And if that's your model of working, it doesn't matter that you have good code, that it has high quality, because your whole reason for existing is very temporary.

speaker-0 (18:02.668)
And I think this is the default model that companies fall into without good code reviews, because it's just an extension of how the company operates. And whether or not they're committing to that end goal or they they believe that's the end goal they want, they're gonna end up getting there because you know, customers will leave because their stuff is not great, et cetera, et cetera. And maybe customers don't totally care. But I I think subconsciously they do care a little bit that the stuff that they're using works.

Yeah, I think that it's a big shift in how we run businesses. I've worked at some companies where retention was critically important. That was like the main metric to care about. If it doesn't leak, you can fill a bucket a drop at a time. other version of this is if you have a water faucet that pours enough water, it doesn't matter how big the leak is, as long as there's more water to fill. There are different phases of software development. And one version of this is I want the very best load balancer and load shedder available.

Right. Yeah. I love that analogy.

speaker-1 (18:57.024)
Okay. That's a pretty well defined problem set. Like two approaches. One is like meticulously looking at the things, and another is lots of LLM spent. And I think meticulous in that world probably pays off, similar to how like Redis, how Redis works. You're like, there are people, they pay a lot of attention to the details. Maybe they're AI assisted, but it's mostly just a lot of human thought. I think that there's another form of development where we don't even know what we're building. We're kind of exploring a solution space with software and

We're like, I don't know what these people want. These people don't know what they want. So let's generate something that's kind of like what they want and see if they like it. And if they like it, we'll do more. And if they don't like it, we'll do less.

There is this thought that there is some area where LLMs are viable to use. The thing is that my own experimentation with prototyping, we have a bunch of products that we're we're trying to come up with right now. And we've tried going down the LLM experimentation for rapid prototyping. And what we found is that if you can sort of one-shot the creation of that, like maybe screen by screen or whatever, it's fine. But if you go to a user and say, hey, is this okay? And they're like, actually, I want to see something a little bit different. If you go back and try to change it,

The whole thing crashes, like terrible. And I I I think it's because the code that the LM has generated is not maintainable even by an LLM. And so if you make a change in one part of the application, you know, let's say there's 10 pages, 10 screens, if you make a change in one of those screens, each one of those screens has its own custom implementation. And so it doesn't change consistently across everywhere. And so it's d and it's difficult to catch that that didn't happen. And so, like, unless you're going to every single one of those screens and then iterating on each one of them.

like manually. I mean and by manually I mean like having the LLM audit what's there and then making the same change. You're gonna fall into this pattern where your the standard software development strategy of like, we probably wrote this in a in a scalable maintainable way. Hopefully there's just one shared database with with all the relevant details in because it's a prototype and we don't care. And it's like, well no actually there's every single one of these data types has their own custom column or in custom screen and has to be changed. And the LM of course, you know, doesn't know that because it did you didn't feed that information in.

speaker-0 (21:06.518)
And so when you go to try to change it, you end up with serious problems. So much so that like, you know, you and then spend a lot of time trying to debug

If you want good software, you have to give context. LLMs can't make software that's good outside of having context. And it turns out neither can humans. Like if you tell the human nothing about how to do software delivery, and then you're like, hey, could you please generate me a website? You're like, yeah, I know HTML. And you're like, okay, but I want it to be interactive. And you're like, well, I also have read this book on JavaScript. And you're like, okay, but now my thing, I'd like it to change. And you're like, well, I didn't plan that.

Because I forgot that users have requests and I didn't know that yet. And so I've especially been there in my career, where I was given a requirement that I didn't question and that went very poorly for me. I've also like missed requirements, right? And so like it's nothing specific to LLMs. I

Think that you're you were onto something with this the addiction to speed. It's almost like going slower caused us to create processes in place which enabled us to make better software. But going faster sidesteps the challenges that we like the real challenges that we would have had to overcome. By doing so, we're basically creating a worse process for us to deal with. And the only solution is to actually go slower. And I I I I came to this conclusion after watching something completely unrelated. Rory Sutherland, who has

great conference speaker. He talks about like actually wanting to go slower. And the analogy he used is like if you're driving in a car, and I hate using this analogy since I moved to Switzerland because very few people drive cars. And you you're in a car and you're going like a hundred kilometers an hour and you're like, okay, but what if we could go faster and go like 110 kilometers per hour? And like it turns out that unless you're going like a very, very short distance or a very, very long distance, going faster doesn't change anything. It's like you're

speaker-0 (22:57.102)
Creating a a larger risk by driving faster to get like, you know, let's say across a large landmass, like across Europe or across the United States. And going that much faster on average definitely gets you there sooner. But it's like you get there sooner by like a few hours, maybe. And the risk of, you know, serious injury increases like exponentially. and of course, if you're like driving from here to the the corner gas station.

Like going a hundred kilometers an hour to begin with is like super dictarist. But it's the same thing. And so like unless you're in this weird area where they're under a rush to actually make it happen. Like you have some deadline. Going faster isn't necessarily the right thing. And whereas historically we would have gone slower because that's just what we have. Going faster doesn't fix any of those problems. It just increases the risk.

Yeah. I think there's also like this distinction on speed. Like do you need immediate speed or long term speed?

That is definitely the mindset, right? historically it was you go and work at a company for a very long time, you retire, and everything's fine. And I feel like there has been this shift downwards where getting things like especially with the social media and certain applications out there that really encourage the very clickety or or twitchy experience where you very quickly transition from one topic to another one, very ADD focused, like next idea, next idea.

I think it is all well aligned with this transitioning down away from the safe foundation to doing things as fast as possible. The the addiction with with speed, I think this may be just a human fallacy, just a a foible that that humanity has. And so it does seem like it's better. And I think historically, we're not great at long term trade off. Like there's very few people, I think, that not only are able to see the long term, but actually be like, Hey, look.

speaker-0 (24:39.266)
Like if we do this, there's a real consequence, even five years down the road, or even one year down the road, or one month down the road. Like it's amazing how many conversations I've seen where just one month later is just immediately going to be an issue that no one considered, that no one even talked about for one moment. I just think historically we're just not great at that.

You're saying there's not a solution for people's inability to understand and articulate risk? I think the security people probably deal with this the most. Which is like, what's the likelihood that someone guesses the CEO's password? You're like, probably pretty low. How bad is it? Well, it's pretty bad.

on the long term scale.

speaker-0 (25:13.826)
To look at like the expected value of an incident, like the risk, the right likelihood, the probability of its happening, and the impact, and do that multiplication to get the expected value. There's two challenges with that, I think. Fundamentally, the first one is that you actually sort of have to have those numbers. You have to be you have to understand that you can do this analysis. And I feel like a lot of companies aren't even prepared to take this into consideration in the first place. And the second one is there's other terms in this equation which aren't just like the straight loss, like.

Why did you why does it cost 10 million? Is it like an impact to the brand? Is it a fine? A lot of companies don't even fall into those categories where having an incident is like public attention that just increases. Like having an incident, the ROI is positive. Return on incident equals positive, which is like an absurd thing to say. But companies actually can use this successfully. And I think we are potentially seeing that in the age of LLMs where companies who should

under no circumstance get any praise for doing a thing. Now are getting public recognition for it. Like there's one company that apparently is hacking other companies. And instead of saying like they need to be regulated into the ground or be heavily fined, everyone is standing up and cheering like, how is this like this is so amazing that they can do this. And I'm like, I don't think you understand how this should work.

Yeah, I've I think I've heard that described as gloating about felonies. I had some I had a friend of mine who asked me, do I trust LLMs to ship to production with zero human involvement? My answer, 100% yes. For my blog, I would trust an LLM hundreds of dollars given the setup I have of risk. Not thousands. Certainly not bet my company on. Hundreds of dollars of downside? Yeah, absolutely. Ship that code. But it's not twenty thousand dollars.

Yes.

speaker-0 (27:01.634)
Which which says I think more about you and your blog than about the LLMs. Because there is this thing where like some companies are are are betting on this. Like, yeah, you know what? What if it just costs us a thousand or ten thousand? I still have the rest of our bank account, you know, to sponsor whatever happens. It's it's a experiment, quote unquote. Not really an experiment because A, you already know how it it comes out, and B, you're not measuring anything. So you're not really deciding whether or not you want to continue this experiment. You're just it's more of a speculation. And maybe you'll never find out the problem with it.

Or it will be super successful like getting VC funding. And why I say it says a lot about you and your blog is because I I think it's that the confidence that you could write a one shot or you know, put that in the context that we talked about before to be able to provide effectively says how great of a of an engineer or how much you trust your ability to interact with it to have the right outcome without potentially reviewing it. You know, I see lots of people out there that don't have the

extensive understanding of how to communicate well. And I think communication's a huge part of interacting with LMs. Like you know what's supposed to be released. You know what that code is supposed to be look look like. You know how to talk about that. Then I feel like you're eliminating all of the or most of the failure modes that come out, which are like randomly choosing things, building it the wrong way, or releasing the wrong feature with the wrong technology.

And so I've been thinking about what might it look like if you were not allowed to look at the code. You're only allowed to view it from the outside in, like a car mechanic. You don't get to just see how the engine block was made. It you have to assume that it's made of steel and that it is what it is. What sort of tools could we use and how could we think about that problem in order to understand? Production telemetry is an example. We can look at the production telemetry and see what happened. We could look at the test run reports and say,

15 unit tests pass. But even as we talk about like large scale production systems, there are things we can encode in the process from LLM through the software generation process to the review process through deploy and then in production operations that de-risk the failure modes. So let's take the example of Amazon checkout process. What are the ways that Amazon checkout process could break?

speaker-1 (29:08.396)
Well, one of them is I click checkout and it just errors. Okay, cool. Could we have a robot click checkout every five minutes? Yeah, cool. Okay, so we've covered that. So now you're limited to every f like five minutes of dollar loss. Okay, that's not great. How could we reduce that further? Well, we could roll it out to a percentage of traffic and test it there first. Okay, we could roll it out to just one request and test my one request and make sure that works before we roll it out further. Okay, great. We have tools for all of that. We've had them for years. What else could be broken? sales tax collection could be wrong.

Okay, what's the best tool we have to validate sales tax collection? You're like unit tests. Okay, cool. Do we have those? Do we have a process by which we assert that we have those tests through coverage? Things like that. Yes, we have that. Okay, what else could go wrong? Well, the LLM could generate wrong features. Like, okay, what's a wrong feature? What does that mean? Well, it doesn't do the thing I expected to. Okay, how would you know? Okay, well, we would know if when an engineer demoed it to me it didn't look how I wanted. Okay, could we have an LLM generate a demo video for you?

Yeah, we could do that. Okay, cool. What else could go wrong? And so through this process of kind of continual introspection about the artifact as it makes its way to production, we can de-risk that process in such a way that we have pretty good confidence that the thing doesn't suck. Today we underinvest in that deploy pipeline and they put what I would say is unearned confidence in human code review. Mm-hmm. Because like, when was the last time a human actually caught your bug? Like

I ran a survey at my last company and it was like it's not that recent.

I think there is definitely a huge struggle to basically do anything. We see a lot of companies trying to solve the AI SRE thing. Like you will ship bad code to production and now we have tools in place to sort of automatically create pull requests that find those bugs and push it up and, you know, automatically merge and fix that. And my answer has always been, sure, you can have that. It may be appropriate for some companies, but don't you realize you're

speaker-0 (31:03.478)
you gotta stop the bad coach from entering production in the first place. Imagine you had a funnel where software is being thrown at the funnel and it's how can you filter it down to only getting the stuff that makes sense?

My current thinking is that it is the job of a human to resolve ambiguity and provide judgment when that's needed. Yeah. Like is this feature good enough to show our users is a human question. What should we build is largely a human decision. And I think it starts looking a lot more like product work and a lot less about specifically where does the semicolon go? Or should this be an abstract base class or should we use composition? To some degree, who cares?

It's hard as a like a experienced engineer seeing some of the code that's being generated and not having strong opinions about what's there. Early on I tried to be like, Yeah, you know what? I'm gonna try the vibe coding strategy where I don't look at it and I just try to get it to fix things. But over time I realized that a lot of the sort of issues that I was seeing were the result of mistakes that were made.

during the generation process and could only have been fixed by having a more intentioned approach to as the code is being generated, steering it better, because after the fact it's now like permanently affecting the context in a negative way. It it's completely poisoned it.

How do you as a human know if the rate code's being generated?

speaker-0 (32:17.72)
See, this is the interesting thing. This is why I think I am both a great and terrible software engineer. I'll say that I have opinions on the aesthetic beauty of some code. And the closer to that aesthetic outcome that I I'm hoping to have or that I see as pretty tells me whether or not it's good.

Yeah, I think that that there's a difference between software as an object of art and software as a utilitarian thing. Now, I happen to own clothes like the shirt I'm wearing that is largely utilitarian. No one cared about the weave of this fa this cotton fabric, the structure of it. Someone then put logos on it and added an artistic layer on top. I also own sweaters that were handmade by someone who really cared.

About that sweater. And those are different objects that cost different amounts and serve different purposes. I think similarly, we can have software that's not made so that I feel good as an author. In the same way that whoever made this shirt didn't feel good about the process. That's just that work. That's the job. And then there's a different scale where you're like, No, no, I I'm gonna care and I'm going to pull out my loom and like weave this thing. And like that's a different hobby. I I

With what we were going with here. And I I think the problem is that that view, I think, may be shared amongst a lot of people, but they're not taking it to its natural conclusion. Which for me is there only exists two types of companies that make software: giant monolith software production facilities that have optimized the very low quality but can generate tons of stuff. And those companies that believe that the software they're writing is a competitive advantage and are writing the highest quality.

artisanal software outward artifacts that you could imagine. And if there is no in-between here, the thought that I have that goes along with that is if you are just someone that comes into the job and you are clicking a button to generate the code and you click that button over and over again, or to quote Dorota, pull the hot garbage slot machine lever one more time and see and see what comes out, you're basically saying I don't matter in this process. My company doesn't care about

speaker-0 (34:28.545)
the quality of what comes out, I'm not necessary here. And and worse, you are losing your skills in writing better quality software over time. You're literally building the thing in a way to replace you. So I think you need to be conscious of whether or not you're going towards the I'm trying to be the best software engineer, build the best, highest quality software that hits whatever

you know, very unique luxury standpoint, or you're going toward the direction of software factory output as much as you can job, even though they're not doing differentiated work from a com corporate standpoint.

Think through the project that you have that you felt the most attached to and you felt s the proudest about. That it most embodies for you this artistic representation of give a shit. Do you have something like in mind roughly?

See, I don't know if this argument works on me because I just hate everything I've ever made. It's literally the worst software that has ever been generated.

Yeah, okay. So like if you're gonna ship crap anyway, why don't you just ship crap faster? Even in the most artistically beautiful thing, you didn't write your own date time library. Like there are times when it doesn't make sense to write it yourself. And I'm not saying that every every company needs to do one hundred percent of all things through from the factory. for sure. And then someone printed the design on. Right. So there's a version of this where

speaker-0 (35:26.134)
Yeah,

speaker-1 (35:50.702)
To the creation of this product was part factory, part human, like bringing w the human pieces to it. So it's not just a generic black shirt.

If the step that you're providing as value is doing the automation on top, then you're not creating any sort of competitive advantage amongst every other company that is literally doing the thing. And I think, you know, maybe there's still an argument here because there's still tons of companies out there that rent a fabric press for shirts, right? And another company rents that thing. They go and they just use the machine, and that's the whole job. There is no differentiating aspect there. The one difference with the physical world process, though, is locality. They can sell that shirt in a

you know, in an area that's around wherever the business is is centered, right? They have a competitive advantage on location versus everyone else in the world that can do it. Now, obviously your neighbor can compete with you by doing the same thing. And so there has to be other strategies you you bring to play with. Like you have a a commitment from your customers or from your suppliers and where you get the shirts from, you know, you know someone who's willing to sell you them a little bit cheaper or

You have lock in because you're buying so many shirts from your your shirt manufacturer that you get a good deal and then you can sell them at a lower cost in order to compete with you, someone else would have to come in and take a hit on how much they're selling the shirts for in order to actually have a lower until they put you out of business and they can raise the price again. And so, you know, you're playing a fundamentally different game there. But I think fundamentally if your job is, you know, you take the lever and push it down to, you know, s print out the stamp on the shirt, like your job is fundamentally, you know, replaceable.

Convert this thing that the product person says into working code, please. And there's actual judgment that comes. Do we have a monorepo or polyrepo set up? Are we going to use a SQL database or a non-SQL database? How important is that database decision today versus five years from now? And how easily could we reverse it? Is like now the time to make that change? I also edit a lot of YAML. And so like of those, I'd like to get rid of the YAML editing. And and I'd like to stay more on the like, what are we doing? Why are we doing it? And what are the implications?

speaker-0 (37:47.182)
Are you optimistic that the trajectory of your career, you know, maybe in the next five, ten years is towards the direction where you'll be able to make more of those decisions and write less YAML or write more YAML and have L LMs or other decision based AI out there to take over the responsibility of figuring out what is the appropriate database and to deliver the b relevant business value?

I'm more likely to move the up the stack towards intent than down the stack into edit.

On your path to becoming a principal engineer and obviously the way you're you're talking about, no matter what it is, there is a very deliberate nature to how you're approaching that and what problem you're trying to solve and which tool you're you know you're pulling in to make that happen. How specifically would you think that the addition of utilizing L LMs has has changed the process of what you're doing or in the types of problems that you're solving?

Types of problems that I'm solving are identical. I'm I'm still having the same conversation that I had with those people at eBay ten years later. And that didn't get harder or easier with LLMs. It became more important because we have a lot more pull requests showing up. The questions we can ask and the way that we communicate is more important, but also there's a an asymmetry to effort versus output that is really dangerous for us that we need to pay attention to because we don't want to denial a service our coworkers.

I have this theory that the amount of value that the thing is being produced is still proportional with a ceiling of the amount of human timeslash effort that's put in. So as a human, you put in less effort or less time into it, it's going to actually decrease the quality or the value of the thing that comes out. I mean, obviously I have no no no evidence for this whatsoever. But it does feel like this sort of thing where if you aren't paying attention to what you're doing and you are just passing outputs from LM LM.

speaker-0 (39:32.44)
You're not putting in a lot of effort and therefore the output product can't can't be good. Can't be good enough.

How that usually works. where you spend six hours trying to automate something that takes one hour a year.

and maybe the problem is when you say use an L L it's like a proportion of value, like the amount of time that you're spent to making yourself to create something of durable value, does has that changed?

Yeah, it's definitely changed. I can have an L L write me an end to end test with playwright with no additional L L investment and I I maintain value out of that.

Your relative confidence level on test writing that LMs have, like low, medium, high. Interesting. Do you know what makes the difference?

speaker-1 (40:08.748)
Alternatingly low and high.

Sometimes I'm like, Do I do I think that an LLM could write a good suite of unit tests with the aid of mutation testing that would accurately test an algorithm? Hundred percent. Do I think that the L LM can ensure my UI is functional through clicking? No. No, I don't. and so like at different levels it'll be different levels of good.

I like the idea that there's such a high disparity here between the two sides of the spectrum that it flips between depending on really the the type of focus task. But I I do have seen that there's this weird thing where maybe it's great at sort of first level design and UI generation, much more than I am. But when it comes to validation of it, I do agree that there is this it's not as great. Whereas it's much better at following a spec and generating code and the tests for that. And so I I'm not I'm not sure where I'm going with that. It's just sort of an interesting learning.

So at this point, Justin, I think we should move over to to picks. So what did you what did you bring for the audience today?

Yeah. so there's a genre of thing that I have found through TikTok. and it is called Isakai. Isakai is a genre of literature w that's based on this premise. You die, you are reincarnated, and you are reincarnated into a video game. Now you have like a stats screen that you can look at and you have skills and they get better as you like invest. And I didn't realize that this was a genre. I picked up this book called Dead World Isakai. It's a trilogy.

speaker-1 (41:43.032)
Of a guy who is Isakai into a world that has no inhabitants. It has no life. It's just this dead world. And he's there. He's like, okay, now what do I do? And it's him, it's like a man versus machine kind of story. And Isakai is, I think, really interesting. And that that's my pick, because I don't think many people ha know about it. And they get real off the rails. one of the examples is someone who is isekai into a pig. There's like this anim anime about it.

and I've seen like little s snippets of it from TikTok and it's I need to get my s like stamina up, okay. So I need to like drink this milk and eat this food and become a big pig. And they're taking me somewhere to the to the market. I wonder what that's about. I need to escape and like it's so it's this this video game anime thing. It's fascinating. I don't

Know if I want to ruin this for you. it's such a huge anime genre, so much so that the thing I don't know the pig one is, is like a parody of other Isekais on how ridiculous they are. The most parody one is that the main character got reborn into a vet being a vending machine. That's that's the whole concept. I don't think it's good, but it's a thing that someone can enjoy. and there's like multiple seasons of that. For like I don't know what you do after the first one, honestly.

But that goes on for a while. There there are quite some things. And you know what the interesting thing is like it is it has definitely grown and it's grown so ridiculously that now the titles for those have also sort of gotten quite ridiculous. Where like I'm sure the title for the vending machine one like I got reincarnated into being a vending machine in my next life or something. Like that's it. Like that's it's not even trying to come up with a like a clever, like a real name for it like you would see in in Western media. I think the most canonical one is probably sort sort art online.

So these guys isn't necessarily limited to just video games. can be just anything in a new world, which I think there's a huge percentage of them that are like I have a stat screen that no one else has for some reason. Yeah, okay. Well that what a pick. I can I can guarantee you that no one has no one has picked that so far on the podcast. So you've got something unique there. Yeah, my pick. always a challenge. What I'm going with is this research that was published earlier this year on

speaker-1 (43:45.624)
Fun, fun little book.

speaker-0 (43:54.718)
The total sum of domain name registrations, it's like 35 pages, so it's it's it's a little bit long, but it has quite some interesting stats in it. And it turns out that like 10% of all domain names that are registered are malicious in nature. Like someone registered it purely for malicious intent. It's the inter IEL Consulting Group. I don't know if there's like a title for this report other than like malicious registrations in the domain market. There'll be a link in the description of the episode. There's some interesting things in here, like

Malicious actors just like burning domains left and right. Like there are so many domains that just don't get renewed because they were only registered for malicious behavior. And after after they're used, they obviously burned every every denialist says no, you can't use this domain because it's associated with malicious activity. And so over time you can start creating some stats about where these domains are coming from. There are some registrars that are just way more malicious than others, and not necessarily the registrar themselves, but the domains that are registered through that registrar.

Are more likely to be for malicious actors than not. And certain TLDs like the top-level domain, like dot coms and whatever, could also be associated with malicious activity. And I just find it really interesting that there are some domains that are just known for how likely the a bad actor is using them. So the things like the.xyz domains or dot top or dot info have a disproportionately huge number of actors that are using them for malicious purposes, more so than anything else.

And I just I don't know what to make of this research. Yeah.

I mean those aren't free. I'm very curious. It's gotta be worth it.

speaker-0 (45:26.774)
Obviously, if you're gonna do something, I mean they're like one to a hundred dollars per domain. So if you're gonna go out on a some sort of fishing expedition, it's probably worth it to buy one. And I the question I'd actually have is I wonder what the like the half life is for using the domain. Like obviously you get it for a year by renewal, but I wonder if it stops being used like after one month or two months, like once you run the campaign already. And so there's domains floating around there. It's a good reminder that there are so many domains that are being registered and by certain

TLDs that you should be careful about which TLDs you're picking because the whole TLD might be blocked. Like for us, I know in one of our products, we automatically block any XYZ usage. I know we're blocking it at corporate email receiving level. Like if you have an XYZ domain, your email is not going to get to us just because statistically it's really bad. And if you're buying a new domain, you really need to check where who's had it before because it's highly likely that someone's picked it up, burned it.

And now you're basically paying for this experience of using this new domain. Yep.

I have that same concern. I my my domain is dot S, which is Montesarat, which is usually not an issue but is occasionally an issue.

I don't think I've ever seen a dot MS usage there. So there's an interesting thing with the G TLDs, the generic TLDs, the ones that aren't assigned to INA set for for countries that I'm always worried about. but that's that's definitely another t another topic that we're not gonna get into here. So thank you, Justin, for for coming on the show. I've I've enjoyed your your hot takes in the world of LLM coding and pull requests. and thanks for all the viewers for tuning into this episode and I hope to see everyone back again next.

speaker-1 (47:02.318)
Thanks having me.