speaker-0 (00:07.906)
Welcome back to Adventures in DevOps. In October, we brought in John Papa to talk about IDE's co-pilot and critical thinking's place in software development lifecycle. It was a fantastic episode, but we only got so far. To extend on the unanswered questions, we're diving deeper today by questioning hopefully everything. And to do that, I brought in Tudor Girba, CEO of Think. Welcome to the show.

speaker-1 (00:30.878)
nice to meet you again and thank you for having me.

speaker-0 (00:33.25)
We chatted before and you got into a lot of philosophical aspects of the work we do, knowledge work, not necessarily, you know, hands on technology and it was it was just something that I wanted to share with

speaker-1 (00:44.29)
Yes, I'm I'm glad you I'm glad you did.

speaker-0 (00:46.2)
Humans short term focused historically and I think even more recently have a focus of switching to LLM generated software development. And I think realistically have understanding the philosophy and strategy behind how we do software development and really any technical thing is important.

speaker-1 (01:03.798)
Shouldn't have the thing all along. Let's put things in perspective. We are building a whole new world on top of this new kind of platform. There's a reasonable amount of responsibility that should come with that. And so should it not be expected of us to spend some reasonable amount of time to understand what is it that we do, to understand the properties of the you know, the effects of our work. But on the other hand

If you look at what developers actually do, they spend a small portion of their time actually building anything. The vast majority of the time is being spent on reading. Now, today, this is much more accepted. People find themselves, I'm reading because the LLM is doing all the writing, and the only thing I'm doing, I'm reading software engineering has fundamentally changed. But the thing is, I I I got to talk with

thousands of developers, well before there were any LLMs even inside. And I asked them, Do you agree that you spend more than fifty percent of your time reading code? And basically developers were saying yes. And this is a fascinating thing because 50% or more is misses is the single largest class we have. There's nothing larger than that. Right? And then I I was asking them another question. When was the last time you talked about how you read code? It's a very simple question. What do you mean?

How to read code. Given that this is the single largest expense we have, if we don't talk about it, this is means this is not explicit, but if it's not explicit, it has never been optimized. The single largest cost. So we end up being an industry that starts building the whole new world without actually understanding how it is that we're doing things.

speaker-0 (02:50.958)
Companies are in different places on the spectrum though. Like I've definitely seen in some enterprise organizations lots of meetings dedicated to not just what color should this button be, but what should the hierarchy of our thousand microservice monolith directory or repo should be? Like should we have a folder called tools and a folder called utilities and a folder called, I don't know, like address management? And those things can be referenced each other. And I like I've seen a lot of conversations wasted.

Trying to figure out, like seeing that there is a problem with how we've managed our source code and just only slightly grasping that something needs to be different, but those conversations usually don't end well. Like they're based off of a couple of different people, usually with an architect title, discussing or debating on what the optimal directory structure is, but not really have any first principles to get to a reasonable place. It's based off of opinions and maybe you could do some sort of experimentation. maybe you

you put forward hypothesis that an issue is quickly finding what tools are available in order to utilize them. And then from there you can do some sort of research. But I find that very rarely happens in practice.

speaker-1 (04:02.466)
Yeah, so I I I also find but I think at least this conversation, right? So what's the structure of the project is it does that does tend to happen. where it starts to fall back is when people agree and then different projects will still look differently. So that their their their directory structure will still not be uniform across even after they make that decision. And the question is why is that? W what do we do when we want to ensure that the system behaves the way we expect it?

behave. In fact, hopefully today, most organizations have internalized the idea that you shouldn't release if there's even one test that is red, we automated the answering of functional questions about our systems. Is this is this particular functionality working the way we expect it to work? This model works exactly because it is made out of little tiny tools. We call them tests.

What is our what are those tests? They're just analysis tool. They take lots of lots of information and they transform into red and green types of signals. And the totality of them helps us build a model of the system that allows us to steer much larger systems, much more complicated types of functionality, much more certainly and much more comfortably.

speaker-0 (05:17.526)
We've created an invention as far as the software development lifecycle goes of how we build and manage sort of the quality gate. And if it really is a mechanism that it sits on top of the foundational layer of the source code we write, we should be utilizing it as a mechanism for driving understanding about what the service or product or platform actually does. But I find in practice, very rarely are tests the driving factor for understanding.

of what's actually happening. It's always like a after effect. Like it's the way you validate what is there is working in practice. I've very few companies are actually using their test be like, how does this function work? let me go to the unit test and I'll look at the test and the test will tell me how this function works. Well, I don't think that's usually the process, which is it seems like in a way completely unfortunate. Because we if if our tests are really built up, if TDD is really what a company is doing

Tests are a priceless opportunity to use as an endpoint or an entry point into the understanding process that's being completely eliminated or ignored.

speaker-1 (06:23.672)
When you go with you make it so there's a test which is red, then you go and you fix it. And at some point people say, you you you clean up. But nobody says how do you go from the red test to fixing it? And nobody r says I how do you go from this solution here to the cleaned up solution? Yeah. Whatever that is. Those things are left to the imagination of the of the reader, which is of course it's development. So if you go back to why do people read code?

is because they want to understand enough to make a choice. The because reality everybody knows that reality in this in the system. it's not in documentation, it's not in people's heads. It's also not in AI summaries, by the way. But

speaker-0 (07:05.836)
I I'm trying to figure out how does the corollary of how we built tests apply to the change that we should make to our software development process.

speaker-1 (07:14.99)
So answering any question about the system should be intermediated through deterministic contextual tools. Let's go back to your problem with what's the directory structure of my system. So imagine I take the directory structure, which is basically gonna be a tree, and I imagine I will be building I can build a grammar. And then I can take that grammar and I run it against all the all the directories that I have in my in my system, and therefore I'm gonna have a test.

But this is this is not gonna be a str a functional test, gonna be structural test. There are gonna be other kinds of questions for which we do not know the answer upfront. a case where we that we worked on where there was a a telecom company having a data pipeline. and they their problem was that they worked on it for three years and they wanted to optimize it. And three years back, it took about one day from the moment they had the data until the moment

Some offers were going to lots of customers that they were having. This was their main marketing data pipeline. And three years later, so this was like a big problem because this was a business-to-consumer business, and you know, reacting, having a one-day reaction time from an event to an offer, it was very, very long. And the board knew about this. And the CIO demanded this must be much faster. three years later, somebody measured, it took exactly one day.

From the moment they had the data until the moment they were sending offers. And so now the question was, what what happened in three years? Now this is like a different kind of question. It's a question for which we do not know the answer. we might have potential hypotheses. So in their case, when we asked them, we say, Well, what's your hypothesis? Well, we probably are producing too much data and this is why it takes one day. But we don't know why this has not been optimized. So like where did the effort go?

Still an unknown. So we started with saying, so what is your data pipeline? And they drew for it. We say, draw it on the whiteboard. They didn't have documentation for it. So they drew it on they they drew it on the whiteboard. sounds, you know, like sounds funny, but at least they were honest. Because there are others that say that they have documentation and it's still wrong. So in this case, in this case, right, they have they draw for us the the the diagram on the whiteboard and it was they drew for us four boxes.

speaker-1 (09:41.1)
So now what do we do? We make it the job of the system to give us back this picture, but this time through a tool that directly reads it from the system. So we start to build this one. Now in the process, we discovered it in the first box. They had a homegrown domain-specific language, which was actually implemented in Excel. So some like was a very elaborate Excel spreadsheet in which people were describing.

their transformations, which were then transpired into Oracle instructions. In this process, we noticed, yeah, the output of the first box doesn't seem to quite match the input of the second box. It was like in two weeks into this project. And we went back into to to them and said, well, are you sure you don't have something in between the first box and the second and the second box? And they discovered the whole system.

speaker-0 (10:30.606)
How has that happened in practice? Like what like what world

speaker-1 (10:34.254)
Because we thought, this is this was so rich when we saw that. It was really funny, actually. It was really nice for them to accept that yes, right, our understanding was really poor. But I've been having conversation with lots of CTOs and they or CIOs and they tell me, yeah, I can tell you a story about this. How how is that normality? Because this shouldn't be normal. but unfortunately, so maybe at that level, maybe it's not the case everywhere, hopefully.

But if you go one, two levels deeper, you will find immediately where people tell you that the system is something, but the system is actually something else. And what this leads to is a discrepancy between our mental model and what the system is. Our mental model and reality. And so you we end we can in this situation, we can end up easily, you know, dancing around the fire to make rain happen. Right? So and then and then we will have debates whether or not

We should be dancing clockwise or counterclockwise and there should be ev then there will be evidence gathered as to why clockwise dancing is actually much better than and the counterclockwise to make rain happen. And so and this is unfortunately the kind of thing that we are in that we find ourselves often today. Because why? Because we just don't talk about how do we go from a question about the system to getting the answer to the system.

speaker-0 (11:56.334)
So I feel like in theory it like I agree. It there's should never happen in practice. But then I think about actual organizations where what you've built, and even more I think is true today, that you have something that's just unmaintainable because the people who have built it and had the mental model even in their heads, assuming you could maintain it, have long left the company. And often these challenges or these questions that want to be answered by the business of like, why does it take twenty-four hours or a week or a month to

close the sales cycle is trying to be driven by someone that's outside of even engineering or wherever the problem is being set up. And then they're applying that process to the rest of the organization to give them back an answer. And the individual engineering teams or organizations or what have you don't have the resources or like time or understanding of the system to actually give a a a correct answer. And so they're of course guessing in some regard.

and so I sort of like in practice I can see this as being a reasonable n and almost dif definitive and conclusion. Every single system we build, if it needs to be in a way self-documenting, because no one on the outside will ever have a correct model. It was there's whatever the saying is, right? Like all models are wrong, but some are useful. but correct model is obviously more useful than an incorrect one.

I I'm surprised that for answering questions, more organizations don't actually jump in and do the analysis that's necessary. I feel like in the sales world or at least in marketing, we see lots of ascribing to the benefits of making an A B test on a website and seeing how many users click by and whether or not that actually represents the reality correctly is a whole other question. But we do see them actually even appro like approximating the scientific process in some regards.

speaker-1 (13:48.696)
data scientist getting closer to where business decisions happen so that when people have a question, a tool is being created specifically for that question so that an answer then can be used afterwards. So those, you know, A B tests, that's a little tiny experiment that requires a little tiny tool that re that will project whatever the answer is so that the people, the stakeholder, the ones that has the question and knows what to do with the information can make the choice.

It doesn't solve the whole problem. It's not like a magic solution for everything. It just removes the largest amount of noise. And in software engineering right now, because we choose not to talk about how we read code, we people miss this enormous opportunity to create value that is much, much cheaper than people believe it is.

speaker-0 (14:38.496)
Is there a distinction between an expert coming in and being able to apply and know which are the right tools to utilize to find the answer? And a fallacy that the fundamental sort of flaw is the approach that is being taken. And like I I can imagine, let's say we're talking about the performance of a system. I can go out and find tools that are built on your favorite package manager, PyPy or NPM, NuGet, et cetera.

And download one of those, install it into my repository, or pay for a open source or SaaS solution that goes and monitors and instruments my project, business solution, whatever service, and get an answer and then make decisions based off of that. Is it that the wrong tools exist? Is it that the right tools exist but the we don't know how to apply them? Or is there some fundamental philosophy about how we're building our systems that's flawed, that needs to change?

speaker-1 (15:29.368)
We definitely do not have a shortage of tools. Right, so that that's one observation. The other observation is this idea of developers reading code for the largest part of their work is is not new. And the oldest reference to the subject goes back when about the nineteen seventy-nine that I could find. So we have like half a half a century. So the amount of tools grew, but the problem didn't get solved. It's also obvious that the tools we've built are not solving the problem because we can still observe the effect.

So then the question is why not? We were building the wrong kinds of tools. And by this I mean, yes, you can go and download a tool from the web, but if you use it as is, then it's like downloading a test from the web and using it as is to see to to ask it about the system that you have. Nobody's doing this because it makes no sense.

Right, everybody will say, Yes, I will be downloading an engine in which I will be writing my own tests. Why? Because nobody knows what makes my my system valuable. It is like this. So if you then inverse the situation and you internalize the skill of building the tool. So this whole idea is based on on the premise that building tools is expensive. When this idea of testing came first, pe I I remember p talking with people in the two thousand

2000, 2001 and we yeah, but you cannot apply this one to UI. Well it turns out you can. but for real time this for real time systems you cannot you cannot apply any kind of testing. Well it turns out that you can. So the and and today we're saying but this is impossible. How can you possibly be build a system that parses a new language very quickly? How can you possibly build an architecture diagram very quickly that is highly specific to exactly our protocols and so on and so forth.

very quickly.

speaker-0 (17:25.134)
And you sort of just threw this out there that we have the tools available, but they're maybe not the right tools. But it's really the cost of implementation or utilizing those, which is problematic. And I think you made a reference to this in your book as well, that realistically, if tools were negligible to generate and create to solve any sort of problem, you would spend all your time j first generating the appropriate tool to extract the data that you want, and then just get it generated and then get the data.

but we're we're we haven't been doing that. And I I'm wondering what is the like why did that happen? Is it because the primitives we have, the programming languages that we have available? Is it about the interfaces that the hardware exposes us? Are are they just all wrong in some way? And we need to push back the learnings that we've had in the last twenty to thirty years to a lower layer so that the abstractions that we're actually given, the interfaces that are actually available to us contain the the right pieces of information, or is there some other fundamental problem that we haven't been addressing?

speaker-1 (18:23.576)
There hasn't been a shortage of of exploiting or exploring what languages could be. And for sure, there are some languages that are more inviting than others for you know for scrutiny or make it make things more available. There is no way to perceive anything inside a system, inside a computer, except through a some sort of an interface. Which basically means that tools are essential. For tools to be effective, we they have to be sp contextualized.

to the problem that we have. So it's not just some generic tool that we're downloading but from somewhere, but but something that actually matches our questions about the about the system at this point in time.

speaker-0 (19:04.76)
I think there are lots of engineers out there that are every single organization that that say, yeah, you know what, I will just write I have a whole repository of scripts that I I I I keep and maintain.

to to solve a particular problem or you know what, I'll just generate a CLI to to do this thing. And the engineer the most common strategy I think is like there's an engineer working at a company. They feel like they don't have the time or space to solve the real problem that that company is is facing. So they leave, they start their own quote unquote company. They make a CLI, it does something, then they release it and then they wonder why they're not getting any users. But I think they are making tools in some regard. I I I I don't know that they're just not doing that.

speaker-1 (19:44.982)
for sure, people are making tools. They're just not making enough of them.

speaker-0 (19:49.765)
More software. We should have more software.

speaker-1 (19:52.046)
Absolutely. No, but it's like it's like with it it's again, if you go back to tests, there was a time when people were writing no tests and people say, I'm gonna write some tests and say, Well, how many do you have? Fifty. Yeah. They you probably don't yet have it you're don't you're not yet testing your system, right? So in two thousand and two there was in when the when T D D by example by Ken Beck came was published, how many tests do you need and does this actually s this idea does it scale to

to large system. And in there, towards the end of the book, there is one little case study which says there was this organization which is called LifeWare. And they they were having 4,000 tests. And not only they had 4,000 tests, they had a 500,000 lines of code system. And half of these 450, 500,000 lines of code systems were test code. Half of it.

And this was the argument as to why they were saying this was the largest test suite that Ken Beck has ever seen at this at at the time. And this was basically the argument, this is why they knew that it actually does skate. Now, we actually have the l the the the pleasure and the privilege actually to work with Lifeware since a few years. And today they have a hundred seventy thousand tests. So in two thousand and two their four thousand tests ran in twenty minutes, which was

a remarkable achievement at the time. and today they have 170,000 tests and they run in like ten minutes. They have thirty-five million lines of code system and half of that code is test code. Yes, we didn't have enough tests in back in the two thousand. So they are embarked on a new exponential that they are basically exploiting. And yes, we do not have enough of those tools.

to navigate through our system. What we're saying is in a reasonable system you should have thousands at least of things that are being shown to you.

speaker-0 (21:57.644)
I and I'm wondering if we're doing something unique that not everyone's doing and this sort of points to what you're actually suggesting that companies can do to be effective, or org engineering organizations or individual engineers to be effective, or whether or not there's still a flaw in our approach. So I think there's a spectrum on one side, and I think Simon Wardly is to actually get all the credit for thinking about this, like which side of the spectrum are you on? Are you building something totally bespoke or is it a commodity Wardly map, right?

and I I feel like for software engineering, there's usually this rule of three. If you do a thing multiple times, you know, you should think about automating it or refactoring it in some regard. And so when we have a bug or a problem and we have logs and we have to jump in and do something manual, we think about well, why did this have to be manual? Can we build a tool to expose the answer to the question specifically? And so we have this escalation process in how we write software.

We start with logs. And if you have an error in your logs, like an all reference exception or a 500, you you say, well, why did this happen? And if you don't have the answer to that question, the first step is to start logging additional stuff. But I think this is where a lot of other companies get stuck or misguided, is that they end up stopping there. But we take the next approach. Inside where those error conditions are, we will wrap that log statement with database queries or other traces where well

improve the log statement in our source code with that additional data. So the next time that log shows up, it says, here's the problem. But also you may want to know this was the customer that was impacted. This is how many users they have. This is the sort of request they were making. Here's the other request they made for the last five minutes. And here's another query that you can automatically run if you're the unfortunate engineer who got woken up at 2 a.m. in the morning and has to actually investigate this, just click this link. It will open a UI and automatically have the query execute.

And then the process continues. you know, what's at the result of that query? Maybe more data, et cetera, et cetera. we call this sort of like dynamic UIs or dynamic queries for on-call support. And I'm wondering if that is well aligned with what you're thinking about. what I'll say is that when your source code is more constrained in the with a bounded context, the and a smaller size.

speaker-1 (24:10.222)
Absolutely.

speaker-0 (24:20.182)
You don't need to spend as much time thinking about the complexity that was created by having the incorrect directory structure. So in a way, smaller repositories gave us the benefit of having to s getting to sidestep that particular problem of having the incorrect directory structure because you could still find whatever you were looking for. You didn't need to have it perfect.

speaker-1 (24:39.212)
The question is, is the freedom of having an incorrect directory structure important? In other words, do people need to have a different directory structure because it liberates something? Or is just the case that they were not exactly aware if it should be SRC or source? Does it need to be the case that the people that are coming up with the new structure not know what the actual structure should be? How do I make that answer more readily available? But there's another one thing that you mentioned before you talked about the rule of three.

The rule of three says, right, you were saying that if I do it three times that I want to automate it. When it comes to, for example, testing, this rule of three is not encouraged. Not. No, it says the rule of one, right? Before I'm gonna do anything, I want to in see if I can capture it in my hypothesis, right? And now in order for that to work, right? The rule of three was made like this because it was still too expensive to do it.

speaker-0 (25:19.739)
for sure.

speaker-1 (25:38.338)
But if I done it three times then it's probably that I'll be doing multi multiple times. Therefore I need to have the business case before I'm gonna build invest the time to build the tool so that I admize it over some longer amount of time.

speaker-0 (25:50.318)
Let's fight about that. Because I I think I think it's that's totally true. And we can look at the X KCD on like should I automate based off the amount of time I save versus the amount of time that would cost to actually perform that activity in ad at infinim. the I think the rule of three really points to the fact that we don't have a perfect capture of the requirements and can never fully have enough information. And it's like that three-standard deviation.

actually tells us that we have enough information so that we know that whatever refactor or change we make or whatever automation we're putting in place best represents the reality to a better degree than if there's just the first time.

speaker-1 (26:29.558)
And if the if the problem didn't appear often enough, then I'm not gonna be building the tool because it's too expensive to build the tool and it will just increase the time I in I need to actually solve the problem that I'm paid for, right? however, if we can bring the cost of a tool to be lower that so that the the constructing the tool and solving the problem is less costly than solving the problem without building the tool.

Then I will be building tools for every problem. And some of those tools will be reused. Some most not. For example, I give you an example where this happens. Most of the time when when I ask people, so if I have give you a problem in the database, what's the first thing you do? I'm gonna write a query. But what is a query, right? What is a query? It's a little tool that takes lots of data and then compresses into a little smaller data, right?

It's really fascinating now to just consider coming back to, you know, what needs to change. So if I take the same developer and say, Well, what's I have a problem in the system, what you're gonna what what would you like in this code, what you will do is that they will start reading. Now what's the difference? Why why one behavior in one place and the other one in the other one? Because the code is also data. now let's take a look like a quick look at the interface that they do the activities through. So when they when they are exposed to the database problem, what do they see?

It's a database tool with a big query at the top and the data at the bottom. So what you're gonna do? Well, you're gonna write the query at the top and you're gonna see the data at the bottom. It's very interesting, right? When you switch to to looking into investigating the code, what do you have? You have a giant text editor. So what are you gonna do?

speaker-0 (28:13.974)
No, use the search use the search functionality. With a re with a magic regex. You regex you re regex to find exactly your problem.

speaker-1 (28:21.24)
Right. Yeah, yeah, yeah, for sure. because why are you n why are not people rejexing through their database, by the way? Right? Because because it doesn't make sense. Why would you do a way reject means every line of code has the same meaning, same structure as any other any other line of code, which is absolutely not true. Code is incredibly structured.

speaker-0 (28:28.076)
Ha ha ha.

speaker-1 (28:43.894)
And doing it and by the way, the editor that you're using already has all that information. Yeah. But it's hidden in that box that you're not allowed to open and God forbid you will actually, you know, get good at what you're doing and not buy off the shelf tools that actually don't solve your problem.

speaker-0 (28:59.65)
I think compiler there is the biggest challenge that a lot of the information that you wanna get at is only exposed after it's been com compiled into a binary and gone through the assembly and linking process. And it's only recent languages that have been able to reflect this back into the development experience to have the answer right away. And I'd say, you know, realistically the thing that is always missing for me is like hitting F twelve or F eleven or whatever on a keyword or class or whatever and being able to jump

in to understanding what that is, but I I totally agree in the in the aspect where what information is exposed in that way is always much less than I want it to be.

speaker-1 (29:40.44)
So something that would take the Excel and transpile it into this. So you would say, yes, because only the interpreter has it, I don't have access. Yeah, but we could still build a tool that reverse engineered that whole language and puts it together within the context of the entire pipeline. By the way, on the other side of the spectrum, there was a a low-code platform with called Pega and they had twenty two thousand scripts in there. Twenty two thousand scripts. Pega is advertised as, you know, built faster and they built fast.

Except that they built twenty two thousand scripts and now they had to reason about them. But Pega has no tool to give you any dependency analysis and definitely no performance of anything and definitely no data flows that goes going through those. So we had to now reverse engineer that one. But Pega doesn't even have an expert properly of their source code. So we took the backup format and reverse engineered the whole thing out of that. So no, the compilers are absolutely not the blockers. it is

It is absolutely possible to build tools much, much faster than people think it is. and there is no excuse not to build those. The only excuse we find is because we refuse to talk about the problem. And because we refuse to talk about the problem, we do not observe the opportunities that are all around us. They are exist.

speaker-0 (30:57.528)
So I think this is where we're a little bit unique because we try to automate everything that we do in to a ridiculous degree. Like when we get a support ticket where our customer is asking a question, that answer we think about not, we have some sort of giant knowledge base that we can throw an LM at and it can dynamically generate the answer, but realistically, is there a deterministic tool that allows us to

directly answer it. And if we don't have that, then we go rule of three. The third time we get a support ticket asking about a piece of information, we end up stacking up the internal process. And we have one for this actually. We start by adding some code to our existing repository for our service. I I think an example is like we frequently got a request like when a new user gets into our login system for that we build for our customers.

why doesn't that user link to an existing user? Like, maybe the email address is different or maybe the identity provider is a little bit different, et cetera. And we get that question pretty frequently. And on the backside, we started building up an answer to this. We started by making the right database queries so that we could quickly answer. But then when we got more requests, we said, well, a database query isn't sufficient. We should have multiple database queries with multiple, you know, data alignment and understanding how it

it is cohesive together. And then that wasn't sufficient. So then we expose it on an endpoint that could be called. And for security reasons, of course, you know it has to have the right authorization and whatnot. And then we took the last step of exposing it back to our end users. And I feel like we have this capability to think in this way. And I'm just wondering why isn't that no one else is doing this? Is it that the ROI just doesn't seem like it's high enough? Is that a short-term problem that people are too focused? Or is it like

It requires a deep under I because I feel like a lot of what you're talking about requires a deep understanding of the technical challenges. And I the one thing I think you really bring to this conversation is this idea that building new tools is is incredibly cheap. And with LLMs, it's disposable software that should be almost free to make happen. So I feel like if anything, we're at the exact right point in time to say that everyone should be doing this to solve every single problem that they have.

speaker-1 (33:07.638)
LLMs essential for this, the answer is for sure not. Yeah. The LLMs, can LLMs help? For sure, yes. Right. So but just don't use the LLM like you were saying. Right? You don't ask the LLM, tell me why this is the case, and then it And then you get your proposal and then you can investigate the tool because now the bias of because whatever the LLM says, it's an opinion. It's just like how your colleague says it's an opinion. So so you take the opinion and then you externalize

speaker-0 (33:21.184)
Use it to build the tool.

speaker-1 (33:36.546)
The opinion, you externalize the bias, you can investigate it. You you can investigate it yourself and then decide whether or not you're gonna delegate to that deterministic tool. Right now, you mentioned why is it the case. So the geekest detail in the whole of software engineering, that is how a commit how how does code go from a commit to a deployment? Like, who cares? Like the is like this is like the most boring possible thing, in in the whole of software, right? But it turns out

Then if you go from manual, a manual solution to an automated solution, you can create trillion dollars worth of economy, which is like, wow. that's kind of interesting, right? but if the if in your organization that now does manual deployment and you just make it now automatic completely overnight, but you don't change anything else, you won't get you will get some benefits, but they will be marginal. If you want to get to the to the real leaps,

You have to change all the processes that happen on top of it. Because basically these manual delay this deployments they introduce delays and delays then reper they had they created unnecessary processes that poof should disappear basically overnight when this transition happens. Because now all of a sudden the deployment becomes from a manual, a custom customly produced artifact to a commodity and therefore coming back to the wording map, it creates a new whole new space on the top left. Okay.

speaker-0 (35:03.678)
In reading the book and also our conversation here, that one of the problems I think we have is that the construct that we have to store our tools or our toolkits is so inflated with things that have nothing to do with the process in which we evaluate or get answers to questions that we ask. I think of systems where you have or an organization that has hundreds of thousands of scripts that they can run, finding the right script at that moment is a challenge.

in a lot of different ways. And even as you think about like how can we optimize this to find the right script, you're building a system on top of that, which has its own complexities and and challenges in a way. And it's like in a way turtles all the way down. And I don't think it's getting any better in any regard. And then you end up with whole organizations and companies that are dedicated just to building the the tools to be able to interrogate the the database or

be provided to engineering team to be able to answer the questions. And I I don't know that we have a very mature response to how to deal with package management, I think, in the world. I mean it's not great. There's still like dependency change and diamond dependency problems and supply chain attacks and and S bombs, you know, like the bill of materials. but I think when it comes to the tools, I think there's like a whole space here

that is so poorly managed as far as how to do this, other than, yeah, we have a repository with a bunch of things in it.

speaker-1 (36:32.77)
That that that's right. what if I can treat the data the data pipeline problem to be the same as the observability problem, to be the same as the email linking to the user problem? because this this goes quite against the the industry where the data pipeline problem is a billion dollar vertical, and the observability problem is another billion dollar pro vertical and the other ones and there'll be all sorts of verticals with billion dollars

worth of tool vendors offering you off-the-shelf solutions. and our view is well all based all of these all of these propositions are based on the idea that tools are expensive to build. Well what if tools are not expensive to build and that you can have an environment where all you can approach arbitrary problems in the same way or in a uniform fashion and therefore you can internalize the skill once

And then build a uniform environment where you're just hosting all of these perspectives.

speaker-0 (37:35.318)
What is the smallest change to the software development workflow or reality even that would cause us to be pushed in the right direction? Is this a matter of having an integrated development environment like our our IDE or just say like you know Visual Studio Code or something else, whatever using writer, et cetera, to have some fundamental difference on how information is exposed to a technical end user? Or is there something fundamental that we ha like a is there a

a change in the way we build software products that has to be considered? Is there, you know, the connection? Is it some other primitive? Like what what is the first thing that really would need to change and we could see the output from? Like not just, change your mindset, but specifically some artifact that would prove we're on the right place.

speaker-1 (38:22.05)
We want an IDE, we need that integrated experience, but the current ones fail quite dramatically. What I'm proposing is not an increment. Spend fifteen minutes per day, you know, during your coffee break, just starting the conversation. How do we read code? That's it. This is where it starts. Make it a subject of conversation before you even accept that this is a problem.

There's no way we're gonna make any like no amount of tooling will change anything. That's the simplest change. And today, right, especially we can take this opportunity that the LM brought, to actually make it the subject of conversation because finally people recognize that they have to read code for most of their time and that they also accept that it is annoying.

But they shouldn't be reading codes. And yes, they shouldn't be reading code to the ex to the extent that they do I

speaker-0 (39:19.714)
I think, you know, as I even going further and saying, and maybe this is too much of a shill, that the the first few chapters of the moldable development, the rewildling software engineering book that you've written with Simon Wardley, is definitely an can be provided as an eye-opening experience for how how wrong, you know, some companies are really doing it. And you have some like absolute great examples in there. And like, you know, the example that I'm thinking about is the one with the silicon die hardware manufacturer that that you had.

speaker-1 (39:48.236)
So that that was actually the first time when we saw this idea work at scale. So th this the story goes like this. it's one of the companies, reasonably large companies. I mean, they are not allowed, for example, to trade with China today. So reasonably important and they build wafer technology, so the factories that produce that manipulate wafers. They basically build these rooms and they they work with with you know, Nobel laureates in chemistry and physics and this is is like an amazing place to be and

To visit. It's absolutely a an experience. But these robots that take these rooms, right? They have these robot arms that are manipulating you know often like what is worth like a million dollars worth of material inside. And they have to be con distribu they have to be controlled in some way, and they use a distributed system for that, and that distributed system used you know relied on a protocol. And they didn't know how that protocol worked anymore.

and they gave us this you know challenge and we we were doing it as a research project for them and so they gave us the sources and the idea was you know take take a mesh of of nodes written in their technology and then basically put another another node written in another language to show that you can communicate following the same protocol. And so a month later, which is usually how, you know, session like meetings are set, you know, you have a cadence of things.

So a month later people say, Okay, are you ready to start? and then we said, Well, we are done. And so, what do you mean you're done? So we showed them, Yeah, here's a new note, and we, you know, it they communicate and say, Yeah, but how could you be done? Because we had this bet inside or bets inside the company saying that there's no way you can do it in a year. They gave us a year to look into the problem. because we have no idea how we could do it. And so, yeah. and then he asked us, so how did you do it? Well

Well, basically we spend three weeks building a debugger for to figure out your protocol. and then another week to actually, you know, implement the the rest of the thing. And and that's it. It's it's that simple. The solution is much less expensive than people believe it is. It's very, very powerful. computation can be powerful.

speaker-1 (42:11.17)
But you cannot perceive anything inside the computation except through a tool, which makes the tool fundamental. If you change the tool, you change your perception. If you change your perception, you change the way you think. And applies, by the way, also to to the new world of AI, right? Don't don't use AI through through standard tools.

speaker-0 (42:32.05)
I there's a lot you know, there's a lot of great arguments against LLMs that I just don't think people bring up and this one is one of one of my favorites.

For for sure. the other one that comes to mind fundamentally is that, yeah, humans are computational units and so are LLMs. So therefore anything a human can do, an LM can do. And so obviously LLMs are gonna be better in the future. And I'm like, you don't even know where the future of humans are going, let alone that they're computational units. So it's like the whole argument is flawed, honestly. but before we get to too much philosophy into into LMs, I think maybe.

This is the moment we we we we shift over to the end of the episode. let's move over to picks. So Tudor, what did you bring for the audience today?

speaker-1 (43:18.178)
I I spent a week in Normandy with my family and we visited the D-Day beaches and it was very impressive. of course I mean the the human side of things is the thing that that captures the you know the attention there. But if you if you spend enough time there, there are all sorts of the y you get to see the the magnitude of the engineering

effort, the engineering and research effort that had to all come together for this massive undertaking to actually be possible. I found that to be mind-boggling. We I knew about some of it, but taking the time to be there and see it, I was unprepared for that impression. Like and all the way to the details of you know, how do you build a bridge?

of a water, which was not a thing that was possible before, in in a short amount of time, with not two skilled people or with without having two sp super specialized equipment there, or how do you build the whole artificial port in a matter of days, right? the all of these questions were just not known just a couple of years before. Yeah. I found that to be absolutely impressive.

And is also a a really interesting testament of what we can do as a human species when we put our mind to to it.

speaker-0 (44:50.518)
I think there's something actually quite amazing about the beaches. And the one thing that comes to mind is can you like not not you, because obviously you visited there and you probably know the answer, but just you may know that landing on the beaches was basically the end of the war. But why? Why what what did how did landing on the beaches in Normandy cause the end of the war? Like what is the connection there? How like Germany's on the other side.

What does that have to do? Wha how does that impact the army realistically? And I find that as like as soon as you ask that question, there's so many gaps there in the can on on what's actually happened from a s not just military strategy, but also human strategy that you really have to wonder, like, what do you actually know about about the history? And I feel like it is one thing that's often left out of the history books that is actually talked about the not just the scale, but the feats that had to go in and why it was so important.

for that l location to be picked in order to counterattack. So I love I love your pick. is there like a particular set of museums or location or tour map that you would suggest to people that I maybe we can share a link for in case anyone else is interested.

speaker-1 (45:59.448)
time it took a week to visit the you know the entirety. I think the the size it's only I don't I don't know if there's any one particular thing that was you know a must visit. I I just found the the size of it. I just visited everything.

speaker-0 (46:19.212)
Yeah. How how do you know what everything is? Like I what did you do like a lot of background research first?

speaker-1 (46:26.286)
this they're they're good sites and so the the the official French sites are are very thorough. So you get an idea. You get a good idea about that.

speaker-0 (46:39.788)
okay. I I I I love it. well mine my pick is gonna pale in comparison. There were definitely a few different things I was inspired to to share and maybe there'll be there'll be future future picks related on the topic. But for this I have something totally unrelated and that is the well, I don't know if I can really claim it's the the condiment, but I was a huge fan of the Hoi Fong Sriracha, the you know, red red sauce with the rooster on it with the green top.

bottled in California and there was actually such a controversy that happened basically at the pen pandemic where it got worse. It disappeared and got worse. And it's never really recovered. And I don't want to spoil too much about what actually happened there, but I will say they were getting their peppers from another from a grower and they had a a pretty sweet deal. And after and they switched they switched growers basically.

And the original grower of the jalapenos that were being used for sriracha b started bottling their own sauce. And so if you want the authentic sriracha, it's out there. Underwood Ranch, that's the you know, that's the company that was making it. And go to town and there'll be a link in the description because I absolutely love the original sriracha. That was that was my go to hot sauce.

speaker-1 (48:01.758)
Good to know.

speaker-0 (48:03.638)
Yeah. I I mean I I am I am super into hot sauces and I could probably have ten more ten more picks of of different hot sauces. and I've had I've had to go out and sort of expand my repertoire after after the pandemic because it just it had so far gone downhill that it's you know, sometimes not even worth getting. And it's actually the reason why you see so many counterfeits show up now, in quite po popular style because

I think the biggest one is like there's like golden dragon and then there's flying goose is the other one. They're always like a adjective with a or gerund with a with an animal associated with it. And they're just not they're they're not the same.

speaker-1 (48:44.97)
I understand.

speaker-0 (48:47.916)
Well, thank you Tudor for coming on and walking us through this whole new mental model of software development. I I think it's great and something that everyone can go and check out and read up more on.

speaker-1 (48:59.318)
No, thank you thank you for your thank you for your time. I really enjoyed it.

speaker-0 (49:03.756)
Yeah, so that's multiple development and it's rewilding engineering.

speaker-1 (49:08.056)
So multiple development.com is the site where you can read about rewilding software engineering. That's the book that Simon and I are writing. I know

speaker-0 (49:15.34)
The links for this will be will be in the description for the episode for anyone that's interested. And you may have spotted a hint that in the future we're hoping to have an episode with an expert on formal verification of software, which I think is very interesting and required, but we're still looking for the appropriate expert to come on and talk with us about that. So if you're listening to this episode and know the right person, please forward them along so we can get do a deep dive on that. And I just will

I just want to thank all of the listeners for tuning in for this week's episode and hopefully we'll see you all back again.