1
00:00:07,906 --> 00:00:09,667
Welcome back to Adventures in DevOps.

2
00:00:09,667 --> 00:00:17,269
In October, we brought in John Papa to talk about IDE's co-pilot and critical thinking's
place in software development lifecycle.

3
00:00:17,269 --> 00:00:20,531
It was a fantastic episode, but we only got so far.

4
00:00:20,531 --> 00:00:25,652
To extend on the unanswered questions, we're diving deeper today by questioning hopefully
everything.

5
00:00:25,652 --> 00:00:29,494
And to do that, I brought in Tudor Girba, CEO of Think.

6
00:00:29,494 --> 00:00:30,878
Welcome to the show.

7
00:00:30,878 --> 00:00:33,250
Um, nice to meet you again and thank you for having me.

8
00:00:33,250 --> 00:00:42,961
We chatted before and you got into a lot of philosophical aspects of the work we do,
knowledge work, not necessarily, you know, hands on technology and it was it was just

9
00:00:42,961 --> 00:00:44,290
something that I wanted to share with

10
00:00:44,290 --> 00:00:46,707
Yes, I'm I'm glad you I'm glad you did.

11
00:00:46,750 --> 00:00:55,354
short term focused historically and I think even more recently have a focus of switching
to LLM generated software development.

12
00:00:55,354 --> 00:01:03,798
And I think realistically have understanding the philosophy and strategy behind how we do
software development and really any technical thing is important.

13
00:01:03,798 --> 00:01:05,859
Shouldn't have the thing all along.

14
00:01:05,859 --> 00:01:07,160
Let's put things in perspective.

15
00:01:07,160 --> 00:01:12,304
We are building a whole new world on top of this new kind of platform.

16
00:01:12,304 --> 00:01:16,546
There's a reasonable amount of responsibility that should come with that.

17
00:01:16,787 --> 00:01:31,146
And so should it not be expected of us to spend some reasonable amount of time to
understand what is it that we do, to understand the properties of the you know, the

18
00:01:31,146 --> 00:01:32,407
effects of our work.

19
00:01:32,407 --> 00:01:33,654
But on the other hand

20
00:01:33,654 --> 00:01:41,008
If you look at what developers actually do, they spend a small portion of their time
actually building anything.

21
00:01:41,008 --> 00:01:43,960
The vast majority of the time is being spent on reading.

22
00:01:43,960 --> 00:01:47,312
Now, today, this is much more accepted.

23
00:01:47,312 --> 00:01:56,268
People find themselves, I'm reading because the LLM is doing all the writing, and the only
thing I'm doing, I'm reading software engineering has fundamentally changed.

24
00:01:56,268 --> 00:02:00,500
But the thing is, I I I got to talk with

25
00:02:00,738 --> 00:02:07,100
thousands of developers, well before there were any LLMs even uh inside.

26
00:02:07,201 --> 00:02:11,642
And I asked them, Do you agree that you spend more than fifty percent of your time reading
code?

27
00:02:11,923 --> 00:02:14,424
And basically developers were saying yes.

28
00:02:14,424 --> 00:02:20,026
And this is a fascinating thing because uh 50% or more is misses is the single largest
class we have.

29
00:02:20,026 --> 00:02:21,807
There's nothing larger than that.

30
00:02:21,807 --> 00:02:22,107
Right?

31
00:02:22,107 --> 00:02:24,128
And then I I was asking them another question.

32
00:02:24,128 --> 00:02:26,858
When was the last time you talked about how you read code?

33
00:02:26,858 --> 00:02:28,270
It's a very simple question.

34
00:02:28,270 --> 00:02:29,420
What do you mean?

35
00:02:29,538 --> 00:02:30,739
How to read code.

36
00:02:30,739 --> 00:02:39,685
Given that this is the single largest expense we have, if we don't talk about it, this is
means this is not explicit, but if it's not explicit, it has never been optimized.

37
00:02:40,406 --> 00:02:41,587
The single largest cost.

38
00:02:41,587 --> 00:02:50,958
So we end up being an industry that starts building the whole new world without actually
understanding how it is that we're doing things.

39
00:02:50,958 --> 00:02:53,610
Companies are in different places on the spectrum though.

40
00:02:53,610 --> 00:03:05,008
Like I've definitely seen in some enterprise organizations lots of meetings dedicated to
not just what color should this button be, but what should the hierarchy of our thousand

41
00:03:05,008 --> 00:03:08,580
microservice monolith directory or repo should be?

42
00:03:08,580 --> 00:03:15,075
Like should we have a folder called tools and a folder called utilities and a folder
called, I don't know, like address management?

43
00:03:15,075 --> 00:03:16,756
And those things can be referenced each other.

44
00:03:16,756 --> 00:03:19,694
And I like I've seen a lot of conversations wasted.

45
00:03:19,694 --> 00:03:29,429
Trying to figure out, like seeing that there is a problem with how we've managed our
source code and just only slightly grasping that something needs to be different, but

46
00:03:29,429 --> 00:03:31,000
those conversations usually don't end well.

47
00:03:31,000 --> 00:03:41,646
Like they're based off of a couple of different people, usually with an architect title,
discussing or debating on what the optimal directory structure is, but not really have any

48
00:03:41,646 --> 00:03:44,928
first principles to get to a reasonable place.

49
00:03:44,928 --> 00:03:48,139
It's based off of opinions and maybe you could do some sort of experimentation.

50
00:03:48,139 --> 00:03:49,568
Uh maybe you

51
00:03:49,568 --> 00:03:56,963
you put forward hypothesis that uh an issue is quickly finding what tools are available in
order to utilize them.

52
00:03:56,963 --> 00:03:58,934
And then from there you can do some sort of research.

53
00:03:58,934 --> 00:04:02,176
But I find that very rarely happens in practice.

54
00:04:02,466 --> 00:04:07,769
Yeah, so I I I also find but I think at least this conversation, right?

55
00:04:07,769 --> 00:04:13,272
So what's the structure of the project is it does that does tend to happen.

56
00:04:13,272 --> 00:04:19,775
Um where it starts to fall back is when people agree and then uh different projects will
still look differently.

57
00:04:19,775 --> 00:04:25,859
So that their their their directory structure will still not be uniform across um even
after they make that decision.

58
00:04:25,859 --> 00:04:27,306
And the question is why is that?

59
00:04:27,306 --> 00:04:32,216
W what do we do when we want to ensure that the system behaves the way we expect it?

60
00:04:32,216 --> 00:04:32,866
behave.

61
00:04:32,866 --> 00:04:43,832
In fact, hopefully today, uh most organizations have internalized the idea that you
shouldn't release if there's even one test that is red, we automated the answering of

62
00:04:43,832 --> 00:04:45,553
functional questions about our systems.

63
00:04:45,553 --> 00:04:50,415
Is this is this particular functionality working the way we expect it to work?

64
00:04:50,415 --> 00:04:55,137
This model works exactly because it is made out of little tiny tools.

65
00:04:55,377 --> 00:04:56,918
We call them tests.

66
00:04:56,938 --> 00:04:58,249
What is our what are those tests?

67
00:04:58,249 --> 00:04:59,230
They're just analysis tool.

68
00:04:59,230 --> 00:05:04,385
They take lots of lots of information and they transform into red and green um types of
signals.

69
00:05:04,385 --> 00:05:16,808
And the totality of them helps us build a model of the system that allows us to steer much
larger systems, much more complicated types of functionality, much more certainly and much

70
00:05:16,808 --> 00:05:17,526
more comfortably.

71
00:05:17,526 --> 00:05:26,311
We've created an invention as far as the software development lifecycle goes of how we
build and manage sort of the quality gate.

72
00:05:26,311 --> 00:05:35,577
And if it really is a mechanism that it sits on top of the foundational layer of the
source code we write, we should be utilizing it as a mechanism for driving understanding

73
00:05:35,577 --> 00:05:38,949
about what the service or product or platform actually does.

74
00:05:38,949 --> 00:05:44,962
But I find in practice, very rarely are tests the driving factor for understanding.

75
00:05:45,176 --> 00:05:46,487
of what's actually happening.

76
00:05:46,487 --> 00:05:49,869
It's always like a uh after effect.

77
00:05:49,869 --> 00:05:54,131
Like it's the way you validate what is there is working in practice.

78
00:05:54,131 --> 00:05:59,424
I've very few companies are actually using their test be like, how does this function
work?

79
00:05:59,424 --> 00:06:03,496
let me go to the unit test and I'll look at the test and the test will tell me how this
function works.

80
00:06:03,496 --> 00:06:07,819
Well, I don't think that's usually the process, which is it seems like in a way completely
unfortunate.

81
00:06:07,819 --> 00:06:14,104
Because we if if our tests are really built up, if TDD is really what a company is doing

82
00:06:14,104 --> 00:06:23,614
Tests are a priceless opportunity to use as an endpoint or an entry point into the
understanding process that's being completely eliminated or ignored.

83
00:06:23,672 --> 00:06:28,263
When you go with you make it so there's a test which is red, then you go and you fix it.

84
00:06:28,263 --> 00:06:32,945
And at some point people say, um you you you clean up.

85
00:06:32,945 --> 00:06:36,666
But nobody says how do you go from the red test to fixing it?

86
00:06:36,666 --> 00:06:41,567
And nobody r says I how do you go from this solution here to the cleaned up solution?

87
00:06:41,567 --> 00:06:41,957
Yeah.

88
00:06:41,957 --> 00:06:43,128
Whatever that is.

89
00:06:43,128 --> 00:06:49,899
Those things are left to the imagination of the um of the reader, which is of course it's
development.

90
00:06:49,899 --> 00:06:52,404
So if you go back to why do people read code?

91
00:06:52,404 --> 00:06:54,907
is because they want to understand enough to make a choice.

92
00:06:54,907 --> 00:06:59,274
The because reality everybody knows that reality in this in the system.

93
00:06:59,274 --> 00:07:03,147
Um, it's not in documentation, it's not in people's heads.

94
00:07:03,147 --> 00:07:05,430
It's also not in AI summaries, by the way.

95
00:07:05,430 --> 00:07:05,836
But

96
00:07:05,836 --> 00:07:14,986
I I'm trying to figure out how does the corollary of how we built tests apply to the
change that we should make to our software development process.

97
00:07:14,990 --> 00:07:20,733
So answering any question about the system should be intermediated through deterministic
contextual tools.

98
00:07:20,733 --> 00:07:24,866
Let's go back to your problem with uh what's the directory structure of my system.

99
00:07:24,866 --> 00:07:32,750
So imagine I take the directory structure, which is basically gonna be a tree, and I
imagine I will be building uh I can build a grammar.

100
00:07:32,750 --> 00:07:42,816
And then I can take that grammar and I run it against all the all the directories that I
have in my in my system, and therefore I'm gonna have a test.

101
00:07:43,266 --> 00:07:46,948
But this is this is not gonna be a str uh a functional test, gonna be structural test.

102
00:07:46,948 --> 00:07:51,430
There are gonna be other kinds of questions for which we do not know the answer upfront.

103
00:07:51,430 --> 00:07:58,672
Um a case where we that we worked on where there was a a telecom company having a data
pipeline.

104
00:07:59,573 --> 00:08:05,666
and they their problem was that they worked on it for three years and they wanted to
optimize it.

105
00:08:05,666 --> 00:08:11,212
And three years back, um, it took about one day from the moment they had the data until
the moment

106
00:08:11,212 --> 00:08:14,493
Some offers were going to lots of customers that they were having.

107
00:08:14,493 --> 00:08:17,164
This was their main marketing data pipeline.

108
00:08:17,265 --> 00:08:29,569
And um three years later, so this was like a big problem because this was a
business-to-consumer uh business, and uh, you know, reacting, having a one-day reaction

109
00:08:29,569 --> 00:08:33,261
time from an event to an offer, uh, it was very, very long.

110
00:08:33,261 --> 00:08:34,511
And the board knew about this.

111
00:08:34,511 --> 00:08:37,002
And the CIO demanded this must be much faster.

112
00:08:37,002 --> 00:08:41,154
Um, three years later, somebody measured, it took exactly one day.

113
00:08:41,592 --> 00:08:45,743
From the moment they had the data until the moment they were sending offers.

114
00:08:45,803 --> 00:08:49,264
And so now the question was, what what happened in three years?

115
00:08:49,264 --> 00:08:51,065
Now this is like a different kind of question.

116
00:08:51,065 --> 00:08:53,962
It's a question for which we do not know the answer.

117
00:08:53,962 --> 00:08:56,786
Uh we might have potential hypotheses.

118
00:08:56,786 --> 00:08:59,647
So in their case, when we asked them, we say, Well, what's your hypothesis?

119
00:08:59,647 --> 00:09:05,428
Well, we probably are producing uh too much data and this is why it takes one day.

120
00:09:05,849 --> 00:09:09,229
But we don't know why this has not been optimized.

121
00:09:09,229 --> 00:09:11,370
So like where did the effort go?

122
00:09:11,370 --> 00:09:12,470
Still an unknown.

123
00:09:12,470 --> 00:09:15,211
So we started with saying, so what is your data pipeline?

124
00:09:15,211 --> 00:09:16,161
And they drew for it.

125
00:09:16,161 --> 00:09:18,292
We say, draw it on the whiteboard.

126
00:09:18,332 --> 00:09:20,172
They didn't have documentation for it.

127
00:09:20,172 --> 00:09:23,613
So they drew it on they they drew it on the whiteboard.

128
00:09:23,613 --> 00:09:27,034
Uh sounds, you know, like sounds funny, but at least they were honest.

129
00:09:27,034 --> 00:09:30,325
Because there are others that say that they have documentation and it's still wrong.

130
00:09:30,325 --> 00:09:41,100
So um in this case, uh in this case, right, they have they draw for us the the the diagram
on the whiteboard and it was they drew for us four boxes.

131
00:09:41,100 --> 00:09:42,451
So now what do we do?

132
00:09:42,451 --> 00:09:50,038
We make it the job of the system to give us back this picture, but this time through a
tool that directly reads it from the system.

133
00:09:50,038 --> 00:09:51,219
So we start to build this one.

134
00:09:51,219 --> 00:09:54,742
Now in the process, we discovered it in the first box.

135
00:09:54,742 --> 00:10:02,328
They had a homegrown domain-specific language, which was actually implemented in Excel.

136
00:10:02,809 --> 00:10:08,514
So some like was a very elaborate Excel spreadsheet in which people were describing.

137
00:10:08,522 --> 00:10:12,205
their transformations, which were then transpired into Oracle instructions.

138
00:10:12,205 --> 00:10:18,639
In this process, we noticed, yeah, the output of the first box doesn't seem to quite match
the input of the second box.

139
00:10:18,639 --> 00:10:21,171
It was like in two weeks into this project.

140
00:10:21,532 --> 00:10:28,377
And we went back into to to them and said, well, are you sure you don't have something in
between the first box and the second and the second box?

141
00:10:28,377 --> 00:10:30,728
And they discovered the whole system.

142
00:10:30,968 --> 00:10:32,513
How has that happened in practice?

143
00:10:32,513 --> 00:10:34,228
Like what like what world

144
00:10:34,254 --> 00:10:37,447
Because we thought, this is this was so rich when we saw that.

145
00:10:37,447 --> 00:10:40,379
It was really funny, actually.

146
00:10:40,379 --> 00:10:45,513
It was really nice for them to accept that yes, right, our understanding was really poor.

147
00:10:45,513 --> 00:10:53,310
But I've been having conversation with lots of CTOs and they or CIOs and they tell me,
yeah, I can tell you a story about this.

148
00:10:53,434 --> 00:10:55,442
How how is that normality?

149
00:10:55,442 --> 00:10:57,033
Because this shouldn't be normal.

150
00:10:57,033 --> 00:11:03,042
Um but unfortunately, so maybe at that level, maybe it's not the case everywhere,
hopefully.

151
00:11:03,042 --> 00:11:11,044
But if you go one, two levels deeper, you will find immediately where people tell you that
the system is something, but the system is actually something else.

152
00:11:11,104 --> 00:11:17,086
And what this leads to is um a discrepancy between our mental model and what the system
is.

153
00:11:17,086 --> 00:11:19,467
Our mental model and reality.

154
00:11:19,467 --> 00:11:26,049
And so you we end we can in this situation, we can end up easily, you know, dancing around
the fire to make rain happen.

155
00:11:26,189 --> 00:11:26,519
Right?

156
00:11:26,519 --> 00:11:29,442
So and then and then we will have debates whether or not

157
00:11:29,442 --> 00:11:38,440
We should be dancing clockwise or counterclockwise and there should be ev then there will
be evidence gathered as to why clockwise uh dancing is actually much better than and the

158
00:11:38,440 --> 00:11:40,512
counterclockwise to make rain happen.

159
00:11:40,512 --> 00:11:47,578
And so and this is unfortunately the kind of thing that we are in that we find ourselves
often today.

160
00:11:47,759 --> 00:11:49,360
Because why?

161
00:11:49,360 --> 00:11:56,146
Because we just don't talk about how do we go from a question about the system to getting
the answer to the system.

162
00:11:56,334 --> 00:11:59,364
So I feel like in theory it like I agree.

163
00:11:59,364 --> 00:12:01,176
It there's should never happen in practice.

164
00:12:01,176 --> 00:12:11,000
But then I think about actual organizations where what you've built, and even more I think
is true today, that you have something that's just unmaintainable because the people who

165
00:12:11,000 --> 00:12:17,283
have built it and had the mental model even in their heads, assuming you could maintain
it, have long left the company.

166
00:12:17,283 --> 00:12:25,198
And often these challenges or these questions that want to be answered by the business of
like, why does it take twenty-four hours or a week or a month to

167
00:12:25,198 --> 00:12:34,381
close the sales cycle uh is trying to be driven by someone that's outside of even
engineering or wherever the problem is being set up.

168
00:12:34,381 --> 00:12:40,482
And then they're applying that process to the rest of the organization to give them back
an answer.

169
00:12:40,482 --> 00:12:50,665
And the individual engineering teams or organizations or what have you don't have the
resources or like time or understanding of the system to actually give a a a correct

170
00:12:50,665 --> 00:12:51,285
answer.

171
00:12:51,285 --> 00:12:54,846
And so they're of course guessing in some regard.

172
00:12:54,854 --> 00:13:02,961
Uh and so I sort of like in practice I can see this as being uh a reasonable n and almost
uh dif definitive and conclusion.

173
00:13:02,961 --> 00:13:11,330
Every single system we build, uh if it needs to be in a way self-documenting, because no
one on the outside will ever have a correct model.

174
00:13:11,330 --> 00:13:13,161
It was there's whatever the saying is, right?

175
00:13:13,161 --> 00:13:16,084
Like um all models are wrong, but some are useful.

176
00:13:16,084 --> 00:13:20,828
Uh but correct model is obviously more useful than an incorrect one.

177
00:13:20,844 --> 00:13:27,517
I I'm surprised that for answering questions, more organizations don't actually jump in
and do the analysis that's necessary.

178
00:13:27,517 --> 00:13:39,952
I feel like in the sales world or at least in marketing, we see lots of ascribing to the
benefits of making an A B test on a website and seeing how many users click by and whether

179
00:13:39,952 --> 00:13:43,684
or not that actually represents the reality correctly is a whole other question.

180
00:13:43,684 --> 00:13:48,636
But we do see them actually even appro like approximating the scientific process in some
regards.

181
00:13:48,696 --> 00:14:00,293
data scientist getting closer to where business decisions happen so that when people have
a question, a tool is being created specifically for that question so that an answer then

182
00:14:00,293 --> 00:14:01,694
can be used afterwards.

183
00:14:01,694 --> 00:14:11,146
So those, you know, A B tests, that's a little tiny experiment that requires a little tiny
tool that re that will project whatever the answer is so that the people, the stakeholder,

184
00:14:11,146 --> 00:14:15,062
the ones that has the question and knows what to do with the information can make the
choice.

185
00:14:15,062 --> 00:14:16,853
It doesn't solve the whole problem.

186
00:14:16,853 --> 00:14:19,642
It's not like a magic solution for everything.

187
00:14:19,642 --> 00:14:22,725
It just removes the largest amount of noise.

188
00:14:22,725 --> 00:14:37,020
And in software engineering right now, because we choose not to talk about how we read
code, we people miss this enormous opportunity to create value that is much, much cheaper

189
00:14:37,020 --> 00:14:38,410
than people believe it is.

190
00:14:38,496 --> 00:14:46,432
Is there a distinction between an expert coming in and being able to apply and know which
are the right tools to utilize to find the answer?

191
00:14:46,432 --> 00:14:52,957
And a fallacy that the fundamental sort of flaw is the approach that is being taken.

192
00:14:52,957 --> 00:14:56,440
And like I I can imagine, let's say we're talking about the performance of a system.

193
00:14:56,440 --> 00:15:04,086
ah I can go out and find tools that are built on your favorite package manager, PyPy or
NPM, NuGet, et cetera.

194
00:15:04,086 --> 00:15:13,825
And download one of those, install it into my repository, or pay for a open source or SaaS
solution that goes and monitors and instruments my project, business solution, whatever

195
00:15:13,825 --> 00:15:17,478
service, and get an answer and then make decisions based off of that.

196
00:15:17,759 --> 00:15:20,302
Is it that the wrong tools exist?

197
00:15:20,302 --> 00:15:23,755
Is it that the right tools exist but the we don't know how to apply them?

198
00:15:23,755 --> 00:15:29,280
Or is there some fundamental philosophy about how we're building our systems that's
flawed, that needs to change?

199
00:15:29,368 --> 00:15:32,259
We definitely do not have a shortage of tools.

200
00:15:32,539 --> 00:15:34,149
Right, so that that's one observation.

201
00:15:34,149 --> 00:15:40,941
The other observation is this idea of developers reading code for the largest part of
their work is is not new.

202
00:15:40,941 --> 00:15:46,043
And the oldest reference to the subject goes back when about the nineteen seventy-nine
that I could find.

203
00:15:46,043 --> 00:15:49,083
So we have like uh half a half a century.

204
00:15:49,083 --> 00:15:53,705
So the amount of tools grew, but the problem didn't get solved.

205
00:15:53,705 --> 00:15:59,676
It's also obvious that the tools we've built are not solving the problem because we can
still observe the effect.

206
00:16:00,044 --> 00:16:01,855
So then the question is why not?

207
00:16:01,855 --> 00:16:05,157
We were building the wrong kinds of tools.

208
00:16:05,738 --> 00:16:20,167
And by this I mean, yes, you can go and download uh a tool from the web, but if you use it
as is, then it's like downloading a test from the web and using it as is to see to to ask

209
00:16:20,167 --> 00:16:22,129
it about the system that you have.

210
00:16:22,129 --> 00:16:24,640
Nobody's doing this because it makes no sense.

211
00:16:24,812 --> 00:16:30,967
Right, everybody will say, Yes, I will be downloading an engine in which I will be writing
my own tests.

212
00:16:31,167 --> 00:16:31,908
Why?

213
00:16:31,908 --> 00:16:36,051
Because nobody knows what makes my my system valuable.

214
00:16:36,051 --> 00:16:36,662
It is like this.

215
00:16:36,662 --> 00:16:42,597
So if you then inverse the situation and you internalize the skill of building the tool.

216
00:16:42,597 --> 00:16:48,362
So this whole idea is based on uh on the premise that building tools is expensive.

217
00:16:48,362 --> 00:16:54,286
When this idea of testing came first, pe I I remember p talking with people in the two
thousand

218
00:16:54,286 --> 00:16:58,748
uh 2000, 2001 and we yeah, but you cannot apply this one to UI.

219
00:16:58,748 --> 00:17:00,388
Well it turns out you can.

220
00:17:00,499 --> 00:17:05,351
but for real time this for real time systems you cannot you cannot apply any kind of
testing.

221
00:17:05,351 --> 00:17:06,771
Well it turns out that you can.

222
00:17:06,771 --> 00:17:10,783
So the uh and and today we're saying but this is impossible.

223
00:17:10,783 --> 00:17:16,155
How can you possibly be um build a system that parses a new language very quickly?

224
00:17:16,155 --> 00:17:24,268
How can you possibly build an architecture diagram very quickly that is highly specific to
exactly our protocols and so on and so forth.

225
00:17:24,386 --> 00:17:25,134
very quickly.

226
00:17:25,134 --> 00:17:30,997
And you sort of just threw this out there that we have the tools available, but they're
maybe not the right tools.

227
00:17:30,997 --> 00:17:35,210
But it's really the cost of implementation or utilizing those, which is problematic.

228
00:17:35,210 --> 00:17:45,165
And I think you made a reference to this in your book as well, that realistically, if
tools were negligible to generate and create to solve any sort of problem, you would spend

229
00:17:45,165 --> 00:17:53,780
all your time j first generating the appropriate tool to extract the data that you want,
and then just get it generated and then get the data.

230
00:17:53,902 --> 00:17:55,923
but we're we're we haven't been doing that.

231
00:17:55,923 --> 00:17:58,865
And I I'm wondering what is the like why did that happen?

232
00:17:58,865 --> 00:18:02,127
Is it because the primitives we have, the programming languages that we have available?

233
00:18:02,127 --> 00:18:05,398
Is it about the interfaces that the hardware exposes us?

234
00:18:05,398 --> 00:18:07,430
Are are they just all wrong in some way?

235
00:18:07,430 --> 00:18:15,484
And we need to push back the learnings that we've had in the last twenty to thirty years
to a lower layer so that the abstractions that we're actually given, the interfaces that

236
00:18:15,484 --> 00:18:23,518
are actually available to us contain the the right pieces of information, or is there some
other fundamental problem that we haven't been addressing?

237
00:18:23,576 --> 00:18:28,787
There hasn't been a shortage of of exploiting or exploring what languages could be.

238
00:18:28,787 --> 00:18:38,510
And for sure, there are some languages that are more inviting than others for um you know
for scrutiny or make it make things more available.

239
00:18:38,510 --> 00:18:45,172
There is no way to perceive anything inside a system, inside a computer, except through a
some sort of an interface.

240
00:18:45,172 --> 00:18:47,282
Which basically means that tools are essential.

241
00:18:47,282 --> 00:18:52,834
For tools to be effective, we they have to be um sp contextualized.

242
00:18:52,834 --> 00:18:54,076
to the problem that we have.

243
00:18:54,076 --> 00:19:03,993
So it's not just some generic tool that we're downloading but from somewhere, but uh but
something that actually matches our questions about the about the system at this point in

244
00:19:03,993 --> 00:19:04,734
time.

245
00:19:04,760 --> 00:19:14,010
I think there are lots of engineers out there that are every single organization that that
say, yeah, you know what, I will just write I have a whole repository of scripts that I I

246
00:19:14,010 --> 00:19:16,033
I I keep and maintain.

247
00:19:16,126 --> 00:19:21,208
to to solve a particular problem or you know what, I'll just generate a CLI to to do this
thing.

248
00:19:21,208 --> 00:19:25,810
And the engineer the most common strategy I think is like there's an engineer working at a
company.

249
00:19:25,810 --> 00:19:30,672
They feel like they don't have the time or space to solve the real problem that that
company is is facing.

250
00:19:30,672 --> 00:19:33,253
So they leave, they start their own quote unquote company.

251
00:19:33,253 --> 00:19:38,656
They make a CLI, it does something, then they release it and then they wonder why they're
not getting any users.

252
00:19:38,656 --> 00:19:41,491
But I think they are making tools in some regard.

253
00:19:41,491 --> 00:19:44,658
I I I I don't know that they're just not doing that.

254
00:19:44,982 --> 00:19:46,726
for sure, people are making tools.

255
00:19:46,726 --> 00:19:48,649
They're just not making enough of them.

256
00:19:49,765 --> 00:19:50,895
More software.

257
00:19:50,895 --> 00:19:52,426
We should have more software.

258
00:19:52,426 --> 00:19:53,106
Absolutely.

259
00:19:53,106 --> 00:20:01,349
No, but it's like it's like with it it's again, if you go back to tests, there was a time
when people were writing no tests and people say, I'm gonna write some tests and say,

260
00:20:01,349 --> 00:20:02,789
Well, how many do you have?

261
00:20:02,789 --> 00:20:03,869
Fifty.

262
00:20:04,169 --> 00:20:04,689
Yeah.

263
00:20:04,689 --> 00:20:08,911
They you probably don't yet have it you're don't you're not yet testing your system,
right?

264
00:20:08,911 --> 00:20:20,234
So in two thousand and two there was uh in uh when the when T D D by example by Ken Beck
came uh was published, how many tests do you need and does this actually s this idea does

265
00:20:20,234 --> 00:20:21,400
it scale to

266
00:20:21,400 --> 00:20:22,350
to large system.

267
00:20:22,350 --> 00:20:30,412
And in there, towards the end of the book, there is one little case study which says there
was this organization which is called LifeWare.

268
00:20:30,412 --> 00:20:36,254
um And they uh they were having 4,000 tests.

269
00:20:37,074 --> 00:20:42,656
And not only they had 4,000 tests, they had a 500,000 lines of code system.

270
00:20:42,916 --> 00:20:49,758
And half of these 450, 500,000 lines of code systems were test code.

271
00:20:50,018 --> 00:20:51,078
Half of it.

272
00:20:51,532 --> 00:20:58,426
And this was the argument as to why they were saying this was the largest test suite that
Ken Beck has ever seen at this at at the time.

273
00:20:58,426 --> 00:21:03,989
And this was basically the argument, this is why they knew that it actually does skate.

274
00:21:03,989 --> 00:21:12,554
Now, we actually have the l the the the pleasure and the privilege actually to work with
Lifeware since a few years.

275
00:21:12,554 --> 00:21:16,376
And um today they have a hundred seventy thousand tests.

276
00:21:16,376 --> 00:21:20,898
So in two thousand and two their four thousand tests ran in twenty minutes, which was

277
00:21:21,026 --> 00:21:24,477
Um a remarkable achievement at the time.

278
00:21:25,488 --> 00:21:30,090
and today they have 170,000 tests and they run in like ten minutes.

279
00:21:30,090 --> 00:21:35,592
They have thirty-five million lines of code system and half of that code is test code.

280
00:21:35,592 --> 00:21:41,314
Yes, we didn't have enough tests uh in back in the two thousand.

281
00:21:41,314 --> 00:21:46,077
So they are embarked on a new exponential that they are basically exploiting.

282
00:21:46,077 --> 00:21:49,912
And yes, we do not have enough of those tools.

283
00:21:49,912 --> 00:21:51,406
to navigate through our system.

284
00:21:51,406 --> 00:21:57,644
What we're saying is in a reasonable system you should have thousands at least of things
that are being shown to you.

285
00:21:57,644 --> 00:22:05,437
I and I'm wondering if we're doing something unique that not everyone's doing and this
sort of points to what you're actually suggesting that companies can do to be effective,

286
00:22:05,437 --> 00:22:11,880
or org engineering organizations or individual engineers to be effective, or whether or
not there's still a flaw in our approach.

287
00:22:11,880 --> 00:22:21,304
So I think there's a spectrum on one side, and I think Simon Wardly is to actually get all
the credit for thinking about this, like which side of the spectrum are you on?

288
00:22:21,304 --> 00:22:27,686
Are you building something totally bespoke or is it a commodity Wardly map, right?

289
00:22:28,390 --> 00:22:32,522
and I I feel like for software engineering, there's usually this rule of three.

290
00:22:32,522 --> 00:22:39,475
If you do a thing multiple times, you know, you should think about automating it or uh
refactoring it in some regard.

291
00:22:39,475 --> 00:22:47,308
And so when we have a bug or a problem and we have logs and we have to jump in and do
something manual, we think about well, why did this have to be manual?

292
00:22:47,308 --> 00:22:51,800
Can we build a tool to expose the answer to the question specifically?

293
00:22:51,800 --> 00:22:57,004
And so we have this uh escalation process in how we write software.

294
00:22:57,004 --> 00:22:59,055
We start with um logs.

295
00:22:59,055 --> 00:23:05,686
And if you have an error in your logs, like an all reference exception or a 500, you you
say, well, why did this happen?

296
00:23:05,686 --> 00:23:10,098
And if you don't have the answer to that question, the first step is to start logging
additional stuff.

297
00:23:10,098 --> 00:23:15,919
But I think this is where a lot of other companies get stuck or uh misguided, is that they
end up stopping there.

298
00:23:15,919 --> 00:23:17,980
But we take the next approach.

299
00:23:17,980 --> 00:23:26,708
Inside where those error conditions are, we will wrap that log statement with database
queries or other uh traces where well

300
00:23:26,708 --> 00:23:30,541
improve the log statement in our source code with that additional data.

301
00:23:30,541 --> 00:23:33,813
So the next time that log shows up, it says, oh, here's the problem.

302
00:23:33,813 --> 00:23:37,446
But also you may want to know this was the customer that was impacted.

303
00:23:37,446 --> 00:23:39,107
This is how many users they have.

304
00:23:39,107 --> 00:23:40,758
This is the sort of request they were making.

305
00:23:40,758 --> 00:23:42,738
Here's the other request they made for the last five minutes.

306
00:23:42,738 --> 00:23:49,024
And here's another query that you can automatically run if you're the unfortunate engineer
who got woken up at 2 a.m.

307
00:23:49,024 --> 00:23:52,356
in the morning and has to actually investigate this, just click this link.

308
00:23:52,356 --> 00:23:55,638
It will open a UI and automatically have the query execute.

309
00:23:55,638 --> 00:23:57,048
And then the process continues.

310
00:23:57,048 --> 00:23:59,059
Uh, you know, what's at the result of that query?

311
00:23:59,059 --> 00:24:00,760
Maybe more data, et cetera, et cetera.

312
00:24:00,760 --> 00:24:05,861
Uh we call this sort of like dynamic UIs or dynamic queries for on-call support.

313
00:24:05,861 --> 00:24:10,363
And I'm wondering if that is well aligned with what you're thinking about.

314
00:24:11,098 --> 00:24:20,182
Uh what I'll say is that when your source code is more constrained in the with a bounded
context, the and a smaller size.

315
00:24:20,182 --> 00:24:26,625
You don't need to spend as much time thinking about the complexity that was created by
having the incorrect directory structure.

316
00:24:26,625 --> 00:24:35,979
So in a way, smaller repositories gave us the benefit of having to s getting to sidestep
that particular problem of having the incorrect directory structure because you could

317
00:24:35,979 --> 00:24:37,269
still find whatever you were looking for.

318
00:24:37,269 --> 00:24:39,212
You didn't need to have it perfect.

319
00:24:39,212 --> 00:24:44,465
The question is, is the freedom of having an incorrect directory structure important?

320
00:24:44,465 --> 00:24:51,389
In other words, do people need to have a different directory structure because it
liberates something?

321
00:24:51,389 --> 00:24:56,551
Or is just the case that they were not exactly aware if it should be SRC or source?

322
00:24:56,551 --> 00:25:02,955
Does it need to be the case that the people that are coming up with the new structure not
know what the actual structure should be?

323
00:25:02,955 --> 00:25:05,496
How do I make that answer more readily available?

324
00:25:05,496 --> 00:25:09,438
But there's another one thing that you mentioned before you talked about the rule of
three.

325
00:25:09,750 --> 00:25:14,452
The rule of three says, right, you were saying that if I do it three times that I want to
automate it.

326
00:25:14,633 --> 00:25:19,835
When it comes to, for example, testing, this rule of three is not encouraged.

327
00:25:19,936 --> 00:25:20,696
Not.

328
00:25:20,696 --> 00:25:22,797
No, it says the rule of one, right?

329
00:25:22,797 --> 00:25:29,861
Before I'm gonna do anything, I want to in see if I can capture it in my hypothesis,
right?

330
00:25:30,342 --> 00:25:32,116
And now in order for that to work, right?

331
00:25:32,116 --> 00:25:38,336
The rule of three was made like this because it was still too expensive to do it.

332
00:25:38,338 --> 00:25:42,401
But if I done it three times then it's probably that I'll be doing multi multiple times.

333
00:25:42,401 --> 00:25:50,318
Therefore I need to have the business case before I'm gonna build invest the time to build
the tool so that I admize it over some longer amount of time.

334
00:25:50,318 --> 00:25:51,579
Let's fight about that.

335
00:25:51,579 --> 00:25:54,280
Because I I think I think it's that's totally true.

336
00:25:54,280 --> 00:26:03,426
And we can look at the X KCD on like should I automate based off the amount of time I save
versus the amount of time that would cost to actually perform that activity in ad uh at

337
00:26:03,426 --> 00:26:03,966
infinim.

338
00:26:03,966 --> 00:26:13,792
Uh the I think the rule of three really points to the fact that we don't have a perfect
capture of the requirements and can never fully have enough information.

339
00:26:13,792 --> 00:26:16,736
And it's like that three-standard deviation.

340
00:26:16,736 --> 00:26:25,457
actually tells us that we have enough information so that we know that whatever refactor
or change we make or whatever automation we're putting in place best represents the

341
00:26:25,457 --> 00:26:29,558
reality to a better degree than if there's just the first time.

342
00:26:29,558 --> 00:26:38,874
And if the if the problem didn't appear often enough, then I'm not gonna be building the
tool because it's too expensive to build the tool and it will just increase the time I in

343
00:26:38,874 --> 00:26:41,745
I need to actually solve the problem that I'm paid for, right?

344
00:26:41,745 --> 00:26:55,293
Um however, if we can bring the cost of a tool to be lower that so that the the
constructing the tool and solving the problem is less costly than solving the problem

345
00:26:55,293 --> 00:26:56,864
without building the tool.

346
00:26:57,068 --> 00:27:00,079
Then I will be building tools for every problem.

347
00:27:00,079 --> 00:27:04,020
And some of those tools will be reused.

348
00:27:04,420 --> 00:27:06,480
Some most not.

349
00:27:06,561 --> 00:27:09,221
For example, I give you an example where this happens.

350
00:27:09,221 --> 00:27:15,863
Most of the time when when I ask people, so if I have give you a problem in the database,
what's the first thing you do?

351
00:27:16,203 --> 00:27:17,724
I'm gonna write a query.

352
00:27:17,724 --> 00:27:19,364
But what is a query, right?

353
00:27:19,364 --> 00:27:20,325
What is a query?

354
00:27:20,325 --> 00:27:25,826
It's a little tool that takes lots of data and then compresses into a little smaller data,
right?

355
00:27:25,942 --> 00:27:32,247
It's really fascinating now to just consider coming back to, you know, what needs to
change.

356
00:27:32,247 --> 00:27:39,703
So if I take the same developer and say, Well, what's I have a problem in the system, what
you're gonna what what would you like in this code, what you will do is that they will

357
00:27:39,703 --> 00:27:40,854
start reading.

358
00:27:40,854 --> 00:27:41,795
Now what's the difference?

359
00:27:41,795 --> 00:27:44,977
Why why one behavior in one place and the other one in the other one?

360
00:27:44,977 --> 00:27:46,438
Because the code is also data.

361
00:27:46,438 --> 00:27:52,644
Um now let's take a look like a quick look at the interface that they do the activities
through.

362
00:27:52,644 --> 00:27:55,894
So when they when they are exposed to the database problem, what do they see?

363
00:27:55,894 --> 00:28:00,339
It's a database tool with a big query at the top and the data at the bottom.

364
00:28:00,339 --> 00:28:01,210
So what you're gonna do?

365
00:28:01,210 --> 00:28:04,984
Well, you're gonna write the query at the top and you're gonna see the data at the bottom.

366
00:28:04,984 --> 00:28:06,806
It's very interesting, right?

367
00:28:06,806 --> 00:28:11,312
When you switch to to looking into investigating the code, what do you have?

368
00:28:11,312 --> 00:28:12,573
You have a giant text editor.

369
00:28:12,573 --> 00:28:13,974
So what are you gonna do?

370
00:28:13,974 --> 00:28:16,322
No, use the search use the search functionality.

371
00:28:16,322 --> 00:28:18,088
With a re with a magic regex.

372
00:28:18,088 --> 00:28:21,240
You regex you re regex to find exactly your problem.

373
00:28:21,240 --> 00:28:21,470
Right.

374
00:28:21,470 --> 00:28:22,932
Yeah, yeah, yeah, for sure.

375
00:28:22,932 --> 00:28:28,070
Uh because why are you n why are not people rejexing through their database, by the way?

376
00:28:28,570 --> 00:28:29,197
Right?

377
00:28:29,197 --> 00:28:31,459
Because because it doesn't make sense.

378
00:28:31,459 --> 00:28:41,468
Why would you do a way reject means every line of code has the same meaning, same
structure as any other any other line of code, which is absolutely not true.

379
00:28:41,468 --> 00:28:43,894
Code is incredibly structured.

380
00:28:43,894 --> 00:28:48,250
And doing it and by the way, the editor that you're using already has all that
information.

381
00:28:48,250 --> 00:28:48,530
Yeah.

382
00:28:48,530 --> 00:28:58,112
But it's hidden in that box that you're not allowed to open and God forbid you will
actually, you know, get good at what you're doing and not buy off the shelf tools that

383
00:28:58,112 --> 00:28:59,650
actually don't solve your problem.

384
00:28:59,650 --> 00:29:10,585
I think compiler there is the biggest challenge that a lot of the information that you
wanna get at is only exposed after it's been com compiled into a binary and gone through

385
00:29:10,585 --> 00:29:12,976
the assembly and linking process.

386
00:29:12,976 --> 00:29:20,970
And it's only recent languages that have been able to reflect this back into the
development experience to have the answer right away.

387
00:29:20,970 --> 00:29:29,334
And I'd say, you know, realistically the thing that is always missing for me is like
hitting F twelve or F eleven or whatever on a keyword or class or whatever and being able

388
00:29:29,334 --> 00:29:29,634
to jump

389
00:29:29,634 --> 00:29:40,428
in to understanding what that is, but I I totally agree in the in the aspect where what
information is exposed in that way is always much less than I want it to be.

390
00:29:40,440 --> 00:29:43,411
So something that would take the Excel and transpile it into this.

391
00:29:43,411 --> 00:29:48,672
So you would say, yes, because uh only the interpreter has it, I don't have access.

392
00:29:48,672 --> 00:29:55,394
Yeah, but we could still build a tool that reverse engineered that whole language and puts
it together within the context of the entire pipeline.

393
00:29:55,394 --> 00:30:03,357
By the way, on the other side of the spectrum, there was a uh a low-code platform with
called Pega and they had twenty two thousand scripts in there.

394
00:30:03,357 --> 00:30:04,707
Twenty two thousand scripts.

395
00:30:04,707 --> 00:30:08,438
Pega is advertised as, you know, built faster and they built fast.

396
00:30:08,438 --> 00:30:12,211
Except that they built twenty two thousand scripts and now they had to reason about them.

397
00:30:12,211 --> 00:30:22,209
But Pega has no tool to give you any dependency analysis and definitely no performance of
anything and definitely no data flows that goes going through those.

398
00:30:22,209 --> 00:30:23,971
So we had to now reverse engineer that one.

399
00:30:23,971 --> 00:30:28,965
But uh Pega doesn't even have an expert properly uh of their source code.

400
00:30:28,965 --> 00:30:33,198
So we took the backup format and reverse engineered the whole thing out of that.

401
00:30:33,198 --> 00:30:36,821
So no, the compilers are absolutely not the blockers.

402
00:30:36,821 --> 00:30:38,218
Uh it is

403
00:30:38,218 --> 00:30:43,982
It is absolutely possible to build tools much, much faster than people think it is.

404
00:30:44,764 --> 00:30:46,625
and there is no excuse not to build those.

405
00:30:46,625 --> 00:30:51,448
The only excuse we find is because we refuse to talk about the problem.

406
00:30:51,669 --> 00:30:56,303
And because we refuse to talk about the problem, we do not observe the opportunities that
are all around us.

407
00:30:56,303 --> 00:30:57,528
They are exist.

408
00:30:57,528 --> 00:31:05,681
So I think this is where we're a little bit unique because we try to automate everything
that we do in to a ridiculous degree.

409
00:31:05,681 --> 00:31:15,747
Like when we get a support ticket where our customer is asking a question, that answer we
think about not, we have some sort of giant knowledge base that we can throw an LM at and

410
00:31:15,747 --> 00:31:21,352
it can dynamically generate the answer, but realistically, is there a deterministic tool
that allows us to

411
00:31:21,352 --> 00:31:22,373
directly answer it.

412
00:31:22,373 --> 00:31:24,925
And if we don't have that, then we go rule of three.

413
00:31:24,925 --> 00:31:32,101
The third time we get a support ticket asking about a piece of information, we end up
stacking up the internal process.

414
00:31:32,101 --> 00:31:33,953
And we have one for this actually.

415
00:31:33,953 --> 00:31:37,796
We start by adding some code to our existing repository for our service.

416
00:31:37,796 --> 00:31:45,632
I I think an example is like uh we frequently got a request like when a new user gets into
our login system for that we build for our customers.

417
00:31:46,408 --> 00:31:49,389
why doesn't that user link to an existing user?

418
00:31:49,389 --> 00:31:53,560
Like, maybe the email address is different or maybe the identity provider is a little bit
different, et cetera.

419
00:31:53,560 --> 00:31:55,601
And we get that question pretty frequently.

420
00:31:55,601 --> 00:31:58,612
And on the backside, we started building up an answer to this.

421
00:31:58,612 --> 00:32:02,673
We started by making the right database queries so that we could quickly answer.

422
00:32:02,673 --> 00:32:05,824
But then when we got more requests, we said, well, a database query isn't sufficient.

423
00:32:05,824 --> 00:32:11,530
We should have multiple database queries with multiple, you know, data alignment and
understanding how it

424
00:32:11,530 --> 00:32:13,031
it is cohesive together.

425
00:32:13,031 --> 00:32:14,221
And then that wasn't sufficient.

426
00:32:14,221 --> 00:32:16,652
So then we expose it on an endpoint that could be called.

427
00:32:16,652 --> 00:32:20,924
And for security reasons, of course, you know it has to have the right uh authorization
and whatnot.

428
00:32:20,924 --> 00:32:24,475
And then we took the last step of exposing it back to our end users.

429
00:32:24,475 --> 00:32:28,587
And I feel like we have this capability to think in this way.

430
00:32:28,587 --> 00:32:32,969
And I'm just wondering why isn't that no one else is doing this?

431
00:32:32,969 --> 00:32:36,601
Is it that the ROI just doesn't seem like it's high enough?

432
00:32:36,601 --> 00:32:39,302
Is that a short-term problem that people are too focused?

433
00:32:39,302 --> 00:32:40,426
Or is it like

434
00:32:40,426 --> 00:32:47,749
It requires a deep under I because I feel like a lot of what you're talking about requires
a deep understanding of the technical challenges.

435
00:32:47,749 --> 00:32:55,383
And I the one thing I think you really bring to this conversation is this idea that
building new tools is is incredibly cheap.

436
00:32:55,383 --> 00:33:00,085
And with LLMs, it's disposable software that should be almost free to make happen.

437
00:33:00,085 --> 00:33:07,238
So I feel like if anything, we're at the exact right point in time to say that everyone
should be doing this to solve every single problem that they have.

438
00:33:07,638 --> 00:33:10,819
LLMs essential for this, the answer is for sure not.

439
00:33:10,819 --> 00:33:12,090
Yeah.

440
00:33:12,090 --> 00:33:13,681
The LLMs, can LLMs help?

441
00:33:13,681 --> 00:33:14,491
For sure, yes.

442
00:33:14,491 --> 00:33:14,761
Right.

443
00:33:14,761 --> 00:33:17,498
So but just don't use the LLM like you were saying.

444
00:33:17,498 --> 00:33:17,862
Right?

445
00:33:17,862 --> 00:33:28,417
You don't ask the LLM, tell me why this is the case, and then it And then you get your
proposal and then you can investigate the tool because now the bias of because whatever

446
00:33:28,417 --> 00:33:29,737
the LLM says, it's an opinion.

447
00:33:29,737 --> 00:33:32,458
It's just like how your colleague says it's an opinion.

448
00:33:32,458 --> 00:33:36,546
So so you take the opinion and then you externalize

449
00:33:36,546 --> 00:33:39,588
The opinion, you externalize the bias, you can investigate it.

450
00:33:39,588 --> 00:33:46,483
You you can investigate it yourself and then decide whether or not you're gonna delegate
um to that deterministic tool.

451
00:33:46,483 --> 00:33:49,115
Right now, you mentioned why is it the case.

452
00:33:49,115 --> 00:33:55,749
So the geekest detail in the whole of software engineering, that is how a commit how how
does code go from a commit to a deployment?

453
00:33:55,749 --> 00:33:57,050
Like, who cares?

454
00:33:57,050 --> 00:34:05,185
Like the is like this is like the most boring possible thing, uh, in in the whole of
software, right?

455
00:34:05,185 --> 00:34:06,284
But it turns out

456
00:34:06,284 --> 00:34:13,979
Then if you go from manual, a manual solution to an automated solution, you can create
trillion dollars worth of economy, which is like, wow.

457
00:34:13,979 --> 00:34:16,381
Um that's kind of interesting, right?

458
00:34:16,381 --> 00:34:27,339
Um but if the if in your organization that now does manual deployment and you just make it
now automatic completely overnight, but you don't change anything else, you won't get you

459
00:34:27,339 --> 00:34:30,010
will get some benefits, but they will be marginal.

460
00:34:30,291 --> 00:34:33,666
If you want to get to the to the real leaps,

461
00:34:33,666 --> 00:34:36,807
You have to change all the processes that happen on top of it.

462
00:34:36,807 --> 00:34:47,151
Because basically these manual delay this deployments they introduce delays and delays
then reper uh they had they created unnecessary processes that poof should disappear

463
00:34:47,151 --> 00:34:49,452
basically overnight when this transition happens.

464
00:34:49,452 --> 00:35:01,627
Because now all of a sudden the deployment becomes from uh a manual, a custom uh customly
produced artifact to a commodity and therefore coming back to the wording map, it creates

465
00:35:01,627 --> 00:35:03,028
a new whole new space on the top left.

466
00:35:03,028 --> 00:35:03,678
Okay.

467
00:35:03,678 --> 00:35:17,042
In reading the book and also our conversation here, that one of the problems I think we
have is that the construct that we have to store our tools or our toolkits is so inflated

468
00:35:17,042 --> 00:35:23,934
with things that have nothing to do with the process in which we evaluate uh or get
answers to questions that we ask.

469
00:35:23,934 --> 00:35:33,282
I think of systems where you have or an organization that has hundreds of thousands of
scripts that they can run, finding the right script at that moment is a challenge.

470
00:35:33,282 --> 00:35:35,163
uh in a lot of different ways.

471
00:35:35,163 --> 00:35:44,168
And even as you think about like how can we optimize this to find the right script, you're
building a system on top of that, which has its own complexities and and challenges in a

472
00:35:44,168 --> 00:35:45,029
way.

473
00:35:45,029 --> 00:35:48,051
And it's like in a way turtles all the way down.

474
00:35:48,051 --> 00:35:51,012
And I don't think it's getting any better in any regard.

475
00:35:51,012 --> 00:36:01,238
And then you end up with whole organizations and companies that are dedicated just to
building the the tools to be able to interrogate the the database or

476
00:36:01,356 --> 00:36:04,769
be provided to engineering team to be able to answer the questions.

477
00:36:04,769 --> 00:36:12,787
And I I don't know that we have a very mature um response to how to deal with package
management, I think, in the world.

478
00:36:12,787 --> 00:36:13,468
I mean it's not great.

479
00:36:13,468 --> 00:36:21,365
There's still like dependency change and diamond dependency problems and supply chain
attacks and and uh S bombs, you know, like the bill of materials.

480
00:36:21,365 --> 00:36:26,422
Uh but I think when it comes to the tools, I think there's like a whole space here

481
00:36:26,422 --> 00:36:32,770
that is so poorly managed as far as how to do this, other than, yeah, we have a repository
with a bunch of things in it.

482
00:36:32,770 --> 00:36:33,991
That that that's right.

483
00:36:33,991 --> 00:36:45,375
Um what if I can treat the data the data pipeline problem to be the same as the
observability problem, to be the same as the email linking to the user problem?

484
00:36:46,276 --> 00:36:57,980
because this this goes quite against the the industry where the data pipeline problem is a
billion dollar vertical, uh and the observability problem is another billion dollar pro

485
00:36:57,980 --> 00:37:02,092
vertical and the other ones and there'll be all sorts of verticals with billion dollars

486
00:37:02,092 --> 00:37:07,325
worth of tool vendors um offering you off-the-shelf solutions.

487
00:37:08,226 --> 00:37:16,492
and um our view is well all based all of these all of these propositions are based on the
idea that tools are expensive to build.

488
00:37:16,492 --> 00:37:28,060
Well what if tools are not expensive to build and that you can have an environment where
all you can approach arbitrary problems in the same way or in a uniform fashion and

489
00:37:28,060 --> 00:37:30,572
therefore you can internalize the skill once

490
00:37:30,858 --> 00:37:35,318
And then build a uniform environment where you're just hosting all of these perspectives.

491
00:37:35,318 --> 00:37:44,184
What is the smallest change to the software development workflow or reality even that
would cause us to be pushed in the right direction?

492
00:37:44,184 --> 00:37:55,372
Is this a matter of having an integrated development environment like our our IDE or just
say like you know Visual Studio Code or something else, whatever using writer, et cetera,

493
00:37:55,372 --> 00:38:01,476
to have some fundamental difference on how information is exposed to a technical end user?

494
00:38:01,476 --> 00:38:04,498
Or is there something fundamental that we ha like a is there a

495
00:38:04,720 --> 00:38:08,516
a change in the way we build software products that has to be considered?

496
00:38:08,516 --> 00:38:09,958
Is there, you know, the connection?

497
00:38:09,958 --> 00:38:11,320
Is it some other primitive?

498
00:38:11,320 --> 00:38:16,867
Like what what is the first thing that really would need to change and we could see the
output from?

499
00:38:16,867 --> 00:38:22,050
Like not just, change your mindset, but specifically some artifact that would prove we're
on the right place.

500
00:38:22,050 --> 00:38:29,476
We want an IDE, we need that integrated experience, but the current ones fail quite
dramatically.

501
00:38:29,476 --> 00:38:31,498
What I'm proposing is not an increment.

502
00:38:31,498 --> 00:38:37,942
Spend fifteen minutes per day, you know, during your coffee break, just starting the
conversation.

503
00:38:38,003 --> 00:38:39,824
How do we read code?

504
00:38:39,824 --> 00:38:40,505
That's it.

505
00:38:40,505 --> 00:38:41,946
This is where it starts.

506
00:38:41,946 --> 00:38:47,170
Make it a subject of conversation before you even accept that this is a problem.

507
00:38:47,914 --> 00:38:52,177
There's no way we're gonna make any like no amount of tooling will change anything.

508
00:38:52,177 --> 00:38:54,439
That's the simplest change.

509
00:38:54,559 --> 00:39:08,420
And today, right, especially we can take this opportunity that the LM uh brought, uh, to
actually make it the subject of conversation because finally people recognize that they

510
00:39:08,420 --> 00:39:14,174
have to read code for most of their time and that they also accept that it is annoying.

511
00:39:14,350 --> 00:39:15,635
But they shouldn't be reading codes.

512
00:39:15,635 --> 00:39:19,714
And yes, they shouldn't be reading code to the ex to the extent that they do I

513
00:39:19,714 --> 00:39:29,778
I think, you know, as I even going further and saying, and maybe this is too much of a
shill, that the the first few chapters of the moldable development, the rewildling

514
00:39:29,778 --> 00:39:39,722
software engineering book that you've written with Simon Wardley, uh is definitely an can
be provided as an eye-opening experience for how how wrong, you know, some companies are

515
00:39:39,722 --> 00:39:40,214
really doing it.

516
00:39:40,214 --> 00:39:42,223
And you have some like absolute great examples in there.

517
00:39:42,223 --> 00:39:48,236
And like, you know, the example that I'm thinking about is the one with the silicon die
hardware manufacturer that that you had.

518
00:39:48,236 --> 00:39:54,159
So that that was actually the first time when we saw this idea work at scale.

519
00:39:54,159 --> 00:39:55,897
So th this the story goes like this.

520
00:39:55,897 --> 00:39:58,200
it's one of the companies, reasonably large companies.

521
00:39:58,200 --> 00:40:00,711
I mean, they are not allowed, for example, to trade with China today.

522
00:40:00,711 --> 00:40:08,295
So uh reasonably important and they build wafer technology, uh so the factories that
produce uh that manipulate wafers.

523
00:40:08,295 --> 00:40:16,738
They basically build these rooms and they they work with with uh, you know, Nobel
laureates in chemistry and physics and this is is like an amazing place to be and

524
00:40:16,738 --> 00:40:17,188
To visit.

525
00:40:17,188 --> 00:40:20,311
It's absolutely a uh an experience.

526
00:40:20,311 --> 00:40:23,904
But um these robots that take these rooms, right?

527
00:40:23,904 --> 00:40:33,191
They have these uh robot arms that are manipulating you know often like what is worth like
a million dollars worth of material inside.

528
00:40:33,251 --> 00:40:42,889
And um they have to be con distribu they have to be controlled in some way, and they use a
distributed system for that, and that distributed system used you know relied on a

529
00:40:42,889 --> 00:40:43,600
protocol.

530
00:40:43,600 --> 00:40:46,780
And they didn't know how that protocol worked anymore.

531
00:40:48,015 --> 00:40:58,901
and um they gave us this you know challenge and we we were doing it as a research project
uh for them and so they gave us the sources and the idea was you know take take a mesh of

532
00:40:58,901 --> 00:41:08,426
of nodes written in their technology and then basically put another another node written
in another language to show that you can communicate following the same protocol.

533
00:41:08,446 --> 00:41:15,884
And um so a month later, which is usually how, you know, session like meetings are set,
you know, you have a cadence of things.

534
00:41:15,884 --> 00:41:18,775
So a month later people say, Okay, are you ready to start?

535
00:41:19,956 --> 00:41:21,677
and then we said, Well, we are done.

536
00:41:21,677 --> 00:41:22,978
And so, what do you mean you're done?

537
00:41:22,978 --> 00:41:29,482
So we showed them, Yeah, here's a new note, and we, you know, it they communicate and say,
Yeah, but how could you be done?

538
00:41:29,482 --> 00:41:36,285
Because we had this bet inside or bets inside the company saying that there's no way you
can do it in a year.

539
00:41:36,285 --> 00:41:39,507
They gave us a year to look into the problem.

540
00:41:39,587 --> 00:41:41,408
because we have no idea how we could do it.

541
00:41:41,408 --> 00:41:42,349
And so, yeah.

542
00:41:42,349 --> 00:41:44,320
Um and then he asked us, so how did you do it?

543
00:41:44,320 --> 00:41:45,130
Well

544
00:41:45,250 --> 00:41:51,204
Well, basically we spend three weeks building a debugger for um to figure out your
protocol.

545
00:41:51,204 --> 00:41:57,908
Uh and then uh another week to actually, you know, uh implement the the rest of the thing.

546
00:41:57,908 --> 00:41:58,999
And and that's it.

547
00:41:58,999 --> 00:42:00,300
It's it's that simple.

548
00:42:00,300 --> 00:42:05,323
The solution is much less expensive than people believe it is.

549
00:42:05,323 --> 00:42:07,224
It's very, very powerful.

550
00:42:07,224 --> 00:42:11,136
Um computation can be powerful.

551
00:42:11,170 --> 00:42:17,224
But you cannot perceive anything inside the computation except through a tool, which makes
the tool fundamental.

552
00:42:17,224 --> 00:42:20,395
If you change the tool, you change your perception.

553
00:42:20,395 --> 00:42:23,437
If you change your perception, you change the way you think.

554
00:42:23,437 --> 00:42:27,670
And applies, by the way, also to to the new world of AI, right?

555
00:42:27,670 --> 00:42:31,962
Don't don't use AI through uh through standard tools.

556
00:42:32,050 --> 00:42:40,130
I there's a lot you know, there's a lot of great arguments against uh LLMs that I just
don't think people bring up and this one is one of one of my favorites.

557
00:42:40,130 --> 00:42:41,371
For for sure.

558
00:42:41,371 --> 00:42:48,985
the other one that comes to mind fundamentally is that, yeah, humans are computational
units and so are LLMs.

559
00:42:48,985 --> 00:42:51,917
So therefore anything a human can do, an LM can do.

560
00:42:51,917 --> 00:42:53,948
And so obviously LLMs are gonna be better in the future.

561
00:42:53,948 --> 00:42:58,921
And I'm like, you don't even know where the future of humans are going, let alone that
they're computational units.

562
00:42:58,921 --> 00:43:02,053
So it's like the whole argument is flawed, honestly.

563
00:43:02,053 --> 00:43:07,394
Uh but before we get to too much philosophy into into LMs, I think maybe.

564
00:43:07,394 --> 00:43:12,167
This is the moment we we we uh we shift over to the end of the episode.

565
00:43:14,029 --> 00:43:15,300
let's move over to picks.

566
00:43:15,300 --> 00:43:18,178
So Tudor, what did you bring for the audience today?

567
00:43:18,178 --> 00:43:29,083
I I spent a week uh in Normandy with my family and we visited the D-Day beaches and it was
very impressive.

568
00:43:29,083 --> 00:43:36,766
Um of course I mean the the human side of things is the thing that that captures the you
know the attention there.

569
00:43:36,766 --> 00:43:46,470
But if you if you spend enough time there, um there are all sorts of the y you get to see
the the magnitude of the engineering

570
00:43:47,162 --> 00:43:56,508
um effort, the engineering and research effort that had to all come together for this
massive undertaking to actually be possible.

571
00:43:56,849 --> 00:44:00,362
I found that to be mind-boggling.

572
00:44:00,362 --> 00:44:08,928
We I knew about some of it, but taking the time to be there and see it, uh I was
unprepared um for that impression.

573
00:44:08,928 --> 00:44:15,158
Like and all the way to the details of you know, how do you build um a bridge?

574
00:44:15,158 --> 00:44:26,305
of a water, which was not a thing that was possible before, um, in in a short amount of
time, with not two skilled people or with without having two sp super specialized um

575
00:44:26,305 --> 00:44:34,614
equipment um there, or how do you build the whole uh artificial port in a matter of days,
right?

576
00:44:34,614 --> 00:44:38,753
Uh the all of these questions were just not known just a couple of years before.

577
00:44:38,753 --> 00:44:39,133
Yeah.

578
00:44:39,133 --> 00:44:41,610
I found that to be absolutely impressive.

579
00:44:41,610 --> 00:44:50,518
And is also a a really interesting testament of what we can do as a human species when we
put our mind to uh to it.

580
00:44:50,518 --> 00:44:53,979
I think there's something actually quite amazing uh about the beaches.

581
00:44:53,979 --> 00:45:04,002
And uh the one thing that comes to mind is can you like not not you, because obviously you
visited there and you probably know the answer, but just you may know that landing on the

582
00:45:04,002 --> 00:45:07,023
beaches was basically the end of the war.

583
00:45:07,103 --> 00:45:08,023
But why?

584
00:45:08,023 --> 00:45:12,985
Why what what did how did landing on the beaches in Normandy cause the end of the war?

585
00:45:12,985 --> 00:45:14,405
Like what is the connection there?

586
00:45:14,405 --> 00:45:16,886
How like Germany's on the other side.

587
00:45:16,994 --> 00:45:18,074
What does that have to do?

588
00:45:18,074 --> 00:45:20,645
Wha how does that impact the army realistically?

589
00:45:20,645 --> 00:45:29,038
And I find that as like as soon as you ask that question, there's so many gaps there in
the can on on what's actually happened from a s not just military strategy, but also human

590
00:45:29,038 --> 00:45:33,479
strategy that you really have to wonder, like, what do you actually know about about the
history?

591
00:45:33,479 --> 00:45:42,742
And I feel like it is one thing that's often left out of the history books that is
actually talked about the not just the scale, but uh the feats that had to go in and why

592
00:45:42,742 --> 00:45:43,938
it was so important.

593
00:45:43,938 --> 00:45:48,404
for that l location to be picked in order to counterattack.

594
00:45:48,404 --> 00:45:50,156
So I love I love your pick.

595
00:45:50,156 --> 00:45:59,448
is there like a particular set of museums or location or tour map that you would suggest
to people that I maybe we can share a link for in case anyone else is interested.

596
00:45:59,448 --> 00:46:03,831
time it took a week to visit the you know the entirety.

597
00:46:03,831 --> 00:46:15,104
I think the the size uh it's only I don't I don't know if there's any one particular thing
that was uh you know uh a must visit.

598
00:46:15,104 --> 00:46:19,133
I I just found the the size of it.

599
00:46:19,178 --> 00:46:20,874
I just visited everything.

600
00:46:21,249 --> 00:46:23,376
How how do you know what everything is?

601
00:46:23,376 --> 00:46:26,742
Like I what did you do like a lot of background research first?

602
00:46:26,767 --> 00:46:36,171
this they're they're good sites and so the the the official French sites are are very
thorough.

603
00:46:36,171 --> 00:46:37,543
So you get an idea.

604
00:46:37,543 --> 00:46:39,045
You get a good idea about that.

605
00:46:39,045 --> 00:46:39,788
uh

606
00:46:39,788 --> 00:46:40,419
okay.

607
00:46:40,419 --> 00:46:42,160
I I I I love it.

608
00:46:42,681 --> 00:46:45,233
well mine my pick is gonna pale in comparison.

609
00:46:45,233 --> 00:46:52,288
There were definitely a few different things I was inspired to to share and maybe there'll
be there'll be future future picks related on the topic.

610
00:46:52,288 --> 00:47:03,778
But for this I have something totally unrelated and that is the well, I don't know if I
can really claim it's the the condiment, but I was a huge fan of the Hoi Fong Sriracha,

611
00:47:03,778 --> 00:47:07,921
the you know, red red sauce with the rooster on it with the green top.

612
00:47:07,921 --> 00:47:09,042
Uh

613
00:47:09,042 --> 00:47:19,205
bottled in California and there was actually such a controversy that happened basically at
the pen pandemic where it got worse.

614
00:47:19,205 --> 00:47:21,365
It disappeared and got worse.

615
00:47:21,365 --> 00:47:23,326
And it's never really recovered.

616
00:47:23,546 --> 00:47:33,709
And uh I don't want to spoil too much about what actually happened there, but I will say
they were getting their peppers from another uh from a grower and they had a a pretty

617
00:47:33,709 --> 00:47:34,809
sweet deal.

618
00:47:34,809 --> 00:47:38,206
And after and they switched they switched growers basically.

619
00:47:38,206 --> 00:47:45,714
And the original grower of the jalapenos that were being used for sriracha b started
bottling their own sauce.

620
00:47:45,714 --> 00:47:49,812
And so if you want the authentic sriracha, uh it's out there.

621
00:47:49,812 --> 00:47:53,943
Uh Underwood Ranch, that's the you know, that's the company that was making it.

622
00:47:53,943 --> 00:47:59,208
And go to town and there'll be a link in the description because I absolutely love the
original sriracha.

623
00:47:59,208 --> 00:48:01,450
That was that was my go to hot sauce.

624
00:48:01,758 --> 00:48:02,835
Good to know.

625
00:48:03,638 --> 00:48:04,698
Yeah.

626
00:48:04,759 --> 00:48:12,218
I I mean I I am I am super into hot sauces and I could probably have ten more ten more
picks of of different hot sauces.

627
00:48:12,218 --> 00:48:23,204
Uh and I've had I've had to go out and sort of uh expand my repertoire after after the
pandemic because it just it had so far gone downhill that it's you know, sometimes not

628
00:48:23,204 --> 00:48:24,696
even worth getting.

629
00:48:24,696 --> 00:48:32,642
And it's actually the reason why you see so many counterfeits show up now, uh in quite po
popular style because uh

630
00:48:32,642 --> 00:48:36,804
I think the biggest one is like there's like golden dragon and then there's flying goose
is the other one.

631
00:48:36,804 --> 00:48:42,432
They're always like a adjective with a or gerund with a um with an animal associated with
it.

632
00:48:42,432 --> 00:48:44,704
And they're just not they're they're not the same.

633
00:48:44,970 --> 00:48:46,114
I understand.

634
00:48:47,916 --> 00:48:54,258
Well, uh thank you Tudor for coming on and walking us through this whole new mental model
of software development.

635
00:48:54,258 --> 00:48:59,318
I I think it's great and something that everyone can go and check out and read up more on.

636
00:48:59,318 --> 00:49:02,232
No, thank you thank you for your uh thank you for your time.

637
00:49:02,232 --> 00:49:03,495
I really enjoyed it.

638
00:49:03,756 --> 00:49:08,056
Yeah, so that's uh multiple development and it's rewilding engineering.

639
00:49:08,056 --> 00:49:12,921
So multiple development.com is the site where you can read about rewilding software
engineering.

640
00:49:12,921 --> 00:49:14,567
That's the book that Simon and I are writing.

641
00:49:14,567 --> 00:49:15,340
I know

642
00:49:15,340 --> 00:49:19,354
The links for this will be will be in the description for the episode for anyone that's
interested.

643
00:49:19,354 --> 00:49:27,782
And you may have spotted a hint that in the future we're hoping to have an episode with an
expert on uh formal verification of software, which I think is very interesting and

644
00:49:27,782 --> 00:49:31,746
required, but we're still looking for the appropriate expert to come on and talk with us
about that.

645
00:49:31,746 --> 00:49:38,533
So if you're listening to this episode and know the right person, please forward them
along so we can get do a deep dive on that.

646
00:49:38,533 --> 00:49:39,816
And I just will

647
00:49:39,816 --> 00:49:45,970
I just want to thank all of the listeners for tuning in for this week's episode and
hopefully we'll see you all back again.