1
00:00:07,896 --> 00:00:10,367
Welcome back to Adventures in DevOps.

2
00:00:10,367 --> 00:00:15,570
we brought in a real principal engineer, not just one only in name, as we get up to some
things.

3
00:00:15,570 --> 00:00:18,021
Justin Abrams, principal engineer at Parlow.

4
00:00:18,021 --> 00:00:18,531
Welcome.

5
00:00:18,531 --> 00:00:19,612
Thank you so much for having me.

6
00:00:19,612 --> 00:00:29,127
And I see a large set of experiences, Google's, AWS's, Walmarts, and like a whole bunch of
companies in between, as well as migration from like East Coast, US, West Coast, and to

7
00:00:29,127 --> 00:00:29,427
Europe.

8
00:00:29,427 --> 00:00:31,704
I I mean that's just like someone after my own heart.

9
00:00:31,704 --> 00:00:35,767
Yeah, the way that I'd describe it is that I've worked for many Globocorps.

10
00:00:35,767 --> 00:00:37,949
and I've worked for tiny startups you've never heard of.

11
00:00:37,949 --> 00:00:41,442
My last two jobs have been the middlest companies I've ever worked at.

12
00:00:41,442 --> 00:00:44,514
I've mostly spent my time in in Globocorps.

13
00:00:44,514 --> 00:00:47,487
That's where I've am the most normal.

14
00:00:47,487 --> 00:00:49,248
I the I the structure I appreciate.

15
00:00:49,248 --> 00:00:51,576
And middle sized companies, it's a learning process.

16
00:00:51,576 --> 00:00:54,518
I found middling size companies the most easy to jump into.

17
00:00:54,518 --> 00:01:02,135
They don't have such strong regulatory or policy driven approaches and they're not like we
have no idea what we're doing as a startup.

18
00:01:02,135 --> 00:01:06,488
We'll just throw in random tools and just complain about how much everything costs.

19
00:01:06,488 --> 00:01:08,802
Really like to have a scoreboard and I would like to win the game.

20
00:01:08,802 --> 00:01:11,026
Big companies know what the scoreboard is.

21
00:01:11,026 --> 00:01:14,072
Could you please develop us a plan to get this number three numbers bigger?

22
00:01:14,072 --> 00:01:17,324
You're like, Yes, I can absolutely come up with a plan on how to get this number three
numbers bigger.

23
00:01:17,324 --> 00:01:20,347
being able to go after something that's trackable, right?

24
00:01:20,347 --> 00:01:24,941
Rather than let's just do random things and and pray and maybe it will work out all in the
end.

25
00:01:24,941 --> 00:01:26,926
How well does that translate to engineering though?

26
00:01:26,926 --> 00:01:31,439
So during my time at eBay, my job was to like help make software delivery a competitive
advantage.

27
00:01:31,439 --> 00:01:34,470
And that was meeting with people all all day.

28
00:01:34,470 --> 00:01:36,332
Like thirty five hours a week I was in meetings.

29
00:01:36,332 --> 00:01:45,687
I was hooked into the six hundred engineers that were in the payments group and my job was
to like help them deploy more, help them have more effective software delivery, lower

30
00:01:45,687 --> 00:01:47,516
change failure rate, things like that.

31
00:01:47,516 --> 00:01:56,331
through the course of that work I'd moved them for to have six times more deployments and
seventy two percent of their applications were deploying automatically to production with

32
00:01:56,331 --> 00:01:58,062
zero human involvement after merge.

33
00:01:58,062 --> 00:02:00,493
So like that's an example of like number go up.

34
00:02:00,493 --> 00:02:01,846
Number go up, feel good.

35
00:02:01,846 --> 00:02:03,704
I guess there was a question of so number went up.

36
00:02:03,704 --> 00:02:06,105
Did everyone feel good, you know, with those changes?

37
00:02:06,105 --> 00:02:06,488
Yeah.

38
00:02:06,488 --> 00:02:06,838
Yeah, yeah.

39
00:02:06,838 --> 00:02:09,702
No, it was um it was a great initiative.

40
00:02:09,702 --> 00:02:13,576
It was a lot of nervousness in the beginning of what could possibly go wrong.

41
00:02:13,576 --> 00:02:14,708
Answer everything.

42
00:02:14,708 --> 00:02:16,870
Okay, let's talk about what could go wrong.

43
00:02:16,870 --> 00:02:18,052
Okay, that could go wrong.

44
00:02:18,052 --> 00:02:18,572
Okay, great.

45
00:02:18,572 --> 00:02:20,014
How do we mitigate that?

46
00:02:20,014 --> 00:02:21,155
Okay, cool.

47
00:02:21,155 --> 00:02:22,477
Let's that that's a workout.

48
00:02:22,477 --> 00:02:23,678
Let's go do that.

49
00:02:23,678 --> 00:02:25,140
Okay, now what could go wrong?

50
00:02:25,140 --> 00:02:26,978
And kind of iteratively improving.

51
00:02:26,978 --> 00:02:36,071
Especially when things benefit from having the metric, like tracking the, you know, the
Dora stats, it's much easier to actually target that and and create an initiative around

52
00:02:36,071 --> 00:02:38,962
that rather than even having an argument of like, what is Dora?

53
00:02:38,962 --> 00:02:39,852
Should we follow it?

54
00:02:39,852 --> 00:02:41,442
Is it even relevant in the first place?

55
00:02:41,442 --> 00:02:45,203
Like is six hundred engineers for payments, that sounds like a lot of people.

56
00:02:46,404 --> 00:02:48,544
Is that the right number of people to have for for payments?

57
00:02:48,544 --> 00:02:54,048
I mean, maybe in my naive perspective, and I've never worked in any sort of payments
solution, like that just seems like

58
00:02:54,048 --> 00:02:57,262
An absurd number of engineers to dedicate to a single area.

59
00:02:57,262 --> 00:02:59,223
There are many payment providers.

60
00:02:59,223 --> 00:03:02,804
They're not just like accepting only PayPal and s and Stripe.

61
00:03:02,804 --> 00:03:03,694
Like that's not their thing.

62
00:03:03,694 --> 00:03:07,806
They will take your real credit card number and PCI comes with that.

63
00:03:07,986 --> 00:03:12,018
And now they have all whole fraud vectors they need to be aware of.

64
00:03:12,018 --> 00:03:18,476
there's information about how we not only receive money, but also it's because it's a two
sided marketplace, you also have to pay people.

65
00:03:18,476 --> 00:03:24,860
It's like one of those things where when you get to the details, it's like, well, yeah,
each one of those payment options probably has a team dedicated to just doing that sort of

66
00:03:24,860 --> 00:03:25,461
integration.

67
00:03:25,461 --> 00:03:29,844
Maybe a team has like two integrations or three, but then you start going down the list.

68
00:03:29,844 --> 00:03:32,635
And yeah, the the platform aspect, true.

69
00:03:32,716 --> 00:03:33,667
that's definitely another one.

70
00:03:33,667 --> 00:03:38,080
Like actually having them pay out people is is something that a lot of platforms don't
actually have to deal with.

71
00:03:38,080 --> 00:03:41,042
So uh most of them deal with just like receiving money, right?

72
00:03:41,042 --> 00:03:45,154
And so if you had six hundred engineers just on receiving money, that I don't know, maybe
I'd challenge that.

73
00:03:45,154 --> 00:03:47,552
But you would think that getting paid.

74
00:03:47,552 --> 00:03:49,763
Every company is good at this, but apparently not.

75
00:03:49,763 --> 00:03:58,018
Like if you want to give money to someone, I remember not too long ago there was a seller
I was buying a book from who lived in Canada and there was an issue.

76
00:03:58,018 --> 00:04:03,170
I'm not gonna get into it, but the post was involved and they messed up like really bad.

77
00:04:03,210 --> 00:04:11,514
And I wanted to return the money, but I couldn't because like Canada only accepts like
money transfers like internal to the country.

78
00:04:11,725 --> 00:04:16,193
there and you try to do this, it's like, I'm sorry, I have no way of actually giving you
money.

79
00:04:16,193 --> 00:04:23,602
To which point it like we resulted in just giving the money to a third party that was okay
accepting it and they paid the other person.

80
00:04:23,602 --> 00:04:29,179
Uh, and so we have like extra Canadian dollars that one of my friends owes me and that's
it.

81
00:04:29,179 --> 00:04:33,234
And it's like I I don't know if I'll ever get that back, but at this point, this is still
an unresolved problem.

82
00:04:33,234 --> 00:04:33,622
Yeah.

83
00:04:33,622 --> 00:04:35,893
No, interstate commerce is uh no joke.

84
00:04:35,893 --> 00:04:39,505
There was a thing I I learned about through this process of trying to get this book.

85
00:04:39,505 --> 00:04:42,207
But the book was Psychology of Software Teams from Cat Hicks.

86
00:04:42,207 --> 00:04:45,619
Um, I want the ebook version and I can't really get it.

87
00:04:45,619 --> 00:04:48,770
That's not available for sale in a DRM free way.

88
00:04:48,770 --> 00:04:59,056
But I found that there's a service that you can purchase where you can ship them a
physical book, they will slice its spine off, scan every page, OCR it, and give you a pe

89
00:04:59,056 --> 00:05:00,057
give you an EPUB.

90
00:05:00,057 --> 00:05:02,978
And so I'm like seriously considering that process.

91
00:05:04,354 --> 00:05:08,018
With I mean, when you were working at eBay, it was a little while ago, uh, and making
those changes.

92
00:05:08,018 --> 00:05:16,976
Do you think that a similar transition to getting the organization to one of the the far
corners of the door graph, which is like like performing or outperforming teams basically

93
00:05:16,976 --> 00:05:23,993
that are doing automated deployment on merge requests or solving one of the other metrics,
like reducing bugs and and testing, et cetera?

94
00:05:23,993 --> 00:05:29,910
Do you feel like that there would have been a similar approach to solving those things if
you were taking on that initiative today?

95
00:05:29,910 --> 00:05:31,251
Yeah, I think it'd be the same.

96
00:05:31,251 --> 00:05:38,995
It's just a lot of talking with people and you know, I I so at my current job I'm doing
kind of similar work, trying to help people be more effective in their work.

97
00:05:39,015 --> 00:05:45,569
And I often joke that like if we did all the things that I'm suggesting, we would be like
a a really solid twenty ten company.

98
00:05:45,569 --> 00:05:47,320
'Cause like this is all known.

99
00:05:47,320 --> 00:05:52,903
Like there's nothing about AI that makes continuous delivery and small feedback loops,
small batch sizes.

100
00:05:52,903 --> 00:05:56,184
All of this stuff is known, has been known for a decade or more.

101
00:05:56,184 --> 00:05:58,526
Do the things that we as an industry know you should do.

102
00:05:58,526 --> 00:05:59,210
And then

103
00:05:59,210 --> 00:06:05,995
you will get outcomes that we at the industry have have learned over time and have
researched back up that this is important and it works.

104
00:06:05,995 --> 00:06:09,897
And so we can just follow the case studies of everyone else who's done it before and get
very similar results.

105
00:06:09,897 --> 00:06:11,606
It's not easy, but it's simple.

106
00:06:11,606 --> 00:06:19,832
I I think it's really, really a bit sad that a lot of companies still haven't made those
transitions and coming in into them as some sort of senior leader where you're accountable

107
00:06:19,832 --> 00:06:25,896
or at least that's your core responsibility to to bridge this gap, like, hey, we saw our
competitor, they're doing such a great job.

108
00:06:25,896 --> 00:06:28,588
You came over from them and now we want you to do the same thing here.

109
00:06:28,588 --> 00:06:30,249
And you're like, I just did that.

110
00:06:30,249 --> 00:06:36,583
The last five companies I was also at, uh, how is it that there are still more companies
that are are doing this and are are behind in this way?

111
00:06:36,583 --> 00:06:40,606
Why why is it that the whole industry hasn't transitioned to

112
00:06:40,641 --> 00:06:41,900
Handling these aspects.

113
00:06:41,900 --> 00:06:43,080
There's a handful of things.

114
00:06:43,080 --> 00:06:52,355
I'm not sure if you're familiar with the eternal September phenomenon, where it was the
case in the before times, people who are on these internet servers would get a new class

115
00:06:52,355 --> 00:06:53,175
every September.

116
00:06:53,175 --> 00:06:58,738
And you would just get this most inane message board posts of people who were like, My
this is my first computer.

117
00:06:58,738 --> 00:07:00,559
I don't really know the social norms.

118
00:07:00,559 --> 00:07:01,880
And then you train them, okay?

119
00:07:01,880 --> 00:07:07,006
And now we've returned to normal levels of discourse, and then a new freshman class shows
up and they've never touched a computer before.

120
00:07:07,006 --> 00:07:08,557
And so sep it was like this September.

121
00:07:08,557 --> 00:07:12,669
And then they realized that like when AOL happened, now you were in an eternal September.

122
00:07:12,669 --> 00:07:17,912
There were always new people showing up who had no clue of the social norms or how to
write an email effectively, et cetera.

123
00:07:17,912 --> 00:07:25,046
While we've known for some time what it means to do effective software delivery, that's
not necessarily evenly spread knowledge.

124
00:07:25,046 --> 00:07:27,237
And also new people join the industry all the time.

125
00:07:27,237 --> 00:07:32,180
A lot of my work has been this is what software development really means and this is how
we do things.

126
00:07:32,180 --> 00:07:35,181
And we actually have like terrible on ramps for things like this.

127
00:07:35,181 --> 00:07:36,414
How do I do a sprint?

128
00:07:36,414 --> 00:07:37,516
Like what is a sprint?

129
00:07:37,516 --> 00:07:39,581
How how do I make that go well?

130
00:07:39,581 --> 00:07:42,005
Answer Well, there's two months of certifications.

131
00:07:42,005 --> 00:07:43,241
Would you like that as the answer?

132
00:07:43,241 --> 00:07:46,224
Or would you like a single document that says what you should get out of each ceremony?

133
00:07:46,224 --> 00:07:48,838
Because we don't have that second one unless someone's written it.

134
00:07:48,954 --> 00:07:53,296
And it's out of date and references things that, you know, aren't necessarily relevant
anymore.

135
00:07:53,296 --> 00:07:55,985
And it's like, well, it turns out that I I love this one.

136
00:07:55,985 --> 00:08:03,180
there was a time where the bit the most interesting thing to do was to say, Hey, look,
there's a thing called the Spotify model that Spotify came out with.

137
00:08:03,180 --> 00:08:07,522
And then like a couple months later, after after the entire world found out about it.

138
00:08:07,522 --> 00:08:14,936
Which I the long story short is, you know, here's this sort of way to run a matrix
organization where teams are cross functional and you can, you know, evaluate them on

139
00:08:14,936 --> 00:08:19,402
their success based on the success of the team and performance management is all rolled
into one.

140
00:08:19,402 --> 00:08:21,810
Uh it's not a org structure, whatever.

141
00:08:21,810 --> 00:08:29,934
It's like, well, turns out actually Spotify was pretty much like right after be like, uh
actually we're not even doing that anymore because there's just fundamental issues with a

142
00:08:29,934 --> 00:08:31,185
matrix organization.

143
00:08:31,185 --> 00:08:36,770
Like we need to have component teams that are responsible for their own or accountable for
their own outcomes and being able to

144
00:08:36,770 --> 00:08:41,314
Go with that full steam without constantly being, you know, have weird reporting lines.

145
00:08:41,314 --> 00:08:44,026
But everyone's like, oh, matrix organization, Spotify model.

146
00:08:44,026 --> 00:08:45,968
And this was like for I think at least a decade.

147
00:08:45,968 --> 00:08:47,274
Like I don't hear it as much anymore.

148
00:08:47,274 --> 00:08:49,160
I think the pandemic broke a lot of things.

149
00:08:49,160 --> 00:08:52,663
One of them was certain spreads of information, like what was working, what wasn't.

150
00:08:52,663 --> 00:09:01,290
But I I do think that companies being behind is part of it is like a lack of of
documentation, uh a lack of how you even do things.

151
00:09:01,290 --> 00:09:03,312
I think we're even further behind than that.

152
00:09:03,312 --> 00:09:06,018
I don't know if I believe that there are people who know.

153
00:09:06,018 --> 00:09:06,948
what they should be doing.

154
00:09:06,948 --> 00:09:09,941
Like we're doing this because it's good and it's not written down.

155
00:09:09,941 --> 00:09:11,702
It's like they don't even know that they should be doing this.

156
00:09:11,702 --> 00:09:12,773
Like, why are you having these meetings?

157
00:09:12,773 --> 00:09:13,894
Like, well, we always had this meeting.

158
00:09:13,894 --> 00:09:14,754
It's tradition, right?

159
00:09:14,754 --> 00:09:24,041
Lack of documentation, the onboarding ramp, or went to a university where they didn't
focus on like, this is how you collaborate, which is not historically an academic thing,

160
00:09:24,041 --> 00:09:26,458
at least in the United States where I got educated.

161
00:09:26,458 --> 00:09:32,457
I had teachers that would forever say, you don't get accepted to companies in groups, like
to jobs, to interviews.

162
00:09:32,457 --> 00:09:33,938
So you should be working by yourself.

163
00:09:33,938 --> 00:09:34,700
And I'm like

164
00:09:34,700 --> 00:09:38,998
And now I'm like, actually, uh, you are definitely accepted for your ability to
collaborate.

165
00:09:38,998 --> 00:09:47,594
And if you if you had practiced at that, you would have been much better and much better
off and being able to actually pass the interview than knowing some rote engineering

166
00:09:47,594 --> 00:09:48,142
activity.

167
00:09:48,142 --> 00:09:52,804
I think the importance of collaboration changes a lot company to company.

168
00:09:52,824 --> 00:09:58,307
Like, so I went through the Google interview process and and worked at Google for a little
while.

169
00:09:58,307 --> 00:10:01,988
And there they like kind of famously teach computer science fundamentals.

170
00:10:01,988 --> 00:10:03,339
There it makes a ton of sense.

171
00:10:03,339 --> 00:10:07,231
And the thing they don't tell you is there's no blueprint for the job here.

172
00:10:07,231 --> 00:10:12,063
Like you can't go look up how someone else did it because you're working in the make cool
shit factory.

173
00:10:12,063 --> 00:10:12,773
You're the first one.

174
00:10:12,773 --> 00:10:14,442
There's no one who's done this before.

175
00:10:14,442 --> 00:10:15,382
So

176
00:10:15,542 --> 00:10:16,293
There's one to learn from.

177
00:10:16,293 --> 00:10:19,308
You just have to kind of go based on the fundamentals you know.

178
00:10:19,308 --> 00:10:25,096
Most companies are like, I would like another crud app, please, uh, with login.

179
00:10:25,096 --> 00:10:27,098
And you're like, Okay, great.

180
00:10:27,219 --> 00:10:29,802
I you don't need to know what a red black tree is for that.

181
00:10:29,802 --> 00:10:32,126
You need to know about a different suite of things.

182
00:10:32,126 --> 00:10:32,342
I

183
00:10:32,342 --> 00:10:40,315
Find some of the engineers that have gone to like bear organizations with you know solving
challenging problems, like they have no idea what these other problems are.

184
00:10:40,315 --> 00:10:41,667
Like I wouldn't call them simple.

185
00:10:41,667 --> 00:10:48,871
Like I worked at a bunch of enterprise organizations and I come out of them, I'm like,
okay, we're gonna build login and access control for this e-commerce platform.

186
00:10:48,871 --> 00:10:51,512
And they're like, I don't know what those words are.

187
00:10:51,512 --> 00:10:56,925
Like, I don't know what LDAP is, I don't know how Microsoft even authenticates, you know,
Windows computers or whatever.

188
00:10:56,925 --> 00:11:02,156
I'm like, you've been running a monolith software in Windows on

189
00:11:02,156 --> 00:11:09,048
Windows servers for almost 20 years and you have no idea how the you like user login and
how you secure an endpoint works.

190
00:11:09,048 --> 00:11:10,728
Like, no, it's like all taken care of.

191
00:11:10,728 --> 00:11:12,019
Like we don't we don't focus on that.

192
00:11:12,019 --> 00:11:12,619
We have no idea.

193
00:11:12,619 --> 00:11:16,560
Um, but if you solve linear algebra problems, yeah, that's what we do all day.

194
00:11:16,560 --> 00:11:21,251
Like we have a very complicated have to figure out like the order to manufacture things.

195
00:11:21,251 --> 00:11:22,622
And that's really important for us.

196
00:11:22,622 --> 00:11:25,632
And so everyone everywhere in the company knows how to do that.

197
00:11:25,632 --> 00:11:31,702
But when it comes to how to integrate with some system to solve what every other company,
especially like in the startup world,

198
00:11:31,702 --> 00:11:33,718
'cause you of course you're making new apps all the time.

199
00:11:33,718 --> 00:11:34,328
They have no

200
00:11:34,328 --> 00:11:34,508
Yeah.

201
00:11:34,508 --> 00:11:45,123
And I think this is a a reality for all of us, that there are things that we ignore even
if we don't know that we're ignoring them.

202
00:11:45,723 --> 00:11:51,206
In kind of my practical day to day, I think a lot about how people deploy code and how
people review code.

203
00:11:51,206 --> 00:11:56,408
I think we're kind of rapidly approaching a world where people don't look at the code as
much as we historically have.

204
00:11:56,548 --> 00:11:59,609
And you may initially think, that's terrifying.

205
00:11:59,769 --> 00:12:01,398
And that's terrifying.

206
00:12:01,398 --> 00:12:09,645
And then the reality is is that you should already be terrified because if you look at any
note modules, you're shifting a hundred thousand lines of code you've never seen before.

207
00:12:09,645 --> 00:12:10,784
What's two thousand more?

208
00:12:10,784 --> 00:12:15,377
Of course the joke is like the heaviest mass in the in the universe is is the node modules
directory, right?

209
00:12:15,377 --> 00:12:26,684
When we think about what code makes sense, what what we are proving in a pull request, it
was do we trust what is here is going to have the right long term implication for us?

210
00:12:26,684 --> 00:12:34,030
And long term is in itself vague and individual specific according to whatever my personal
values are, how I see them.

211
00:12:34,030 --> 00:12:39,586
Like maybe I care about some aspect of the company or I care about what whether I'm going
to be on call or I care about

212
00:12:39,586 --> 00:12:49,019
Having kudos sent to me in the next all hands meeting where someone says my name on stage,
like those those different motivators could change how I particular particularly review

213
00:12:49,019 --> 00:12:49,589
code.

214
00:12:49,589 --> 00:12:53,730
And I think when it comes to the dependencies, I think it was always based on trust.

215
00:12:53,730 --> 00:12:59,192
Like maybe I looked up the number of stars on GitHub or the number of downloads on a
package manager.

216
00:12:59,192 --> 00:13:02,033
And I use that to help me make some informed decision here.

217
00:13:02,033 --> 00:13:06,304
Is that same decision still being made today with the dependencies that you're utilizing?

218
00:13:06,304 --> 00:13:07,554
I don't know if that's true.

219
00:13:07,554 --> 00:13:12,457
And I do think it's very different than the code that's being potentially generated by an
LLM, although maybe not.

220
00:13:12,457 --> 00:13:15,656
May maybe fundamentally where did the LLM get trained on?

221
00:13:15,656 --> 00:13:17,080
It was open source packages.

222
00:13:17,080 --> 00:13:22,674
So uh in in a way, it's literally no different than the code that is in packages.

223
00:13:22,674 --> 00:13:26,486
So that seems like a a weird hill for someone to have to die on.

224
00:13:26,486 --> 00:13:31,958
Yeah, I think the way that you talk about reviewing code is like the best version.

225
00:13:31,958 --> 00:13:36,609
Twenty total engineers who's had that level of intention.

226
00:13:36,609 --> 00:13:38,769
And one of them literally wrote a book on code reviews.

227
00:13:38,769 --> 00:13:46,221
Most of it is kind of performative or I think I need to be looking for this stuff or I
need to say something so that people think I'm doing my job.

228
00:13:46,221 --> 00:13:48,202
It's not an inquisitive thing.

229
00:13:48,202 --> 00:13:49,372
We're not trying to learn.

230
00:13:49,372 --> 00:13:55,842
I think that the way you talk about application code that we're writing, we do need to be
kind of vigilant and

231
00:13:55,842 --> 00:13:58,674
Focused on the long-term maintainability aspects of it.

232
00:13:58,674 --> 00:14:02,106
The inverse of that is we never think about that for the open source library.

233
00:14:02,106 --> 00:14:08,201
We're never saying, oh, this thing from kicker 42 is like m not a maintainable piece of
code.

234
00:14:08,201 --> 00:14:09,952
And you're like, no, it's an abstraction.

235
00:14:09,952 --> 00:14:10,852
It's a box.

236
00:14:10,852 --> 00:14:11,653
It's labeled.

237
00:14:11,653 --> 00:14:12,684
It has an API.

238
00:14:12,684 --> 00:14:17,817
I don't really care what happens inside until it shows up on my trace graph as being too
slow.

239
00:14:17,817 --> 00:14:19,220
And then I'm grumpy.

240
00:14:19,220 --> 00:14:26,499
Mistake that's probably happening more today is it was always the case where you couldn't
trust the library's uh semantic versioning to be correct.

241
00:14:26,499 --> 00:14:33,327
So they upgrade from like, you know, point zero one to uh point oh two and suddenly
there's a critical breaking change there.

242
00:14:33,327 --> 00:14:39,544
And I think in the past we would have just pinned our version and now we're getting sp I
hate to say AI, you know, SAS tool.

243
00:14:39,544 --> 00:14:43,056
That you have to do static analysis is saying, hey, there's some sort of vulnerability in
this package.

244
00:14:43,056 --> 00:14:44,307
You should upgrade to this version.

245
00:14:44,307 --> 00:14:46,159
You're like, well, semantic version looks fine.

246
00:14:46,159 --> 00:14:47,620
I'm just going to merge it in.

247
00:14:47,620 --> 00:14:50,992
And now you've created, I mean, at worst, it's a performance issue.

248
00:14:50,992 --> 00:14:57,776
It could be like a subtle, hidden, silent production bug that is there and affecting
millions or billions of rows in the database.

249
00:14:57,776 --> 00:15:01,749
But maybe we were getting luckier before where there was less versions being rolled.

250
00:15:01,749 --> 00:15:07,973
But you did make a really interesting point, which is I don't recall ever being taught how
to do a code review.

251
00:15:07,973 --> 00:15:09,048
You go to a company.

252
00:15:09,048 --> 00:15:13,379
And they're like, okay, uh, hey, Warren, can you go review this code?

253
00:15:13,379 --> 00:15:15,380
The at one company that didn't even exist.

254
00:15:15,380 --> 00:15:16,290
There was no code reviews.

255
00:15:16,290 --> 00:15:17,401
They also didn't have source control.

256
00:15:17,401 --> 00:15:24,123
But at another company, it was like, I'm going to show you the list of changes by file
name in an SVN commit.

257
00:15:24,123 --> 00:15:28,404
And before I click commit, you're just gonna like look at them and say, like, does that
look okay?

258
00:15:28,404 --> 00:15:30,945
You go through that a couple of times, it's like, what am I doing here?

259
00:15:30,945 --> 00:15:32,626
Like, why am I doing this work?

260
00:15:32,626 --> 00:15:34,392
Like, is this benefiting someone?

261
00:15:34,392 --> 00:15:36,423
And that's just like the actual technology side.

262
00:15:36,423 --> 00:15:41,035
On the on the flip side, like from a business standpoint, you're like, is is this even the
right way to look at this code?

263
00:15:41,035 --> 00:15:42,485
Like, no, I want to go sit at my desk.

264
00:15:42,485 --> 00:15:45,147
I wanna, you know, click out these files one by one.

265
00:15:45,147 --> 00:15:49,328
And I think what I settled on is how I do code reviews before LLMs.

266
00:15:49,328 --> 00:15:50,929
Like maybe we'll get to that in a moment.

267
00:15:50,929 --> 00:15:58,212
Is if I wrote this code from scratch with the ticket that was created, assuming I had all
the context, how would I have written it?

268
00:15:58,212 --> 00:16:01,493
And after I'm done writing it, how does my version compare to what was written?

269
00:16:01,493 --> 00:16:02,634
Did I miss things?

270
00:16:02,634 --> 00:16:03,074
Where

271
00:16:03,074 --> 00:16:06,744
Things included that I had that they didn't, you know, why is that?

272
00:16:06,744 --> 00:16:10,036
was very lucky to work at Google when I did.

273
00:16:10,036 --> 00:16:16,721
One of the people I worked with is a guy named Trevor Foucher, uh, who wrote um the book
Art of Readable Code.

274
00:16:16,721 --> 00:16:19,363
And it's a literally a book about how to do code reviews.

275
00:16:19,363 --> 00:16:23,806
And unsurprising, he was on a team who had excellent code reviews.

276
00:16:23,806 --> 00:16:27,589
And they were writing C for the ads world.

277
00:16:27,589 --> 00:16:31,211
And so very high frequency, low latency things.

278
00:16:31,211 --> 00:16:35,764
Getting a code review from him and that team was just perfect because they were like

279
00:16:35,954 --> 00:16:40,778
we don't do things this way because that leads to this problem, which we don't want
because of this reason.

280
00:16:40,778 --> 00:16:49,184
And it was a very educational process, a dedication and an attention to the to the
artifact that I've not seen a lot since then.

281
00:16:49,184 --> 00:16:52,400
And I don't really think that the industry supports it in the same way.

282
00:16:52,400 --> 00:16:56,889
Like I think we're too interested in output over necessarily outcomes.

283
00:16:56,889 --> 00:17:03,564
There was a good post which was like about being addicted to speed, just the fact that you
did it quickly and how it's like a dopamine kind of thing.

284
00:17:03,564 --> 00:17:03,894
Yeah.

285
00:17:03,894 --> 00:17:13,040
So we actually had a whole episode on this podcast where we talked about like what is
productivity and we came to the conclusion that it's it's a feeling and not a measurable

286
00:17:13,040 --> 00:17:13,341
thing.

287
00:17:13,341 --> 00:17:21,666
Because if it were measurable, you would be like, Oh, we're doing twice as many
deployments as we were doing before, or we are releasing way fewer bugs, or there are way

288
00:17:21,666 --> 00:17:24,829
more tickets making it through our pipeline to completion.

289
00:17:24,829 --> 00:17:28,011
You would just say that we have more tickets completed, we have more deployments.

290
00:17:28,011 --> 00:17:29,932
But instead you say, No, we have higher productivity.

291
00:17:29,932 --> 00:17:32,856
And the fact you're using that word as a proxy for the real thing.

292
00:17:32,856 --> 00:17:35,468
Tells you that you don't actually know what you're measuring.

293
00:17:35,468 --> 00:17:39,111
I do think that there is a focus that is different by company.

294
00:17:39,111 --> 00:17:43,294
Like some companies, realistically, they don't have the same needs, right?

295
00:17:43,294 --> 00:17:49,449
Their whole business model is we spin up our company, we release a product, it lasts five
years and then it dies.

296
00:17:49,449 --> 00:17:53,663
A whole company goes bankrupt, but we we got some people paid, I made some money and I'll
make the next company.

297
00:17:53,663 --> 00:18:02,668
And if that's your model of working, it doesn't matter that you have good code, that it
has high quality, because your whole reason for existing is very temporary.

298
00:18:02,668 --> 00:18:09,791
And I think this is the default model that companies fall into without good code reviews,
because it's just an extension of how the company operates.

299
00:18:09,791 --> 00:18:17,675
And whether or not they're committing to that end goal or they they believe that's the end
goal they want, they're gonna end up getting there because you know, customers will leave

300
00:18:17,675 --> 00:18:20,507
because their stuff is uh not great, et cetera, et cetera.

301
00:18:20,507 --> 00:18:22,518
And maybe customers don't totally care.

302
00:18:22,518 --> 00:18:27,810
But I I think subconsciously they do care a little bit that the stuff that they're using
works.

303
00:18:27,832 --> 00:18:31,994
Yeah, I think that it's a big shift in how we run businesses.

304
00:18:31,994 --> 00:18:35,585
I've worked at some companies where retention was critically important.

305
00:18:35,585 --> 00:18:37,386
That was like the main metric to care about.

306
00:18:37,386 --> 00:18:40,097
If it doesn't leak, you can fill a bucket a drop at a time.

307
00:18:40,097 --> 00:18:47,770
other version of this is if you have a water faucet that pours enough water, it doesn't
matter how big the leak is, as long as there's more water to fill.

308
00:18:48,611 --> 00:18:50,912
There are different phases of software development.

309
00:18:50,912 --> 00:18:57,024
And one version of this is I want the very best load balancer and load shedder available.

310
00:18:57,024 --> 00:18:57,364
Okay.

311
00:18:57,364 --> 00:18:58,995
That's a pretty well defined problem set.

312
00:18:58,995 --> 00:19:00,175
Like two approaches.

313
00:19:00,175 --> 00:19:05,297
One is like meticulously looking at the things, and another is lots of LLM spent.

314
00:19:05,297 --> 00:19:11,440
And I think meticulous in that world probably pays off, similar to how like Redis, how
Redis works.

315
00:19:11,440 --> 00:19:14,981
You're like, there are people, they pay a lot of attention to the details.

316
00:19:14,981 --> 00:19:18,143
Maybe they're AI assisted, but it's mostly just a lot of human thought.

317
00:19:18,143 --> 00:19:22,965
I think that there's another form of development where we don't even know what we're
building.

318
00:19:22,965 --> 00:19:26,766
We're kind of exploring a solution space with software and

319
00:19:26,766 --> 00:19:28,357
We're like, I don't know what these people want.

320
00:19:28,357 --> 00:19:30,568
These people don't know what they want.

321
00:19:30,569 --> 00:19:35,192
So let's generate something that's kind of like what they want and see if they like it.

322
00:19:35,192 --> 00:19:36,643
And if they like it, we'll do more.

323
00:19:36,643 --> 00:19:38,540
And if they don't like it, we'll do less.

324
00:19:38,540 --> 00:19:42,633
There is this thought that there is some area where LLMs are viable to use.

325
00:19:42,633 --> 00:19:48,597
The thing is that my own experimentation with prototyping, we have a bunch of products
that we're we're trying to come up with right now.

326
00:19:48,617 --> 00:19:53,580
And we've tried going down the LLM experimentation for rapid prototyping.

327
00:19:53,580 --> 00:20:01,156
And what we found is that if you can sort of one-shot the creation of that, like maybe
screen by screen or whatever, it's fine.

328
00:20:01,156 --> 00:20:02,947
But if you go to a user and say, hey, is this okay?

329
00:20:02,947 --> 00:20:05,389
And they're like, actually, I want to see something a little bit different.

330
00:20:05,389 --> 00:20:07,638
If you go back and try to change it,

331
00:20:07,638 --> 00:20:10,941
The whole thing crashes, like terrible.

332
00:20:10,941 --> 00:20:16,766
And I I I think it's because the code that the LM has generated is not maintainable even
by an LLM.

333
00:20:16,766 --> 00:20:24,782
And so if you make a change in one part of the application, you know, let's say there's 10
pages, 10 screens, if you make a change in one of those screens, each one of those screens

334
00:20:24,782 --> 00:20:26,534
has its own custom implementation.

335
00:20:26,534 --> 00:20:29,576
And so it doesn't change consistently across everywhere.

336
00:20:29,576 --> 00:20:32,489
And so it's d and it's difficult to catch that that didn't happen.

337
00:20:32,489 --> 00:20:36,642
And so, like, unless you're going to every single one of those screens and then iterating
on each one of them.

338
00:20:36,642 --> 00:20:37,503
like manually.

339
00:20:37,503 --> 00:20:42,606
I mean and by manually I mean like having the LLM audit what's there and then making the
same change.

340
00:20:42,647 --> 00:20:49,933
You're gonna fall into this pattern where your the standard software development strategy
of like, we probably wrote this in a in a scalable maintainable way.

341
00:20:49,933 --> 00:20:55,027
Hopefully there's just one shared database with uh with all the relevant details in
because it's a prototype and we don't care.

342
00:20:55,027 --> 00:21:02,243
And it's like, well no actually there's every single one of these data types has their own
custom column or in custom screen and has to be changed.

343
00:21:02,243 --> 00:21:06,386
And the LM of course, you know, doesn't know that because it did you didn't feed that
information in.

344
00:21:06,518 --> 00:21:09,604
And so when you go to try to change it, you end up with serious problems.

345
00:21:09,604 --> 00:21:13,056
So much so that like, you know, you and then spend a lot of time trying to debug

346
00:21:13,056 --> 00:21:16,147
If you want good software, you have to give context.

347
00:21:16,147 --> 00:21:20,448
LLMs can't make software that's good outside of having context.

348
00:21:20,448 --> 00:21:23,089
And it turns out neither can humans.

349
00:21:23,189 --> 00:21:30,591
Like if you tell the human nothing about how to do software delivery, and then you're
like, hey, could you please generate me a website?

350
00:21:30,591 --> 00:21:32,811
You're like, yeah, I know HTML.

351
00:21:32,811 --> 00:21:34,572
And you're like, okay, but I want it to be interactive.

352
00:21:34,572 --> 00:21:37,733
And you're like, well, I also have read this book on JavaScript.

353
00:21:37,733 --> 00:21:40,420
And you're like, okay, but now my thing, I'd like it to change.

354
00:21:40,420 --> 00:21:42,446
And you're like, well, I didn't plan that.

355
00:21:42,446 --> 00:21:46,826
Because I forgot that users have requests and I didn't know that yet.

356
00:21:46,826 --> 00:21:54,600
And so I've especially been there in my career, um, where I was given a requirement that I
didn't question and that went very poorly for me.

357
00:21:54,600 --> 00:21:57,032
I've also like missed requirements, right?

358
00:21:57,032 --> 00:22:00,066
And so like it's nothing specific to LLMs.

359
00:22:00,066 --> 00:22:00,280
I

360
00:22:00,280 --> 00:22:02,921
Think that you're you were onto something with this the addiction to speed.

361
00:22:02,921 --> 00:22:09,754
It's almost like going slower caused us to create processes in place which enabled us to
make better software.

362
00:22:09,754 --> 00:22:15,697
But going faster sidesteps the challenges that we like the real challenges that we would
have had to overcome.

363
00:22:15,697 --> 00:22:20,659
By doing so, we're basically creating a worse process for us to deal with.

364
00:22:20,659 --> 00:22:23,670
And the only solution is to actually go slower.

365
00:22:23,670 --> 00:22:26,841
And I I I I came to this conclusion after watching something completely unrelated.

366
00:22:26,841 --> 00:22:28,788
Rory Sutherland, who has

367
00:22:28,788 --> 00:22:30,279
uh great conference speaker.

368
00:22:30,279 --> 00:22:32,330
He talks about like actually wanting to go slower.

369
00:22:32,330 --> 00:22:39,224
And the analogy he used is like if you're driving in a car, and I hate using this analogy
since I moved to Switzerland because very few people drive cars.

370
00:22:39,264 --> 00:22:48,449
And you you're in a car and you're going like a hundred kilometers an hour and you're
like, okay, but what if we could go faster and go like 110 kilometers per hour?

371
00:22:48,449 --> 00:22:55,843
And like it turns out that unless you're going like a very, very short distance or a very,
very long distance, going faster doesn't change anything.

372
00:22:55,843 --> 00:22:57,102
It's like you're

373
00:22:57,102 --> 00:23:04,945
Creating a uh a larger risk by driving faster to get like, you know, let's say across a
large landmass, like across Europe or across the United States.

374
00:23:04,945 --> 00:23:08,086
And going that much faster on average definitely gets you there sooner.

375
00:23:08,086 --> 00:23:11,967
But it's like you get there sooner by like a few hours, maybe.

376
00:23:11,987 --> 00:23:15,958
And the risk of, you know, serious injury increases like exponentially.

377
00:23:15,958 --> 00:23:21,410
Uh, and of course, if you're like driving from here to the the corner gas station.

378
00:23:21,410 --> 00:23:24,501
Like going a hundred kilometers an hour to begin with is like super dictarist.

379
00:23:24,501 --> 00:23:25,382
But it's the same thing.

380
00:23:25,382 --> 00:23:29,301
And so like unless you're in this weird area where they're under a rush to actually make
it happen.

381
00:23:29,301 --> 00:23:30,394
Like you have some deadline.

382
00:23:30,394 --> 00:23:32,405
Going faster isn't necessarily the right thing.

383
00:23:32,405 --> 00:23:36,597
And whereas historically we would have gone slower because that's just what we have.

384
00:23:36,597 --> 00:23:38,807
Going faster doesn't fix any of those problems.

385
00:23:38,807 --> 00:23:40,332
It just increases the risk.

386
00:23:40,332 --> 00:23:40,883
Yeah.

387
00:23:40,883 --> 00:23:44,651
I think there's also like this distinction on speed.

388
00:23:44,651 --> 00:23:47,862
Like do you need immediate speed or long term speed?

389
00:23:47,862 --> 00:23:49,212
That is definitely the mindset, right?

390
00:23:49,212 --> 00:23:54,724
Uh historically it was you go and work at a company for a very long time, you retire, and
everything's fine.

391
00:23:54,724 --> 00:24:05,627
And I feel like there has been this shift downwards where getting things like especially
with the social media and uh certain applications out there that really encourage the very

392
00:24:05,627 --> 00:24:14,242
clickety or or twitchy experience where you very quickly transition from one topic to
another one, very ADD focused, like next idea, next idea.

393
00:24:14,242 --> 00:24:20,794
I think it is all well aligned with this transitioning down away from the safe foundation
to doing things as fast as possible.

394
00:24:20,794 --> 00:24:27,646
The the addiction with with speed, I think this may be just a human fallacy, just a a
foible that that humanity has.

395
00:24:27,646 --> 00:24:29,407
And so it does seem like it's better.

396
00:24:29,407 --> 00:24:33,998
And I think historically, we're not great at long term trade off.

397
00:24:33,998 --> 00:24:39,266
Like there's very few people, I think, that not only are able to see the long term, but
actually be like, Hey, look.

398
00:24:39,266 --> 00:24:44,642
Like if we do this, there's a real consequence, even five years down the road, or even one
year down the road, or one month down the road.

399
00:24:44,642 --> 00:24:53,252
Like it's amazing how many conversations I've seen where just one month later is just
immediately going to be an issue that no one considered, that no one even talked about for

400
00:24:53,252 --> 00:24:54,244
one moment.

401
00:24:54,244 --> 00:24:56,406
I just think historically we're just not great at that.

402
00:24:56,406 --> 00:25:02,328
You're saying there's not a solution for people's inability to understand and articulate
risk?

403
00:25:02,869 --> 00:25:05,890
I think the security people probably deal with this the most.

404
00:25:05,890 --> 00:25:09,632
Which is like, what's the likelihood that someone guesses the CEO's password?

405
00:25:09,632 --> 00:25:11,273
You're like, probably pretty low.

406
00:25:11,273 --> 00:25:12,394
How bad is it?

407
00:25:12,394 --> 00:25:13,826
Well, it's pretty bad.

408
00:25:13,826 --> 00:25:21,752
To look at like the expected value of an incident, like the risk, the right likelihood,
the probability of its happening, and the impact, and do that multiplication to get the

409
00:25:21,752 --> 00:25:22,562
expected value.

410
00:25:22,562 --> 00:25:24,433
There's two challenges with that, I think.

411
00:25:24,433 --> 00:25:28,076
Fundamentally, the first one is that you actually sort of have to have those numbers.

412
00:25:28,076 --> 00:25:30,958
You have to be you have to understand that you can do this analysis.

413
00:25:30,958 --> 00:25:35,942
And I feel like a lot of companies aren't even prepared to take this into consideration in
the first place.

414
00:25:35,942 --> 00:25:42,402
And the second one is there's other terms in this equation which aren't just like the
straight loss, like.

415
00:25:42,402 --> 00:25:44,283
Why did you why does it cost 10 million?

416
00:25:44,283 --> 00:25:46,264
Is it like an impact to the brand?

417
00:25:46,264 --> 00:25:47,724
Is it a fine?

418
00:25:47,724 --> 00:25:55,377
A lot of companies don't even fall into those categories where having an incident is like
public attention that just increases.

419
00:25:55,377 --> 00:25:58,129
Like having an incident, the ROI is positive.

420
00:25:58,129 --> 00:26:02,631
Return on incident equals positive, which is like an absurd thing to say.

421
00:26:02,631 --> 00:26:05,752
But companies actually can use this successfully.

422
00:26:05,752 --> 00:26:11,126
And I think we are potentially seeing that in the age of LLMs where companies who should

423
00:26:11,126 --> 00:26:13,848
under no circumstance get any praise for doing a thing.

424
00:26:13,848 --> 00:26:16,310
Now are getting uh public recognition for it.

425
00:26:16,310 --> 00:26:20,733
Like there's one company that apparently is hacking other companies.

426
00:26:20,733 --> 00:26:29,269
And instead of saying like they need to be regulated into the ground or be heavily fined,
everyone is standing up and cheering like, how is this like this is so amazing that they

427
00:26:29,269 --> 00:26:29,779
can do this.

428
00:26:29,779 --> 00:26:33,602
And I'm like, I don't think you understand how this should work.

429
00:26:33,602 --> 00:26:37,164
Yeah, I've I think I've heard that described as gloating about felonies.

430
00:26:38,145 --> 00:26:44,010
I had some I had a friend of mine who asked me, do I trust LLMs to ship to production with
zero human involvement?

431
00:26:44,010 --> 00:26:46,471
My answer, 100% yes.

432
00:26:46,471 --> 00:26:53,356
For my blog, I would trust an LLM hundreds of dollars given the setup I have of risk.

433
00:26:53,356 --> 00:26:54,277
Not thousands.

434
00:26:54,277 --> 00:26:56,358
Certainly not bet my company on.

435
00:26:56,358 --> 00:26:57,959
Hundreds of dollars of downside?

436
00:26:57,959 --> 00:26:59,020
Yeah, absolutely.

437
00:26:59,020 --> 00:26:59,671
Ship that code.

438
00:26:59,671 --> 00:27:01,634
But it's not twenty thousand dollars.

439
00:27:01,634 --> 00:27:06,637
Which which says I think more about you and your blog than about the LLMs.

440
00:27:06,637 --> 00:27:09,668
Because there is this thing where like some companies are are are betting on this.

441
00:27:09,668 --> 00:27:10,329
Like, yeah, you know what?

442
00:27:10,329 --> 00:27:12,480
What if it just costs us a thousand or ten thousand?

443
00:27:12,480 --> 00:27:16,552
I still have the rest of our bank account, you know, to sponsor whatever happens.

444
00:27:16,552 --> 00:27:18,723
It's it's a experiment, quote unquote.

445
00:27:18,723 --> 00:27:23,216
Not really an experiment because A, you already know how it it comes out, and B, you're
not measuring anything.

446
00:27:23,216 --> 00:27:26,518
So you're not really deciding whether or not you want to continue this experiment.

447
00:27:26,518 --> 00:27:28,889
You're just it's more of a speculation.

448
00:27:28,889 --> 00:27:31,272
And maybe you'll never find out the problem with it.

449
00:27:31,272 --> 00:27:33,883
Or it will be super successful like getting VC funding.

450
00:27:33,883 --> 00:27:42,636
And why I say it says a lot about you and your blog is because I I think it's that the
confidence that you could write a one shot or you know, put that in the context that we

451
00:27:42,636 --> 00:27:51,199
talked about before to be able to provide effectively says how great of a of an engineer
or how much you trust your ability to interact with it to have the right outcome without

452
00:27:51,199 --> 00:27:52,189
potentially reviewing it.

453
00:27:52,189 --> 00:27:55,190
You know, I see lots of people out there that don't have the

454
00:27:55,190 --> 00:27:57,551
extensive understanding of how to communicate well.

455
00:27:57,551 --> 00:28:00,330
And I think communication's a huge part of interacting with LMs.

456
00:28:00,330 --> 00:28:02,183
Like you know what's supposed to be released.

457
00:28:02,183 --> 00:28:04,414
You know what that code is supposed to be look look like.

458
00:28:04,414 --> 00:28:06,094
You know how to talk about that.

459
00:28:06,094 --> 00:28:13,767
Then I feel like you're eliminating all of the or most of the failure modes that come out,
which are like randomly choosing things, building it the wrong way, or releasing the wrong

460
00:28:13,767 --> 00:28:15,286
feature with the wrong technology.

461
00:28:15,286 --> 00:28:20,506
And so I've been thinking about what might it look like if you were not allowed to look at
the code.

462
00:28:20,506 --> 00:28:24,262
You're only allowed to view it from the outside in, like a car mechanic.

463
00:28:24,262 --> 00:28:27,244
You don't get to just see how the engine block was made.

464
00:28:27,244 --> 00:28:31,037
It you have to assume that it's made of steel and that it is what it is.

465
00:28:31,037 --> 00:28:36,941
What sort of tools could we use and how could we think about that problem in order to
understand?

466
00:28:36,941 --> 00:28:38,502
Production telemetry is an example.

467
00:28:38,502 --> 00:28:41,144
We can look at the production telemetry and see what happened.

468
00:28:41,144 --> 00:28:44,236
We could look at the test run reports and say,

469
00:28:44,236 --> 00:28:45,867
15 unit tests pass.

470
00:28:45,867 --> 00:28:56,314
But even as we talk about like large scale production systems, there are things we can
encode in the process from LLM through the software generation process to the review

471
00:28:56,314 --> 00:29:01,858
process through deploy and then in production operations that de-risk the failure modes.

472
00:29:01,858 --> 00:29:05,821
So let's take the example of Amazon checkout process.

473
00:29:05,821 --> 00:29:08,396
What are the ways that Amazon checkout process could break?

474
00:29:08,396 --> 00:29:11,788
Well, one of them is I click checkout and it just errors.

475
00:29:11,788 --> 00:29:12,448
Okay, cool.

476
00:29:12,448 --> 00:29:14,970
Could we have a robot click checkout every five minutes?

477
00:29:14,970 --> 00:29:15,690
Yeah, cool.

478
00:29:15,690 --> 00:29:16,951
Okay, so we've covered that.

479
00:29:16,951 --> 00:29:20,613
So now you're limited to every f like five minutes of dollar loss.

480
00:29:20,613 --> 00:29:21,593
Okay, that's not great.

481
00:29:21,593 --> 00:29:23,134
How could we reduce that further?

482
00:29:23,134 --> 00:29:26,956
Well, we could roll it out to a percentage of traffic and test it there first.

483
00:29:26,956 --> 00:29:32,399
Okay, we could roll it out to just one request and test my one request and make sure that
works before we roll it out further.

484
00:29:32,399 --> 00:29:32,879
Okay, great.

485
00:29:32,879 --> 00:29:33,820
We have tools for all of that.

486
00:29:33,820 --> 00:29:34,650
We've had them for years.

487
00:29:34,650 --> 00:29:35,971
What else could be broken?

488
00:29:35,971 --> 00:29:38,218
sales tax collection could be wrong.

489
00:29:38,218 --> 00:29:42,240
Okay, what's the best tool we have to validate sales tax collection?

490
00:29:42,240 --> 00:29:43,280
You're like unit tests.

491
00:29:43,280 --> 00:29:43,930
Okay, cool.

492
00:29:43,930 --> 00:29:44,701
Do we have those?

493
00:29:44,701 --> 00:29:48,652
Do we have a process by which we assert that we have those tests through coverage?

494
00:29:48,652 --> 00:29:49,243
Things like that.

495
00:29:49,243 --> 00:29:50,243
Yes, we have that.

496
00:29:50,243 --> 00:29:51,564
Okay, what else could go wrong?

497
00:29:51,564 --> 00:29:54,505
Well, the LLM could generate wrong features.

498
00:29:54,505 --> 00:29:55,785
Like, okay, what's a wrong feature?

499
00:29:55,785 --> 00:29:56,486
What does that mean?

500
00:29:56,486 --> 00:29:58,486
Well, it doesn't do the thing I expected to.

501
00:29:58,486 --> 00:29:59,847
Okay, how would you know?

502
00:29:59,847 --> 00:30:04,129
Okay, well, we would know if when an engineer demoed it to me it didn't look how I wanted.

503
00:30:04,129 --> 00:30:07,522
Okay, could we have an LLM generate a demo video for you?

504
00:30:07,522 --> 00:30:08,382
Yeah, we could do that.

505
00:30:08,382 --> 00:30:09,183
Okay, cool.

506
00:30:09,183 --> 00:30:10,103
What else could go wrong?

507
00:30:10,103 --> 00:30:21,776
And so through this process of kind of continual introspection about the artifact as it
makes its way to production, we can de-risk that process in such a way that we have pretty

508
00:30:21,776 --> 00:30:23,527
good confidence that the thing doesn't suck.

509
00:30:23,527 --> 00:30:30,279
Today we underinvest in that deploy pipeline and they put what I would say is unearned
confidence in human code review.

510
00:30:30,279 --> 00:30:30,829
Mm-hmm.

511
00:30:30,829 --> 00:30:34,310
Because like, when was the last time a human actually caught your bug?

512
00:30:34,310 --> 00:30:34,858
Like

513
00:30:34,858 --> 00:30:38,558
I ran a survey at my last company and it was like it's not that recent.

514
00:30:38,558 --> 00:30:42,401
I think there is definitely a huge struggle to basically do anything.

515
00:30:42,401 --> 00:30:45,724
We see a lot of companies trying to solve the AI SRE thing.

516
00:30:45,724 --> 00:30:55,051
Like you will ship bad code to production and now we have tools in place to sort of
automatically create pull requests that find those bugs and push it up and, you know,

517
00:30:55,051 --> 00:30:56,773
automatically merge and fix that.

518
00:30:56,773 --> 00:30:59,875
And my answer has always been, sure, you can have that.

519
00:30:59,875 --> 00:31:03,478
It may be appropriate for some companies, but don't you realize you're

520
00:31:03,478 --> 00:31:06,433
you gotta stop the bad coach from entering production in the first place.

521
00:31:06,433 --> 00:31:12,728
Imagine you had a funnel where software is being thrown at the funnel and it's how can you
filter it down to only getting the stuff that makes sense?

522
00:31:12,728 --> 00:31:18,932
My current thinking is that it is the job of a human to resolve ambiguity and provide
judgment when that's needed.

523
00:31:18,932 --> 00:31:19,522
Yeah.

524
00:31:19,522 --> 00:31:23,365
Like is this feature good enough to show our users is a human question.

525
00:31:23,365 --> 00:31:26,687
What should we build is largely a human decision.

526
00:31:26,687 --> 00:31:32,811
And I think it starts looking a lot more like product work and a lot less about
specifically where does the semicolon go?

527
00:31:32,811 --> 00:31:36,593
Or should this be an abstract base class or should we use composition?

528
00:31:36,593 --> 00:31:37,708
To some degree, who cares?

529
00:31:37,708 --> 00:31:44,202
It's hard as a like a experienced engineer seeing some of the code that's being generated
and not having strong opinions about what's there.

530
00:31:44,202 --> 00:31:46,283
Early on I tried to be like, Yeah, you know what?

531
00:31:46,283 --> 00:31:52,646
I'm gonna try the vibe coding strategy where I don't look at it and I just try to get it
to fix things.

532
00:31:52,646 --> 00:32:00,088
But over time I realized that a lot of the sort of issues that I was seeing were the
result of mistakes that were made.

533
00:32:00,088 --> 00:32:09,896
during the generation process and could only have been fixed by having a more intentioned
approach to as the code is being generated, steering it better, because after the fact

534
00:32:09,896 --> 00:32:13,759
it's now like permanently affecting the context in a negative way.

535
00:32:13,759 --> 00:32:15,022
It it's completely poisoned it.

536
00:32:15,022 --> 00:32:17,720
How do you as a human know if the rate code's being generated?

537
00:32:17,720 --> 00:32:19,441
See, this is the interesting thing.

538
00:32:19,441 --> 00:32:22,994
This is why I think I am both a great and terrible software engineer.

539
00:32:22,994 --> 00:32:27,497
I'll say that I have opinions on the aesthetic beauty of some code.

540
00:32:27,497 --> 00:32:35,096
And the closer to that aesthetic outcome that I I'm hoping to have or that I see as pretty
tells me whether or not it's good.

541
00:32:35,096 --> 00:32:44,689
Yeah, I think that that there's a difference between software as an object of art and
software as a utilitarian thing.

542
00:32:44,689 --> 00:32:49,570
Now, I happen to own clothes like the shirt I'm wearing that is largely utilitarian.

543
00:32:49,570 --> 00:32:54,311
No one cared about the weave of this fa this cotton fabric, the structure of it.

544
00:32:54,311 --> 00:32:58,942
Someone then put logos on it and added an artistic layer on top.

545
00:32:58,942 --> 00:33:04,814
I also own sweaters that were handmade by someone who really cared.

546
00:33:04,814 --> 00:33:05,734
About that sweater.

547
00:33:05,734 --> 00:33:10,038
And those are different objects that cost different amounts and serve different purposes.

548
00:33:10,038 --> 00:33:15,782
I think similarly, we can have software that's not made so that I feel good as an author.

549
00:33:15,782 --> 00:33:19,795
In the same way that whoever made this shirt didn't feel good about the process.

550
00:33:19,795 --> 00:33:20,675
That's just that work.

551
00:33:20,675 --> 00:33:21,566
That's the job.

552
00:33:21,566 --> 00:33:31,093
And then there's a different scale where you're like, No, no, I I'm gonna care and I'm
going to pull out my loom and like weave this thing.

553
00:33:31,093 --> 00:33:32,494
And like that's a different hobby.

554
00:33:32,494 --> 00:33:32,896
I I

555
00:33:32,896 --> 00:33:33,806
With what we were going with here.

556
00:33:33,806 --> 00:33:41,248
And I I think the problem is that that view, I think, may be shared amongst a lot of
people, but they're not taking it to its natural conclusion.

557
00:33:41,248 --> 00:33:52,892
Which for me is there only exists two types of companies that make software: giant
monolith software production facilities that have optimized the very low quality but can

558
00:33:52,892 --> 00:33:54,332
generate tons of stuff.

559
00:33:54,332 --> 00:34:01,226
And those companies that believe that the software they're writing is a competitive
advantage and are writing the highest quality.

560
00:34:01,226 --> 00:34:05,170
artisanal software outward artifacts that you could imagine.

561
00:34:05,170 --> 00:34:14,648
And if there is no in-between here, the thought that I have that goes along with that is
if you are just someone that comes into the job and you are clicking a button to generate

562
00:34:14,648 --> 00:34:24,156
the code and you click that button over and over again, or to quote Dorota, pull the hot
garbage slot machine lever one more time ah and see and see what comes out, you're

563
00:34:24,156 --> 00:34:26,769
basically saying I don't matter in this process.

564
00:34:26,769 --> 00:34:28,545
My company doesn't care about

565
00:34:28,545 --> 00:34:31,907
the quality of what comes out, I'm not necessary here.

566
00:34:31,927 --> 00:34:37,450
And and worse, you are losing your skills in writing better quality software over time.

567
00:34:37,450 --> 00:34:40,012
You're literally building the thing in a way to replace you.

568
00:34:40,012 --> 00:34:50,498
So I think you need to be conscious of whether or not you're going towards the I'm trying
to be the best software engineer, uh, build the best, highest quality software that hits

569
00:34:50,498 --> 00:34:51,538
whatever

570
00:34:51,638 --> 00:35:00,305
you know, very unique luxury standpoint, or you're going toward the direction of software
factory output as much as you can job, even though they're not doing differentiated work

571
00:35:00,305 --> 00:35:01,752
from a com corporate standpoint.

572
00:35:01,752 --> 00:35:09,049
Think through the project that you have that you felt the most attached to and you felt s
the proudest about.

573
00:35:09,049 --> 00:35:12,552
That it most embodies for you this artistic representation of give a shit.

574
00:35:12,552 --> 00:35:14,264
Do you have something like in mind roughly?

575
00:35:14,264 --> 00:35:18,373
See, I don't know if this argument works on me because I just hate everything I've ever
made.

576
00:35:18,373 --> 00:35:21,954
It's literally the worst software that has ever been generated.

577
00:35:21,954 --> 00:35:22,534
Yeah, okay.

578
00:35:22,534 --> 00:35:26,577
So like if you're gonna ship crap anyway, why don't you just ship crap faster?

579
00:35:27,699 --> 00:35:33,023
Even in the most artistically beautiful thing, you didn't write your own date time
library.

580
00:35:33,484 --> 00:35:37,047
Like there are times when it doesn't make sense to write it yourself.

581
00:35:37,047 --> 00:35:43,572
And I'm not saying that every every company needs to do one hundred percent of all things
through from the factory.

582
00:35:44,833 --> 00:35:46,665
for sure.

583
00:35:46,665 --> 00:35:48,554
And then someone printed the design on.

584
00:35:48,554 --> 00:35:49,017
Right.

585
00:35:49,017 --> 00:35:50,702
So there's a version of this where

586
00:35:50,702 --> 00:35:56,712
To the creation of this product was part factory, part human, like bringing w the human
pieces to it.

587
00:35:56,712 --> 00:35:58,464
So it's not just a generic black shirt.

588
00:35:58,464 --> 00:36:07,276
If the step that you're providing as value is doing the automation on top, then you're not
creating any sort of competitive advantage amongst every other company that is literally

589
00:36:07,276 --> 00:36:07,667
doing the thing.

590
00:36:07,667 --> 00:36:13,478
And I think, you know, maybe there's still an argument here because there's still tons of
companies out there that rent a fabric press for shirts, right?

591
00:36:13,478 --> 00:36:15,499
And another company rents that thing.

592
00:36:15,499 --> 00:36:18,359
They go and they just use the machine, and that's the whole job.

593
00:36:18,359 --> 00:36:20,400
There is no differentiating aspect there.

594
00:36:20,400 --> 00:36:24,611
The one difference with the physical world process, though, is locality.

595
00:36:24,611 --> 00:36:27,554
They can sell that shirt in a

596
00:36:27,554 --> 00:36:31,638
you know, in an area that's around wherever the business is is centered, right?

597
00:36:31,638 --> 00:36:35,853
They have a competitive advantage on location versus everyone else in the world that can
do it.

598
00:36:35,853 --> 00:36:38,886
Now, obviously your neighbor can compete with you by doing the same thing.

599
00:36:38,886 --> 00:36:42,049
And so there has to be other strategies you you bring to play with.

600
00:36:42,049 --> 00:36:49,357
Like you have a a commitment from your customers or from your suppliers and where you get
the shirts from, you know, you know someone who's willing to sell you them a little bit

601
00:36:49,357 --> 00:36:50,358
cheaper or

602
00:36:50,358 --> 00:36:58,414
You have lock in because you're buying so many shirts from your your shirt manufacturer
that you get a good deal and then you can sell them at a lower cost in order to compete

603
00:36:58,414 --> 00:37:05,191
with you, someone else would have to come in and take a hit on how much they're selling
the shirts for in order to actually have a lower until they put you out of business and

604
00:37:05,191 --> 00:37:06,552
they can raise the price again.

605
00:37:06,552 --> 00:37:08,994
And so, you know, you're playing a fundamentally different game there.

606
00:37:08,994 --> 00:37:17,431
But I think fundamentally if your job is, you know, you take the lever and push it down
to, you know, s print out the stamp on the shirt, like your job is fundamentally, you

607
00:37:17,431 --> 00:37:18,370
know, replaceable.

608
00:37:18,370 --> 00:37:21,851
Convert this thing that the product person says into working code, please.

609
00:37:22,211 --> 00:37:24,472
And there's actual judgment that comes.

610
00:37:24,472 --> 00:37:26,481
Do we have a monorepo or polyrepo set up?

611
00:37:26,481 --> 00:37:29,673
Are we going to use a SQL database or a non-SQL database?

612
00:37:29,673 --> 00:37:33,624
How important is that database decision today versus five years from now?

613
00:37:33,624 --> 00:37:35,295
And how easily could we reverse it?

614
00:37:35,295 --> 00:37:37,105
Is like now the time to make that change?

615
00:37:37,105 --> 00:37:38,626
I also edit a lot of YAML.

616
00:37:38,626 --> 00:37:42,477
And so like of those, I'd like to get rid of the YAML editing.

617
00:37:42,477 --> 00:37:45,147
And and I'd like to stay more on the like, what are we doing?

618
00:37:45,147 --> 00:37:45,938
Why are we doing it?

619
00:37:45,938 --> 00:37:47,182
And what are the implications?

620
00:37:47,182 --> 00:37:55,592
Are you optimistic that the trajectory of your career, you know, maybe in the next five,
ten years is towards the direction where you'll be able to make more of those decisions

621
00:37:55,592 --> 00:38:06,244
and write less YAML or write more YAML and have L LMs or other decision based AI out there
to take over the responsibility of figuring out what is the appropriate database and to

622
00:38:06,244 --> 00:38:08,300
deliver the b relevant business value?

623
00:38:08,300 --> 00:38:13,854
I'm more likely to move the up the stack towards intent than down the stack into edit.

624
00:38:13,854 --> 00:38:21,397
On your path to becoming a principal engineer and obviously the way you're you're talking
about, no matter what it is, there is a very deliberate nature to how you're approaching

625
00:38:21,397 --> 00:38:26,779
that and what problem you're trying to solve and which tool you're you know you're pulling
in to make that happen.

626
00:38:26,779 --> 00:38:35,992
How specifically would you think that the addition of utilizing L LMs has has changed the
process of what you're doing or in the types of problems that you're solving?

627
00:38:35,992 --> 00:38:37,973
Types of problems that I'm solving are identical.

628
00:38:37,973 --> 00:38:43,908
I'm I'm still having the same conversation that I had with those people at eBay ten years
later.

629
00:38:43,908 --> 00:38:46,350
And that didn't get harder or easier with LLMs.

630
00:38:46,350 --> 00:38:50,013
It became more important because we have a lot more pull requests showing up.

631
00:38:50,013 --> 00:38:58,569
The questions we can ask and the way that we communicate is more important, but also
there's a an asymmetry to effort versus output that is really dangerous for us that we

632
00:38:58,569 --> 00:39:02,456
need to pay attention to because we don't want to denial a service our coworkers.

633
00:39:02,456 --> 00:39:12,813
I have this theory that the amount of value that the thing is being produced is still
proportional with a ceiling of the amount of human timeslash effort that's put in.

634
00:39:12,813 --> 00:39:20,698
So as a human, you put in less effort or less time into it, it's going to actually
decrease the quality or the value of the thing that comes out.

635
00:39:20,698 --> 00:39:24,621
I mean, obviously I have no no no evidence for this whatsoever.

636
00:39:24,621 --> 00:39:32,046
But it does feel like this sort of thing where if you aren't paying attention to what
you're doing and you are just passing outputs from LM LM.

637
00:39:32,440 --> 00:39:36,617
You're not putting in a lot of effort and therefore the output product can't can't be
good.

638
00:39:36,617 --> 00:39:37,634
Can't be good enough.

639
00:39:37,634 --> 00:39:39,177
How that usually works.

640
00:39:39,298 --> 00:39:42,660
where you spend six hours trying to automate something that takes one hour a year.

641
00:39:42,732 --> 00:39:52,132
and maybe the problem is when you say use an L L it's like a proportion of value, like the
amount of time that you're spent to making yourself to create something of durable value,

642
00:39:52,132 --> 00:39:53,742
does has that changed?

643
00:39:53,742 --> 00:39:54,904
Yeah, it's definitely changed.

644
00:39:54,904 --> 00:40:03,060
I can have an L L write me an end to end test with playwright with no additional L L
investment and I I maintain value out of that.

645
00:40:03,060 --> 00:40:09,135
Your relative confidence level on test writing that LMs have, like low, medium, high.

646
00:40:10,716 --> 00:40:11,417
Interesting.

647
00:40:11,417 --> 00:40:13,078
Do you know what makes the difference?

648
00:40:13,084 --> 00:40:22,859
I'm like, Do I do I think that an LLM could write a good suite of unit tests with the aid
of mutation testing that would accurately test an algorithm?

649
00:40:22,859 --> 00:40:25,940
Hundred percent.

650
00:40:26,160 --> 00:40:33,283
Do I think that the L LM can ensure my UI is functional through clicking?

651
00:40:33,623 --> 00:40:33,984
No.

652
00:40:33,984 --> 00:40:35,024
No, I don't.

653
00:40:35,904 --> 00:40:39,938
and so like at different levels it'll be different levels of good.

654
00:40:39,938 --> 00:40:48,423
I like the idea that there's such a high disparity here between the two sides of the
spectrum that it flips between depending on really the the type of focus task.

655
00:40:48,423 --> 00:40:55,607
But I I do have seen that there's this weird thing where maybe it's great at sort of first
level design and UI generation, much more than I am.

656
00:40:55,607 --> 00:40:59,589
But when it comes to validation of it, I do agree that there is this it's not as great.

657
00:40:59,589 --> 00:41:03,972
Whereas it's much better at following a spec and generating code and the tests for that.

658
00:41:03,972 --> 00:41:06,235
And so I I'm not I'm not sure where I'm going with that.

659
00:41:06,235 --> 00:41:07,840
It's just sort of an interesting learning.

660
00:41:07,840 --> 00:41:10,897
So at this point, Justin, I think we should move over to to picks.

661
00:41:10,897 --> 00:41:13,186
So what did you what did you bring for the audience today?

662
00:41:13,186 --> 00:41:13,986
Yeah.

663
00:41:14,387 --> 00:41:17,869
so there's a genre of thing that I have found through TikTok.

664
00:41:17,869 --> 00:41:20,591
Uh and it is called Isakai.

665
00:41:20,591 --> 00:41:26,094
Uh Isakai is a genre of literature w that's based on this premise.

666
00:41:26,575 --> 00:41:31,478
You die, you are reincarnated, and you are reincarnated into a video game.

667
00:41:31,478 --> 00:41:36,882
Now you have like a stats screen that you can look at and you have skills and they get
better as you like invest.

668
00:41:36,882 --> 00:41:39,124
And I didn't realize that this was a genre.

669
00:41:39,124 --> 00:41:41,985
Uh I picked up this book called Dead World Isakai.

670
00:41:41,985 --> 00:41:43,032
It's a trilogy.

671
00:41:43,032 --> 00:41:47,745
Of a guy who is Isakai into a world that has no inhabitants.

672
00:41:47,745 --> 00:41:48,725
It has no life.

673
00:41:48,725 --> 00:41:49,856
It's just this dead world.

674
00:41:49,856 --> 00:41:50,386
And he's there.

675
00:41:50,386 --> 00:41:52,227
He's like, okay, now what do I do?

676
00:41:52,227 --> 00:41:55,329
And it's him, it's like a man versus machine kind of story.

677
00:41:55,329 --> 00:41:57,390
And Isakai is, I think, really interesting.

678
00:41:57,390 --> 00:42:00,802
And that that's my pick, uh, because I don't think many people ha know about it.

679
00:42:00,802 --> 00:42:03,274
And they get real off the rails.

680
00:42:03,274 --> 00:42:08,056
Um, one of the examples is someone who is isekai into a pig.

681
00:42:08,056 --> 00:42:11,538
There's like this uh anim anime about it.

682
00:42:11,676 --> 00:42:16,963
and I've seen like little s snippets of it from TikTok and it's I need to get my s like
stamina up, okay.

683
00:42:16,963 --> 00:42:20,939
So I need to like drink this milk and eat this food and become a big pig.

684
00:42:20,939 --> 00:42:23,362
And they're taking me somewhere to the to the market.

685
00:42:23,362 --> 00:42:24,514
I wonder what that's about.

686
00:42:24,514 --> 00:42:28,148
I need to escape and like it's so it's this this video game anime thing.

687
00:42:28,148 --> 00:42:29,019
It's fascinating.

688
00:42:29,019 --> 00:42:29,392
I don't

689
00:42:29,392 --> 00:42:30,683
if I want to ruin this for you.

690
00:42:30,683 --> 00:42:41,127
Um, it's such a huge anime genre, so much so that the thing I don't know the pig one is,
uh, is like a parody of other Isekais on how ridiculous they are.

691
00:42:41,127 --> 00:42:47,210
The most parody one is that the main character got uh reborn into a vet being a vending
machine.

692
00:42:47,210 --> 00:42:48,810
That's that's the whole concept.

693
00:42:48,810 --> 00:42:51,817
I don't think it's good, but it's a thing that someone can enjoy.

694
00:42:51,817 --> 00:42:54,373
and there's like multiple seasons of that.

695
00:42:54,373 --> 00:42:57,516
For like I don't know what you do after the first one, honestly.

696
00:42:57,516 --> 00:42:59,547
But uh that goes on for a while.

697
00:42:59,547 --> 00:43:00,528
There there are quite some things.

698
00:43:00,528 --> 00:43:09,116
And you know what the interesting thing is like it is it has definitely grown and it's
grown so ridiculously that now the titles for those have also sort of gotten quite

699
00:43:09,116 --> 00:43:09,816
ridiculous.

700
00:43:09,816 --> 00:43:16,281
Where like I'm sure the title for the vending machine one like I got reincarnated into
being a vending machine in my next life or something.

701
00:43:16,281 --> 00:43:16,832
Like that's it.

702
00:43:16,832 --> 00:43:24,058
Like that's it's not even trying to come up with a like a clever, like a real name for it
like you would see in in Western media.

703
00:43:24,058 --> 00:43:26,922
I think the most canonical one is probably sort sort art online.

704
00:43:26,922 --> 00:43:29,363
So these guys isn't necessarily limited to just video games.

705
00:43:29,363 --> 00:43:37,005
Uh can be just anything in a new world, which I think there's a huge percentage of them
that are like I have a stat screen that no one else has for some reason.

706
00:43:37,005 --> 00:43:37,795
Yeah, okay.

707
00:43:37,795 --> 00:43:38,665
Well that what a pick.

708
00:43:38,665 --> 00:43:43,367
Uh I can I can guarantee you that no one has no one has picked that so far on the podcast.

709
00:43:43,367 --> 00:43:45,287
So you've got something unique there.

710
00:43:46,968 --> 00:43:48,208
Yeah, my pick.

711
00:43:48,208 --> 00:43:49,509
Uh always a challenge.

712
00:43:49,509 --> 00:43:54,582
What I'm going with is this research that was published earlier this year on

713
00:43:54,718 --> 00:44:03,645
The total sum of domain name registrations, uh, it's like 35 pages, so it's it's it's a
little bit long, but it has quite some interesting stats in it.

714
00:44:03,786 --> 00:44:08,840
And it turns out that like 10% of all domain names that are registered are malicious in
nature.

715
00:44:08,840 --> 00:44:12,153
Like someone registered it purely for malicious intent.

716
00:44:12,153 --> 00:44:13,915
It's the inter IEL Consulting Group.

717
00:44:13,915 --> 00:44:18,919
I don't know if there's like a title for this report other than like malicious
registrations in the domain market.

718
00:44:18,919 --> 00:44:21,040
There'll be a link in the description of the episode.

719
00:44:21,040 --> 00:44:23,010
There's some interesting things in here, like

720
00:44:23,010 --> 00:44:25,351
Malicious actors just like burning domains left and right.

721
00:44:25,351 --> 00:44:31,604
Like there are so many domains that just don't get renewed because they were only
registered for malicious behavior.

722
00:44:31,604 --> 00:44:39,957
And after after they're used, they obviously burned every every denialist says no, you
can't use this domain because it's associated with malicious activity.

723
00:44:39,957 --> 00:44:44,879
And so over time you can start creating some stats about where these domains are coming
from.

724
00:44:44,879 --> 00:44:51,342
There are some registrars that are just way more malicious than others, and not
necessarily the registrar themselves, but the domains that are registered through that

725
00:44:51,342 --> 00:44:52,182
registrar.

726
00:44:52,182 --> 00:44:56,004
Are more likely to be for malicious actors than not.

727
00:44:56,004 --> 00:45:01,866
And certain TLDs like the top-level domain, like dot coms and whatever, could also be
associated with malicious activity.

728
00:45:01,866 --> 00:45:09,869
And I just find it really interesting that there are some domains that are just known for
how likely the a bad actor is using them.

729
00:45:09,869 --> 00:45:20,886
So the things like the.xyz domains or dot top or dot info have a disproportionately huge
number of actors that are using them for malicious purposes, more so than anything else.

730
00:45:20,886 --> 00:45:22,848
And I just I don't know what to make of this research.

731
00:45:22,848 --> 00:45:23,363
Yeah.

732
00:45:23,363 --> 00:45:24,819
Uh I mean those aren't free.

733
00:45:24,819 --> 00:45:25,603
I'm very curious.

734
00:45:25,603 --> 00:45:26,774
It's gotta be worth it.

735
00:45:26,774 --> 00:45:32,286
Obviously, if you're gonna do something, I mean they're like one to a hundred dollars per
domain.

736
00:45:32,286 --> 00:45:37,527
So if you're gonna go out on a some sort of fishing expedition, it's probably worth it to
buy one.

737
00:45:37,527 --> 00:45:41,798
And I the question I'd actually have is I wonder what the like the half life is for using
the domain.

738
00:45:41,798 --> 00:45:49,780
Like obviously you get it for a year by renewal, but I wonder if it stops being used like
after one month or two months, like once you run the campaign already.

739
00:45:49,780 --> 00:45:51,571
And so there's domains floating around there.

740
00:45:51,571 --> 00:45:56,536
It's a good reminder that there are so many domains that are being registered and by
certain

741
00:45:56,536 --> 00:46:01,568
TLDs that you should be careful about which TLDs you're picking because the whole TLD
might be blocked.

742
00:46:01,568 --> 00:46:07,019
Like for us, I know in one of our products, we automatically block any XYZ usage.

743
00:46:07,019 --> 00:46:10,380
I know we're blocking it at corporate email receiving level.

744
00:46:10,380 --> 00:46:15,892
Like if you have an XYZ domain, your email is not going to get to us just because
statistically it's really bad.

745
00:46:15,972 --> 00:46:23,590
And if you're buying a new domain, you really need to check where who's had it before
because it's highly likely that someone's picked it up, burned it.

746
00:46:23,590 --> 00:46:28,253
And now you're basically paying for this experience of using this new domain.

747
00:46:28,253 --> 00:46:28,672
Yep.

748
00:46:28,672 --> 00:46:30,164
I have that same concern.

749
00:46:30,164 --> 00:46:38,854
I my my domain is dot S, which is Montesarat, um, which is usually not an issue but is
occasionally uh an issue.

750
00:46:38,854 --> 00:46:41,075
I don't think I've ever seen a dot MS usage there.

751
00:46:41,075 --> 00:46:49,860
So there's an interesting thing with the G TLDs, the generic TLDs, the ones that aren't
assigned to INA set for for countries that I'm always worried about.

752
00:46:49,860 --> 00:46:54,503
Uh but that's that's definitely another t another topic that we're not gonna get into
here.

753
00:46:54,503 --> 00:46:57,184
So uh thank you, Justin, for for coming on the show.

754
00:46:57,184 --> 00:47:02,647
I've I've enjoyed your your hot takes uh in the world of LLM coding and uh pull requests.

755
00:47:03,167 --> 00:47:08,612
Uh and thanks for all the viewers for tuning into this episode and I hope to see everyone
back again next.

