1
00:00:07,896 --> 00:00:10,058
Welcome back to Adventures in DevOps.

2
00:00:10,058 --> 00:00:12,991
Every episode is a deep dive with an expert guest.

3
00:00:12,991 --> 00:00:18,157
Today's adventure focuses on automation and AI and hopefully a combination of the two.

4
00:00:18,157 --> 00:00:25,864
The expert is long time architecture consultant on everything web and now principal of
DevRel, JavaScript, AI and cloud at Microsoft.

5
00:00:25,864 --> 00:00:27,714
Dan Wallin, welcome to the show.

6
00:00:27,714 --> 00:00:31,118
Hey, great to be here, Warren, and look forward to the conversation with you.

7
00:00:31,118 --> 00:00:38,778
Yeah, you know, I couldn't help but notice on your LinkedIn profile, you were getting out
of consulting at the exact moment that I feel like many people were getting in.

8
00:00:38,778 --> 00:00:41,990
The start of the pandemic must have been quite the transformation for you.

9
00:00:42,146 --> 00:00:45,979
You know, it was I, I ran a consulting company.

10
00:00:45,979 --> 00:00:55,488
did a lot of architecture and then coding and then also a lot of training as well across
pretty much the U S but we also did some international.

11
00:00:55,488 --> 00:01:00,673
But anyway, long story short, did that for about 20 years and I traveled a lot.

12
00:01:00,673 --> 00:01:03,986
And I mean, sometimes three out of four weeks a month.

13
00:01:03,986 --> 00:01:12,771
And when you, you know, when I have kids and a family and all that, and after 20 years,
when that time period hit, I was kind of like, okay, it's kind of nice staying at home,

14
00:01:12,771 --> 00:01:14,612
you know, where everybody else was like, I want to get out.

15
00:01:14,612 --> 00:01:15,953
I'm like, I'm okay.

16
00:01:15,953 --> 00:01:17,694
Like I can stay at home.

17
00:01:17,694 --> 00:01:18,474
This is great.

18
00:01:18,474 --> 00:01:27,220
um And yeah, so that's when an opportunity at Microsoft came up and I'm like, you know,
this feels like a good time to change things up.

19
00:01:27,220 --> 00:01:31,882
And yeah, so here we are today's almost six years later now.

20
00:01:31,884 --> 00:01:34,951
Yeah, it almost seems like it's been forever or it didn't exist.

21
00:01:34,951 --> 00:01:37,086
I don't really know how time works since the pandemic.

22
00:01:37,086 --> 00:01:39,116
It's always been a little bit confusing for me.

23
00:01:39,116 --> 00:01:40,951
Isn't it just the strangest?

24
00:01:40,951 --> 00:01:43,196
Yeah, because I it feels like it was forever.

25
00:01:43,196 --> 00:01:45,632
But at the same time it wasn't that long ago.

26
00:01:45,632 --> 00:01:47,214
But anyway, yeah

27
00:01:47,214 --> 00:01:55,714
Well, I still see stuff happening from 2015 or 2016 and that's been a decade now and it
really doesn't feel that long.

28
00:01:56,216 --> 00:02:04,081
Yeah, as I get older, my mom always used to be like, I'm telling you, enjoy it when you're
young because time moves faster when you get older.

29
00:02:04,081 --> 00:02:05,339
I'm like, nah.

30
00:02:05,339 --> 00:02:07,693
And now I'm like, yeah, mom was right.

31
00:02:07,693 --> 00:02:09,845
Mom was right.

32
00:02:10,146 --> 00:02:17,505
So as the Microsoft Cloud advocate here, I think we need to hear a pitch of what Azure is
doing better than everyone else these days.

33
00:02:18,815 --> 00:02:30,633
I think the big area that Azure excels in in general is we're like the ultimate Lego
building block cloud.

34
00:02:30,633 --> 00:02:32,774
Anything you want is there.

35
00:02:32,774 --> 00:02:42,174
I think the challenge with that, of course, is when you have all the security features and
all the, you know, some of the DevOps stuff, of course, the different services and, you

36
00:02:42,174 --> 00:02:43,492
know, the list goes on and on and on.

37
00:02:43,492 --> 00:02:47,274
I think the challenge we always face is how do you

38
00:02:47,564 --> 00:02:50,419
make it so that there's a really good Lego manual.

39
00:02:51,074 --> 00:02:52,307
like the comparison to LEGOs.

40
00:02:52,307 --> 00:02:58,189
I like the comparison that, you know, they're individual pieces that you can stick
together with security modules, etc.

41
00:02:58,189 --> 00:03:02,114
The one thought that does come to my mind is stepping on LEGO pieces.

42
00:03:02,114 --> 00:03:05,105
Yeah, well, uh I've done that before.

43
00:03:05,105 --> 00:03:06,596
It's very painful.

44
00:03:06,856 --> 00:03:07,846
Only thing worse than that.

45
00:03:07,846 --> 00:03:10,527
I live in the desert in Arizona and is scorpions.

46
00:03:10,527 --> 00:03:11,388
Those are even worse.

47
00:03:11,388 --> 00:03:14,939
But anyway, Legos are right up there with scorpions.

48
00:03:15,780 --> 00:03:25,954
Yeah, and I think that's the challenge, you know, from especially from an enterprise
standpoint, um having worked with enterprises, big, big enterprises for, you know, 20 plus

49
00:03:25,954 --> 00:03:27,844
years in the consulting world.

50
00:03:28,245 --> 00:03:31,352
They absolutely need that flexibility.

51
00:03:31,352 --> 00:03:38,847
But at the same time, if you're uh a startup or something who's brand new, you know, to
the cloud, like you have this fantastic idea.

52
00:03:38,847 --> 00:03:44,280
You used AI to maybe even vibe code the idea just to get a prototype out there to
experiment with.

53
00:03:44,280 --> 00:03:46,951
Because it seems like that's what we're doing these days, Warren.

54
00:03:46,951 --> 00:03:55,777
That's where I think it can be a little challenging, actually, is the you're just kind of
overwhelmed by there's so much power there, you know, that you can take advantage of.

55
00:03:55,777 --> 00:03:58,390
And that's kind of part of.

56
00:03:58,390 --> 00:04:00,511
my job actually that I do now.

57
00:04:00,511 --> 00:04:09,096
I kind of work on end to end solutions with AI all the way from, I have an idea to I need
to get it up there somehow.

58
00:04:09,096 --> 00:04:11,137
You know, what does that process look like?

59
00:04:11,137 --> 00:04:13,358
And it's super fun.

60
00:04:13,899 --> 00:04:16,140
But it's also there's just a lot to know.

61
00:04:16,140 --> 00:04:24,485
You know, and I know you working in this world, you could probably sympathize because
it's, you know, it's not unique to Azure, AWS, uh GCP.

62
00:04:24,485 --> 00:04:26,826
There's just a lot of power available.

63
00:04:27,250 --> 00:04:30,442
I really like this analogy and I'm tempted to bring it even further.

64
00:04:30,442 --> 00:04:37,475
You mentioned basically having the guides or the construction manuals for whatever the end
product is.

65
00:04:37,495 --> 00:04:48,340
And I think this is something that historically used to be concrete building blocks where
you would actually go out and have a repository that was just literally the code, the

66
00:04:48,340 --> 00:04:55,243
infrastructure that should be deployed, the individual services that need to be turned on,
and the configuration that has to be there in order to get the end service product,

67
00:04:55,243 --> 00:04:55,994
whatever that you're building.

68
00:04:55,994 --> 00:04:57,004
Let's say it's a recommendation.

69
00:04:57,004 --> 00:05:07,877
engine or some sort of website and the corollary that I want to draw is that a lot of
times and especially now we don't have the book the construction manual for whatever the

70
00:05:07,877 --> 00:05:17,472
thing is we're building anymore I feel like it's gotten a lot leaner especially since LLMs
have come around I feel like we do get those a lot less frequently or

71
00:05:17,472 --> 00:05:21,966
On the flip side, get a lot more that are somehow like, you could do this weird thing.

72
00:05:22,087 --> 00:05:26,913
So on vibe, like some engineer at Azure, I look at AWS, so I see this all the time.

73
00:05:26,913 --> 00:05:31,378
Someone vibe coded a service with specific functionality to work on AWS.

74
00:05:31,378 --> 00:05:33,080
And I'm just like, what was the point of that?

75
00:05:33,080 --> 00:05:35,372
uh Is that actually helping anyone?

76
00:05:35,616 --> 00:05:41,471
Yeah, and you bring up a really good point though with, you know, LLMs and just the AI in
general.

77
00:05:41,611 --> 00:05:50,820
just literally last week, I'm on a project right now that helps with basically the
deployment of popular open source projects to Azure.

78
00:05:50,820 --> 00:05:58,566
And uh I would argue that if you didn't know, you know, AWS or Azure or GCP or whatever
really well,

79
00:05:58,678 --> 00:06:06,375
It could actually be kind of challenging because some of these these OSS projects,
especially some of the bigger, more popular ones, there's quite a few moving parts to

80
00:06:06,375 --> 00:06:07,106
them.

81
00:06:07,106 --> 00:06:15,213
So it's not just as simple as, know, I'm deploying like a web app service and I'm good or,
you know, serverless function or something like that.

82
00:06:15,213 --> 00:06:16,855
There's there's a bit more to the story.

83
00:06:16,855 --> 00:06:24,842
And I've been pretty amazed, I'll have to admit, with, you know, skills are just a really
big thing now.

84
00:06:24,842 --> 00:06:26,264
in the AI world.

85
00:06:26,264 --> 00:06:28,598
And it turns out it works.

86
00:06:28,598 --> 00:06:32,494
They work really, really well for these deployment scenarios.

87
00:06:32,494 --> 00:06:34,217
So I'll give you an example.

88
00:06:34,217 --> 00:06:36,621
One of the ones I'm working with is uh N8N.

89
00:06:36,621 --> 00:06:37,962
I don't know if you've ever.

90
00:06:38,068 --> 00:06:41,055
Oh yeah, sure, it's the workflow orchestrator.

91
00:06:41,100 --> 00:06:46,832
Yeah, the workflow orchestrator and um it's actually not too bad to actually deploy.

92
00:06:46,832 --> 00:06:51,673
There's like some moving parts, but it's not ridiculous compared to some of the others.

93
00:06:51,673 --> 00:06:57,364
And uh by just setting up a couple core skills that know what to do.

94
00:06:57,364 --> 00:07:00,175
I think in this case, I deployed to Azure Container apps.

95
00:07:00,175 --> 00:07:04,516
Actually, I'm a huge container fan, by the way, in general.

96
00:07:04,516 --> 00:07:11,328
But without the skill, we have something called, uh well, you probably heard Terraform,
I'm sure.

97
00:07:11,770 --> 00:07:15,753
And then, you we have Bicep as well on the Microsoft side too.

98
00:07:15,753 --> 00:07:24,259
And you can with this command line tool called Azure Developer CLI, you can pretty easily
provision and then deploy, you your app.

99
00:07:24,259 --> 00:07:27,801
But you have to have your infrastructure as code in place.

100
00:07:27,801 --> 00:07:31,363
And that's the part like for me, that's just not my expertise, Warren.

101
00:07:32,725 --> 00:07:37,018
You're like, hey, Dan, whip up this, you know, from memory on the fly.

102
00:07:37,018 --> 00:07:40,650
I'd be like, do I get my AI tools?

103
00:07:40,958 --> 00:07:43,718
Because on my own, it's just not what I do.

104
00:07:43,718 --> 00:07:45,238
You know, it's not my specialty.

105
00:07:45,818 --> 00:07:48,378
These skills have really changed the game, though.

106
00:07:48,378 --> 00:07:56,898
Like, you know, it's I can't remember if it was the Matrix or Terminator or whatever the
movie was, but they got to go fly a helicopter and they like, you know, plug in.

107
00:07:56,898 --> 00:07:58,972
How do you not know the reference?

108
00:07:58,972 --> 00:08:00,063
That's just...

109
00:08:01,466 --> 00:08:07,035
It's a matrix because Trinity and Neo need to rescue Morpheus.

110
00:08:07,806 --> 00:08:10,291
There's one in Terminator 2, isn't there?

111
00:08:10,291 --> 00:08:10,779
I don't know.

112
00:08:10,779 --> 00:08:12,435
I haven't watched Terminator in so long.

113
00:08:12,435 --> 00:08:14,378
I was thinking it was Matrix and all...

114
00:08:14,378 --> 00:08:16,061
Hey, I'm gonna defend myself here.

115
00:08:16,061 --> 00:08:21,238
Yeah, you're right though, because I love Matrix, but I haven't seen it in a while.

116
00:08:21,238 --> 00:08:23,349
And I would recommend not going back and watching it.

117
00:08:23,349 --> 00:08:30,611
There are some parts that are so great, but watching some old movies, I just don't think
they stand out.

118
00:08:30,611 --> 00:08:42,524
One thing I do have to ask you is you said you're a huge Container fan, and I worry that
the alternative is being like, oh, I love on-prem hardware-based architecture where I just

119
00:08:42,524 --> 00:08:44,615
go and stand everything up from the basement.

120
00:08:44,615 --> 00:08:47,316
Is that the alternative world that you see that people are utilizing?

121
00:08:47,316 --> 00:08:48,770
you know...

122
00:08:48,770 --> 00:08:54,898
physical machines or virtual machines or are you comparing it amongst like a set of
something more than that?

123
00:08:55,018 --> 00:09:00,683
No, virtual machines is definitely the world I've win, especially back in the consulting
days.

124
00:09:00,683 --> 00:09:02,484
It was almost all virtual machines.

125
00:09:02,484 --> 00:09:04,405
Now there was plenty of on-prem.

126
00:09:04,405 --> 00:09:11,334
I used to work with a lot of uh financial companies, big, big financial companies, and one
was a big credit card company.

127
00:09:11,334 --> 00:09:15,084
In fact, if we want stories, I got stories that go way back.

128
00:09:15,084 --> 00:09:16,207
Let's go into it.

129
00:09:16,207 --> 00:09:19,112
I want to hear about VMs and credit card companies.

130
00:09:19,112 --> 00:09:20,633
Not fun stories.

131
00:09:20,633 --> 00:09:23,906
Yeah, well, this goes way back to the days of calm and calm plus.

132
00:09:23,906 --> 00:09:25,057
So this kind of dates me.

133
00:09:25,057 --> 00:09:28,870
But anyway, we can get to that later if you want.

134
00:09:29,151 --> 00:09:32,133
That was quite the way to break into this world.

135
00:09:32,133 --> 00:09:39,182
anyway, uh yeah, normally VMS and and, you know, just deploying the containers to the VM
at a minimum.

136
00:09:39,182 --> 00:09:45,274
Yeah, I mean, I've seen I've seen some companies as simple as we're going to use Docker
compose and we're going to get the containers running.

137
00:09:45,274 --> 00:09:48,146
And then you're like, what happens if the container goes down?

138
00:09:49,158 --> 00:09:50,538
And how are you going to scale that?

139
00:09:50,538 --> 00:09:55,778
Because, you know, it's not really designed for that unless you're going to Kubernetes or
something like that.

140
00:09:55,778 --> 00:09:59,638
So but that's I love containers just just because of the portability.

141
00:09:59,638 --> 00:10:00,038
Yeah.

142
00:10:00,038 --> 00:10:02,028
It just makes it so much easier to work with.

143
00:10:02,028 --> 00:10:06,100
Well, you bringing this up like sort of the complexity with open source projects,
especially ones.

144
00:10:06,100 --> 00:10:12,245
I want to say just projects, but really products or services that are built in or
distributed as a container to be run.

145
00:10:12,386 --> 00:10:21,472
And I think a lot of people who aren't in this space or aren't used to deploying pods on
Kubernetes or running Docker compose scripts, they don't see how much complexity is really

146
00:10:21,472 --> 00:10:26,436
in the management and maintaining of an ecosystem built off of open source services.

147
00:10:26,436 --> 00:10:29,834
I remember even not too long ago thinking about standing up.

148
00:10:29,834 --> 00:10:40,134
Bitwarden during the heyday of all of the password managers being atrocious in some way
and they're like we offer an open source solution, but it's 20 containers and I'm like

149
00:10:40,134 --> 00:10:46,870
Bitwarden 20 containers like first of all, I'm a tech person and I could do it.

150
00:10:46,870 --> 00:10:49,352
I don't want to do that, but I could.

151
00:10:49,352 --> 00:10:51,754
How do you expect someone to just?

152
00:10:51,854 --> 00:10:53,234
do that though.

153
00:10:53,234 --> 00:10:58,554
The majority of the scenarios, like most people who have experience with containers aren't
going to just spin up 20 of them.

154
00:10:58,554 --> 00:11:00,754
That just seems so ridiculous to maintain a product.

155
00:11:00,754 --> 00:11:01,634
Yeah.

156
00:11:02,208 --> 00:11:07,211
You almost wonder if that's on purpose to make the free aspect a little bit.

157
00:11:07,211 --> 00:11:08,013
I don't know that one.

158
00:11:08,013 --> 00:11:09,406
I've never used it.

159
00:11:09,646 --> 00:11:11,166
I know I'm with you there.

160
00:11:11,166 --> 00:11:20,626
Actually, I do believe that a lot of companies that offer an open source solution, it's
just as a gateway or a runway to a funnel into their paid version.

161
00:11:20,626 --> 00:11:24,626
And so, yes, you open you leave the open source and we joke that it's not really open
source.

162
00:11:24,626 --> 00:11:25,865
It's source available.

163
00:11:25,865 --> 00:11:26,806
You can look at it.

164
00:11:26,806 --> 00:11:29,866
But if you actually try to build and run it, you know, good luck.

165
00:11:29,866 --> 00:11:31,286
You're never getting that off the ground.

166
00:11:31,286 --> 00:11:38,220
So like, like I remember a long time ago, I was integrating with GitLab and

167
00:11:38,220 --> 00:11:40,071
I wanted to add a permission.

168
00:11:40,852 --> 00:11:44,816
The permissions there were like full admin access or nothing.

169
00:11:44,816 --> 00:11:52,522
And I'm like, well, it would be nice as a third party writing a plugin to be able to get
access to people's repositories with just like read only.

170
00:11:52,522 --> 00:12:02,010
uh so I actually went through and tried to do software development and they wrote an
engine to check out repositories from GitLab to build it on your machine.

171
00:12:02,010 --> 00:12:05,052
Like that's how complex GitLab open source is.

172
00:12:05,052 --> 00:12:05,883
So I just gave up.

173
00:12:05,883 --> 00:12:07,458
like, I'm gonna write this code.

174
00:12:07,458 --> 00:12:12,380
and I'm gonna push it directly to the repository and have their build system build and
test it for me.

175
00:12:12,380 --> 00:12:20,362
And it was the longest feedback loop I ever had in my entire engineering career, know,
like multiple days even to sort of get this test run.

176
00:12:20,362 --> 00:12:22,963
But it was the only way I could figure out actually how to do the development.

177
00:12:22,963 --> 00:12:27,024
So like I'm totally with you on the container world and the complexity of this.

178
00:12:27,024 --> 00:12:29,965
So you brought up the solution is basically skills here.

179
00:12:29,965 --> 00:12:33,452
And I have to admit my experience with skills is...

180
00:12:33,452 --> 00:12:34,324
very little.

181
00:12:34,324 --> 00:12:41,538
maybe we can jump into that and talk a little bit more like what skills are, how do they
work for LLMs and specifically agents and how they're being utilized.

182
00:12:41,538 --> 00:12:52,009
Yeah, and this is one that, like you said, I won't claim, I think anyone who claims to be
an expert in any of stuff these days is, I would question that statement if they said that

183
00:12:52,009 --> 00:12:53,710
because it's just so new.

184
00:12:53,718 --> 00:12:58,966
You know, it's like, how do you know all the best practices and how do you know what's
your experience with this long term?

185
00:12:58,966 --> 00:13:00,528
And there is no long term.

186
00:13:00,528 --> 00:13:02,019
It's too new.

187
00:13:02,079 --> 00:13:04,910
So I'll pitch it from that regard that

188
00:13:04,910 --> 00:13:08,590
Yeah, I've been using them a lot, but we're talking over the last like three months.

189
00:13:08,590 --> 00:13:16,410
Yeah, so for those that aren't familiar with skills, Anthropic, you know, through cloud
code and things like that, they introduced it kind of first.

190
00:13:16,410 --> 00:13:17,830
That's where it came out of.

191
00:13:17,830 --> 00:13:23,270
And then now, you know, Codex, GitHub, Copilot CLI, GitHub Copilot.

192
00:13:23,270 --> 00:13:27,250
On GitHub, there's a Copilot that runs up in GitHub.

193
00:13:27,250 --> 00:13:31,290
And then there's also an agent you can run and they can all use these skills.

194
00:13:31,290 --> 00:13:32,490
It's called.

195
00:13:32,806 --> 00:13:37,287
And you can think of a skill as a very, so think of it this way.

196
00:13:37,287 --> 00:13:45,409
If you, if your house needed to be painted, ah you need an electrician, you need a
plumber.

197
00:13:45,509 --> 00:13:47,990
To me, that's like, you know, that's a specialist.

198
00:13:47,990 --> 00:13:49,891
So that's like an agent to me, right?

199
00:13:49,891 --> 00:13:53,952
Because you have a plumber who specializes in the plumbing and electrical.

200
00:13:53,952 --> 00:14:00,633
But if you were doing something really, I don't know, unique with, let's just say
electrical as an example.

201
00:14:00,734 --> 00:14:03,014
So I just had to put in uh

202
00:14:03,018 --> 00:14:09,963
in US terms, a 240 volt line for something here at the house literally last week.

203
00:14:09,963 --> 00:14:13,805
And let's say that most electricians I call just didn't do that.

204
00:14:14,005 --> 00:14:15,506
They don't have that skill.

205
00:14:15,506 --> 00:14:18,508
They're an electrician, but they just don't know that angle.

206
00:14:18,508 --> 00:14:27,474
Well, a skill would be like the super specialty information that you could use either on
its own or you could use it with an agent potentially.

207
00:14:27,474 --> 00:14:36,591
So if you know and let's talk about the complexity here of deploying these let's go back
to your 20 container thing I would argue back in the day that was a big deal right because

208
00:14:36,591 --> 00:14:43,846
if you didn't know what you're doing you I think probably did know what you're doing is
just that it was just really complex at least I'm gonna assume you knew what you're doing

209
00:14:46,232 --> 00:14:48,683
Yeah, we're going to get to the benefit of the doubt here.

210
00:14:48,864 --> 00:14:54,988
So these days, like if I want to deploy uh N8n, um it's pretty manageable, actually.

211
00:14:54,988 --> 00:14:55,919
I'm trying to remember.

212
00:14:55,919 --> 00:14:58,531
I think it's like two containers or something like that.

213
00:14:58,531 --> 00:15:09,218
think I did Postgres and that I used the Postgres flex server, it's called, and then a
flexible option and then ACA for uh the container.

214
00:15:09,479 --> 00:15:14,552
Well, if you have a skill who is specialized in knowing about

215
00:15:14,552 --> 00:15:18,924
Here's how you deploy containers to Azure Container apps or whatever your cloud is, right?

216
00:15:18,924 --> 00:15:20,365
Doesn't have to be Azure.

217
00:15:20,365 --> 00:15:29,070
That skill would have all the details needed that would be required to know, like if
you're not a container expert, it wouldn't matter because that skill has those details.

218
00:15:29,070 --> 00:15:39,096
So now when you hook that skill up and just think of it as like the ultimate knowledge
source on whatever the topic is that you're specializing in.

219
00:15:39,096 --> 00:15:41,117
So we'll say ACA in this case.

220
00:15:41,117 --> 00:15:43,150
Now you plug that into your coding agent.

221
00:15:43,150 --> 00:15:54,439
uh or your agent in the cloud doesn't really any agent really but like GitHub Copilot CLI
is is uh one that is new pretty new and I'm just gonna tell you it like I'm a huge cloud

222
00:15:54,439 --> 00:16:03,026
code fan I'm also now a huge GitHub Copilot CLI fan It's freaking amazing people should
it's the best deal in town.

223
00:16:03,026 --> 00:16:10,432
I'm telling you I would not say that even though I know I work for Microsoft Warren I
would not say that if I didn't believe it because I'm the type of person I'm pretty

224
00:16:10,432 --> 00:16:13,034
transparent, but anyway uh

225
00:16:13,034 --> 00:16:18,878
You now plug that skill in and now I can say, hey, I need to deploy X to Y.

226
00:16:18,878 --> 00:16:25,002
And as long as it knows about that skill, it kicks in and behind the scenes it's going,
okay, I know what to do.

227
00:16:25,002 --> 00:16:32,344
And again, it's like the matrix, I guess, where you jack in and boom, you can fly the
helicopter or whatever it is.

228
00:16:32,344 --> 00:16:34,466
What's the context of what's in the scale?

229
00:16:34,466 --> 00:16:41,771
we talking about basically a written document that explains all the critical aspects of
what the service does and how it's deployed and whatnot?

230
00:16:41,771 --> 00:16:47,328
So canonically what the readme was supposed to do for open source repositories in the
past.

231
00:16:47,328 --> 00:16:57,499
actually a pretty good analogy is yeah there's a skill.md it's all marked down you can
have other assets artifacts resources whatever you want to call it that are associated

232
00:16:57,499 --> 00:17:07,441
with that so you could even have like scripts for example I don't know there could be like
SSH calls that are made and there's a script that makes the call and you know that could

233
00:17:07,441 --> 00:17:09,134
be part of it technically

234
00:17:09,134 --> 00:17:12,694
But at a minimum, yeah, it's the skill.md file.

235
00:17:12,694 --> 00:17:18,034
And it's kind of like you said, it's like what probably should have been somewhere in a
readme or whatever.

236
00:17:18,034 --> 00:17:19,914
But now it's reusable.

237
00:17:19,914 --> 00:17:25,954
And so now if I give you that skill within literally a minute or two, you could also do
the same deployment.

238
00:17:25,954 --> 00:17:34,020
And like we really need this because where historically there has been documentation on
how to do the deployment correctly for open source projects.

239
00:17:34,020 --> 00:17:39,925
I feel like over time that started to degrade even to some point of like, what exact
distribution are you on?

240
00:17:39,925 --> 00:17:49,642
Which version of Linux or which OS are you on and what dependencies do you have and are
using containers or Kubernetes and which version of Docker swarm are you using or using

241
00:17:49,642 --> 00:17:50,693
Nomad?

242
00:17:50,753 --> 00:17:52,086
All that is just.

243
00:17:52,086 --> 00:17:56,200
such a huge burden from a maintainer standpoint to answer all of those questions.

244
00:17:56,200 --> 00:18:05,521
And I feel like some part of that just goes away because you're assuming that it's now a
dependency or responsibility of the installer, but they don't have that expertise either.

245
00:18:05,521 --> 00:18:14,424
So it's now hopefully contained in the agent sphere, but there still has to be the
instructions that explain the.

246
00:18:14,424 --> 00:18:22,491
critical interaction points or maybe where the main function is or the number of
containers or URLs, et cetera, that aren't going to be easily exposed or understood.

247
00:18:22,491 --> 00:18:25,544
An LLM wouldn't have picked up and trained on.

248
00:18:25,564 --> 00:18:27,982
And that information doesn't exist anywhere else.

249
00:18:27,982 --> 00:18:29,402
Yeah, exactly.

250
00:18:29,502 --> 00:18:38,702
going back to your read me kind of comment earlier, there's a lot of stuff that either A,
we just haven't had time to update because it's too detailed or whatever.

251
00:18:38,702 --> 00:18:40,982
Because you have nowadays kind of the standards.

252
00:18:40,982 --> 00:18:44,702
You have your agents dot markdown file, which can be at the root of the repo, of course.

253
00:18:44,702 --> 00:18:50,042
But it's I'd call it more of a generalist about the project that your agents can learn
about.

254
00:18:50,042 --> 00:18:56,502
But then when it gets to really specialized, that's where the skills kind of come into
play.

255
00:18:56,502 --> 00:18:57,654
And it's just

256
00:18:57,654 --> 00:18:59,225
I've been pleasantly surprised.

257
00:18:59,225 --> 00:19:00,977
So I'll give you an example on this one.

258
00:19:00,977 --> 00:19:07,582
I had originally I had, were doing, we're working on three right now, N8N, SuperSET and
Grafana.

259
00:19:07,582 --> 00:19:10,565
Some of those are more complex than others, you know, to deploy.

260
00:19:10,565 --> 00:19:15,619
And originally I had one agent, it was like OSS Deployer or something was my agent.

261
00:19:15,619 --> 00:19:24,336
And then I had like seven skills because I had one for, you know, like Postgres and one
for ACA and one for security and one for whatever.

262
00:19:24,336 --> 00:19:26,606
And then, so I had a call.

263
00:19:26,606 --> 00:19:30,768
probably about three weeks ago now with a buddy mine that works at Microsoft, name's
Shane.

264
00:19:30,768 --> 00:19:32,569
And he's like, why are you doing it that way?

265
00:19:32,569 --> 00:19:33,870
We already have these skills.

266
00:19:33,870 --> 00:19:37,452
And I'm like, because I didn't know.

267
00:19:38,313 --> 00:19:40,654
And so now I've got it down to, think, three.

268
00:19:40,654 --> 00:19:48,308
I have one that just knows about N8N, one that knows about SuperSet, and one that knows
about Grafana, and then all the other skills for going to Azure.

269
00:19:48,308 --> 00:19:50,279
And again, this would be the same for other clouds.

270
00:19:50,279 --> 00:19:51,480
They're already pre-built.

271
00:19:51,480 --> 00:19:54,141
I just have to know how to plug them in, which was easy.

272
00:19:54,141 --> 00:19:55,756
And boom, I'm ready to go.

273
00:19:55,756 --> 00:19:59,431
So I could see uh your complex scenario earlier.

274
00:19:59,431 --> 00:20:10,445
I literally can see that now being down to almost a single prompt with a set of skills
under it and boom, if they wanted to make it easy, which as we talked about, not sure all

275
00:20:10,445 --> 00:20:11,826
companies want to do that.

276
00:20:12,176 --> 00:20:16,694
I think the thing that really connected for me is that the information...

277
00:20:16,694 --> 00:20:24,169
and how to do that may already be contained within the training set that the LLM has
utilized for how to do a deployment in any of those technologies.

278
00:20:24,169 --> 00:20:34,375
But realistically, one challenge is, are you able to construct the correct prompt to
expose that information into the context so that the LLM, the agent, whatever utilizing

279
00:20:34,375 --> 00:20:35,906
can actually do the thing correctly?

280
00:20:35,906 --> 00:20:42,678
And can you get around any potential, uh say, poison or uh injection that was?

281
00:20:42,678 --> 00:20:51,575
set up in the training data to have you not necessarily just install the open source
technology, but also leak your API keys or credentials all over the internet.

282
00:20:51,575 --> 00:20:56,028
This gives you a canonical best strategy for dealing with it.

283
00:20:56,148 --> 00:21:03,473
can review the skills, see what's in there, and rather than having to learn that
information yourself, you are in a way teaching the LLM to...

284
00:21:03,473 --> 00:21:04,404
uh

285
00:21:04,404 --> 00:21:09,816
Utilize it specifically and so you don't have to figure out what the best system prompt is
to actually do that or what?

286
00:21:09,816 --> 00:21:15,217
Magic keywords have to be in the the user prompt in order to have the right thing happen
Right.

287
00:21:15,217 --> 00:21:22,869
I mean I think the fear that I have and I'm sure this is already happening Like where is
the best, know trusted canonical list of skills out there?

288
00:21:22,869 --> 00:21:32,492
You know, I look at the Linux distro package system or issues and just the ones for every
single source code you know, I think you know, everyone hates NPM but

289
00:21:32,492 --> 00:21:40,940
Honestly, it's still the best one I've ever seen for a package manager goes and there used
to be a fight, know, NPM or new get one was owned by Microsoft and the other one not but I

290
00:21:40,940 --> 00:21:43,052
guess now technically both of them are owned by Microsoft.

291
00:21:43,052 --> 00:21:43,893
Yeah.

292
00:21:43,893 --> 00:21:53,562
So uh it's like it's like there's a list of package managers and NPM is what is the worst
except for all the other ones.

293
00:21:53,562 --> 00:21:53,930
Yeah.

294
00:21:53,930 --> 00:21:56,033
That's pretty good.

295
00:21:56,033 --> 00:21:58,896
I think a lot of people feel that way.

296
00:22:00,086 --> 00:22:01,667
I do think it's getting better over time.

297
00:22:01,667 --> 00:22:09,812
But one thing that I've learned to trust when it comes to canonical package managers is
their ability and desire to weed out malicious packages.

298
00:22:09,812 --> 00:22:20,728
And I feel like this is one thing that PMPM has gotten right with this idea of like dwell
time or wait time on new package publishing before utilizing it in case there is a

299
00:22:20,728 --> 00:22:21,859
vulnerability.

300
00:22:21,859 --> 00:22:28,442
But we really leave it up to the package repositories to take care of this for us and find
those security.

301
00:22:28,486 --> 00:22:36,608
vulnerabilities and potentially remove those packages, whether or not they're doing it
directly or through some sort of consensus based algorithm or user reporting, etc.

302
00:22:36,608 --> 00:22:47,911
But I see the same problem going to spill into skills and in a way where uh there's a lot
more people who are maybe less have less expertise in the technical understanding of the

303
00:22:47,911 --> 00:22:50,452
complexities and security issues that around them.

304
00:22:50,512 --> 00:22:55,413
What is like are there are we already starting to see skill like trusted skill repository
set up?

305
00:22:55,413 --> 00:22:57,420
Does Azure have one of them?

306
00:22:57,420 --> 00:22:58,701
Are there ones you point to?

307
00:22:58,701 --> 00:23:02,765
you just going the old Microsoft strategy of like a Windows strategy, really?

308
00:23:02,765 --> 00:23:07,338
Go to the internet, download some EXE and run it on your computer completely untrusted.

309
00:23:07,438 --> 00:23:17,197
No, I mean this is you you hit you know, you hit the nail on the head there This is
absolutely an area where people need to be concerned about just grabbing an off-the-shelf

310
00:23:17,197 --> 00:23:28,216
You know whether it's a skill or an agent because these are just markdown files, of course
But there's a lot of prompt injection things, you know, you can do and and more and So

311
00:23:28,216 --> 00:23:30,258
take to go to your other things out there.

312
00:23:30,258 --> 00:23:35,352
Yes So like one we have for github copilot in general is called awesome

313
00:23:35,352 --> 00:23:47,770
Copilot it's on github and it is one that's vetted very heavily and and It's one where
it's just a repo and people submit pr's and they're reviewed all that so not to the point

314
00:23:47,770 --> 00:23:57,375
where it's like Like npm or you know pip installs for python or whatever where you just
have millions of packages that that's a whole nother scale There's skills.sh.

315
00:23:57,375 --> 00:23:58,302
That's another site.

316
00:23:58,302 --> 00:24:00,757
We're gonna go just find tons of skills.

317
00:24:00,757 --> 00:24:01,934
I literally

318
00:24:01,934 --> 00:24:07,096
ah Run any skill I don't know about though, even if I got it from a trusted site.

319
00:24:07,096 --> 00:24:12,197
I have my, you know, like Copilot CLI or Cloud Code or whatever folks use.

320
00:24:12,197 --> 00:24:15,699
I have them do a security review on it before I use it.

321
00:24:15,699 --> 00:24:17,928
I'll say literally go look at this markdown.

322
00:24:17,928 --> 00:24:19,520
You know, I don't want you to run it.

323
00:24:19,520 --> 00:24:20,637
I just want you to scan it.

324
00:24:20,637 --> 00:24:22,301
Kind of like I do the same thing with code.

325
00:24:22,301 --> 00:24:23,041
Code repos.

326
00:24:23,041 --> 00:24:24,862
I'll have it before I try it.

327
00:24:24,862 --> 00:24:26,342
Go scan this repo.

328
00:24:26,398 --> 00:24:37,206
I think validating untrusted code is one of the unsolved problems in computer science,
though, to the degree of we still get um even solutions.

329
00:24:37,506 --> 00:24:47,054
The most well-known one out there, at least from my standpoint, which is just a small iota
of experience, is AWS's Firecracker, which is what they're using to power Lambda.

330
00:24:47,054 --> 00:24:56,892
uh And it's basically this thing where, well, you you look at the cloud providers, they
clearly have some sort of security walls around one customer's source code, not executing

331
00:24:56,892 --> 00:25:01,265
on someone else's memory or storage space.

332
00:25:01,266 --> 00:25:06,562
And there are some open source projects that do this, but like, how much would you trust
one of those out there?

333
00:25:06,562 --> 00:25:09,885
for untrusted code to run on in a trusted environment.

334
00:25:09,885 --> 00:25:11,566
I'm like, I wouldn't do that.

335
00:25:11,566 --> 00:25:22,724
And that's why I want to ask about the validating some of these skills because I wouldn't
trust sending that to Claude code and hoping that it doesn't do the wrong thing

336
00:25:22,724 --> 00:25:23,205
accidentally.

337
00:25:23,205 --> 00:25:29,980
Because even if it says, don't run this skill, no matter what you do, I just want to
evaluate it specifically.

338
00:25:29,980 --> 00:25:32,970
And I think there are still ways to get around.

339
00:25:32,970 --> 00:25:41,867
just using like the English language or whatever language you're using to attempt to
suggest to the LLM not to execute a potentially dangerous ah instruction.

340
00:25:41,867 --> 00:25:42,806
It's a real thing.

341
00:25:42,806 --> 00:25:48,308
That's the prompt injection, for example, is just one of many techniques that people can
use.

342
00:25:48,308 --> 00:25:51,509
And there's actually cases where this has happened.

343
00:25:51,509 --> 00:25:53,430
know, open clause very popular, right?

344
00:25:53,430 --> 00:25:54,450
Right now.

345
00:25:54,570 --> 00:25:57,571
And I don't know if you've heard it and I actually use it.

346
00:25:57,571 --> 00:26:00,292
I have a very locked down VM I use it on.

347
00:26:00,292 --> 00:26:03,433
Nobody can get to it except for me through my tail scale.

348
00:26:03,573 --> 00:26:05,774
But uh that's another topic.

349
00:26:05,890 --> 00:26:12,934
But I've heard all these stories now of people who just installed started installing all
kinds of skills or agents or whatever it was.

350
00:26:12,934 --> 00:26:17,017
next thing they know, it was doing really uh malicious type things.

351
00:26:17,017 --> 00:26:17,417
Yeah.

352
00:26:17,417 --> 00:26:20,118
Because of what you just said.

353
00:26:20,399 --> 00:26:21,858
First off, didn't get it.

354
00:26:21,858 --> 00:26:24,941
They took it on the Internet.

355
00:26:24,941 --> 00:26:25,902
What could go wrong?

356
00:26:25,902 --> 00:26:26,488
You know.

357
00:26:26,488 --> 00:26:32,990
The sad part is, especially with things like OpenClaw, uh the agent loop is doing it
itself.

358
00:26:32,990 --> 00:26:37,841
You're not even telling it to use this particular skill that you then didn't vet.

359
00:26:37,841 --> 00:26:39,722
You were just telling it, want to do this thing.

360
00:26:39,722 --> 00:26:41,852
And it says, OK, I tried to do it.

361
00:26:41,852 --> 00:26:42,462
I couldn't do it.

362
00:26:42,462 --> 00:26:45,563
I found this random skill on the internet that tells me I can do it.

363
00:26:45,563 --> 00:26:47,744
I went and installed the skill, and then I executed it.

364
00:26:47,744 --> 00:26:55,306
It's like, oops, also, did I tell you that all your API keys for all your cloud providers,
I accidentally published those to a public location.

365
00:26:55,918 --> 00:27:02,798
And so like now there are like hundreds and probably thousands of forks of open claw out
there that promise to be secure.

366
00:27:02,798 --> 00:27:09,538
And I think this is my biggest concern with any of those is that the way in which they
promise security is through the front door.

367
00:27:09,538 --> 00:27:12,338
It's like it used to be like, oh, the gateway is insecure.

368
00:27:12,338 --> 00:27:17,758
Like anyone can fake sending you a telegram message or a Slack message and all of a sudden
there's a prompt injection attack.

369
00:27:17,758 --> 00:27:25,058
I'm like, yes, those are all the standard problems with software that we've had for
almost, it's still over 50 years I'm going to say.

370
00:27:25,058 --> 00:27:26,758
realistically through the API.

371
00:27:26,758 --> 00:27:29,559
Those aren't the attacks I'm actually concerned with with the LLMs.

372
00:27:29,559 --> 00:27:35,871
It's the prompt injection from the data that it's getting itself, that it's choosing to go
out to the internet and grab and pull down.

373
00:27:35,871 --> 00:27:46,484
And so it's just so ridiculous to me where we're already at this point where people who
don't have the technical capabilities of securing their stuff are trusting what we're

374
00:27:46,484 --> 00:27:50,275
getting to be valid and not have any sort of vulnerabilities in it.

375
00:27:50,275 --> 00:27:54,720
So much so that companies like CloudFlare and AWS released open claw

376
00:27:54,720 --> 00:28:02,407
runners basically for you to run your open call and I'm like yeah sure you've closed the
front door you've secured that part but they're still vulnerable to prompt injection

377
00:28:02,407 --> 00:28:06,354
attacks I feel like it's a bit irresponsible to even go down that path.

378
00:28:06,475 --> 00:28:14,803
And you know the funny the front door is a great analogy because you know there's a back
door and there's windows and there's the attic and there's you know There's all kinds of

379
00:28:14,803 --> 00:28:20,519
ways you can get in and anyone who's been around for a long time I liked your analogy a
little while ago.

380
00:28:20,519 --> 00:28:24,896
You said you know I'm just gonna download this executable off the internet and just

381
00:28:24,896 --> 00:28:26,807
run it because I remember those days.

382
00:28:26,807 --> 00:28:28,787
I'm trying to call it two cows.

383
00:28:28,787 --> 00:28:30,858
I think it was called way back in the day.

384
00:28:30,858 --> 00:28:36,749
It was two cows uh and it was this website you could go to to just download like all kinds
of cool apps.

385
00:28:36,749 --> 00:28:40,990
And I kind of feel like that's where we are with some of some of the AI stuff.

386
00:28:40,990 --> 00:28:51,313
And we can circle back to the Lego analogy with this as well, because honestly, I think
that's where if you're doing it 100 percent, you if you're an enterprise and you're like

387
00:28:51,313 --> 00:28:54,110
you're saying you're trusting just the front door.

388
00:28:54,110 --> 00:28:56,871
there's so much more to the story than just the front door.

389
00:28:56,871 --> 00:29:07,227
I think that's where having the cloud, you know, again, going back to the Lego blocks
really matters because in addition to all that, you're going to get back a response from

390
00:29:07,227 --> 00:29:08,538
the LLM at some point.

391
00:29:08,538 --> 00:29:13,501
What happens if it has something that's malicious in it that was put in by the skills?

392
00:29:13,501 --> 00:29:22,105
Yeah, like the skill itself did nothing wrong, but it triggered something that's going to
run that you would not look for normally when you get the response back.

393
00:29:22,105 --> 00:29:23,694
What you know, whatever you're doing with that.

394
00:29:23,694 --> 00:29:29,794
And that's where, you know, and we could get in like that's responsible AI and the
security aspect and all those things.

395
00:29:29,794 --> 00:29:36,174
Because like you said, it's you have to have this like 360 view of it, not just the front
door.

396
00:29:36,174 --> 00:29:45,174
to me, excuse me, to me, that's where the, you know, the I think we actually do a pretty
good job of this, to be honest, where I think the cloud providers are kind of essential

397
00:29:45,174 --> 00:29:48,834
these days, because to do this on your own, it's just not possible.

398
00:29:48,834 --> 00:29:52,334
There's there's no way anyone has that expertise to do it all on your own.

399
00:29:52,334 --> 00:29:53,400
I think we're going to have a

400
00:29:53,400 --> 00:30:02,001
proliferation, like there's a huge opportunity for security companies focused on the AI
angle, not just regular security.

401
00:30:02,001 --> 00:30:07,527
All this new unknown stuff is going to be just a massive opportunity, I think.

402
00:30:07,527 --> 00:30:13,524
So we'll see what happens there and if I'm proven right or wrong, but I'm going to predict
that's probably going to be a thing for sure.

403
00:30:13,902 --> 00:30:15,643
I want to take the pessimistic side.

404
00:30:15,643 --> 00:30:25,505
think security has always been this cost center and companies are trying to use LLMs to
write code and the ones that are doing it are doing it because they're trying to save

405
00:30:25,505 --> 00:30:29,406
money and so they're less likely to then pay it out.

406
00:30:29,406 --> 00:30:39,489
So I think maybe the major players in the game for ones that are generating code will just
have to promise to make more secure stuff.

407
00:30:39,489 --> 00:30:43,490
But realistically, I think one of the biggest challenges that has come about

408
00:30:43,670 --> 00:30:52,445
if we call this a revolution, is that we've pushed the liability, financial and legal,
back onto the consumer in a lot of ways.

409
00:30:52,445 --> 00:30:59,858
Whereas it used to be like, if you're running a service or you're a product and you're
buying a service from a third party, you hold them liable for it.

410
00:30:59,858 --> 00:31:07,683
But now, instead of doing that, they're offering you an agent or an LLM that generates
code and you're taking the liability to make sure that that code works.

411
00:31:07,683 --> 00:31:12,844
And so yeah, you can trust them that that code being generated is more secure in some way.

412
00:31:12,844 --> 00:31:19,894
But I think the reality is that it may not be, and you're gonna be stuck holding the, I
don't know what the analogy is here, holding the bag.

413
00:31:19,894 --> 00:31:21,698
bag or whatever.

414
00:31:21,820 --> 00:31:23,896
mean, I like we could say.

415
00:31:23,896 --> 00:31:24,246
I see.

416
00:31:24,246 --> 00:31:26,427
know, I, my mind went to like bag of groceries.

417
00:31:26,427 --> 00:31:28,328
Like I want to hold the bag, right?

418
00:31:28,328 --> 00:31:30,299
That, you know, I get, that's where the rewards are.

419
00:31:30,299 --> 00:31:32,590
Uh, I actually don't know what, where this comes from.

420
00:31:32,590 --> 00:31:34,051
And maybe that's on me.

421
00:31:34,051 --> 00:31:38,813
Uh, I don't, I don't know what the solution is honestly there, but it doesn't, it doesn't
feel great.

422
00:31:38,813 --> 00:31:47,086
And I don't want to spoil my, my pick for this episode, but I do feel like one of the
problems here is that the ability to do software development is getting more and more

423
00:31:47,086 --> 00:31:47,777
complicated.

424
00:31:47,777 --> 00:31:51,158
Not, not less because you have to know, not just

425
00:31:51,180 --> 00:31:54,813
all everything about software engineering to build the right solution, because you have to
review it.

426
00:31:54,813 --> 00:31:56,424
You also have to know what the agents are doing.

427
00:31:56,424 --> 00:32:00,317
You have to understand that skills can be malicious, for instance, and evaluating those.

428
00:32:00,317 --> 00:32:01,958
Where do you even get good skills?

429
00:32:01,958 --> 00:32:06,121
Understanding the different attack vectors on OpenClaw or whatever agent you're running.

430
00:32:06,121 --> 00:32:13,606
So it's not just like you could just get away with understanding you are, like, I have a
lot of security knowledge and I have very little security knowledge around the AI space.

431
00:32:13,606 --> 00:32:19,362
That means that I don't have the same capability to protect systems as I would if I didn't
introduce an LLM.

432
00:32:19,362 --> 00:32:22,564
That means I think we're getting further away from secure systems.

433
00:32:22,564 --> 00:32:24,335
In some ways, maybe that's fine.

434
00:32:24,335 --> 00:32:32,961
You know, we're just, we're just speed running, you know, every software being insecure
and all those, all those uh movies that came out, especially in like the eighties where

435
00:32:32,961 --> 00:32:36,874
someone's like, they're just mashing on a keyboard and suddenly, I'm in now.

436
00:32:36,874 --> 00:32:38,535
It feels so much more accurate.

437
00:32:38,535 --> 00:32:38,795
that.

438
00:32:38,795 --> 00:32:42,988
I'm just, I'm just typing like.

439
00:32:42,988 --> 00:32:47,182
you know, some sort of fork bombs, like, I'm going to send a fork bomb to this process and
I'll be in.

440
00:32:47,182 --> 00:32:51,317
It's like, well, now actually that's true because you tell the LOM, hey, run this code.

441
00:32:51,317 --> 00:32:53,998
then, you know, it does, does let you in.

442
00:32:53,998 --> 00:32:57,358
It just took like 20 years, 30 years to catch up.

443
00:32:57,358 --> 00:33:07,938
But yeah, like the movie, I don't know if it's really old War Games, you know, like
they're typing all this stuff and it just flows and you're like, that's not how it works.

444
00:33:07,938 --> 00:33:14,218
And now I just released a video last night, Warren, no joke, and on this copilot CLI.

445
00:33:14,218 --> 00:33:17,098
And there's so much there as the LLMS process.

446
00:33:17,098 --> 00:33:19,718
I'm like, just people are going to watch every little line.

447
00:33:19,718 --> 00:33:20,338
Right.

448
00:33:20,338 --> 00:33:23,718
So I'm chopping, chopping, chopping, chopping, chopping to speed it up.

449
00:33:23,820 --> 00:33:25,451
because it's exactly what you just said.

450
00:33:25,451 --> 00:33:32,117
It's just like boom, boom, boom, boom, all this data, you know, I wanted to circle back
though to what you said on.

451
00:33:32,117 --> 00:33:34,659
I do feel like, don't know if you saw there was a blog post.

452
00:33:34,659 --> 00:33:36,080
I'm trying to remember the title.

453
00:33:36,080 --> 00:33:40,143
It was something along the lines of AI is making me tired.

454
00:33:40,143 --> 00:33:53,558
That's not the title, but how the promise was uh that it would make me more productive and
which I honestly, I think once you understand how to leverage features.

455
00:33:53,558 --> 00:33:55,459
and know what's good and what's bad.

456
00:33:55,818 --> 00:33:57,080
I actually think that's a true statement.

457
00:33:57,080 --> 00:34:06,434
Like I'm way more productive in general because I can get started faster even if I end up
having to do, you know, even if I have to personally push it over the finish line, which I

458
00:34:06,434 --> 00:34:11,086
typically I'm still in that like, hey, the human matters in this loop.

459
00:34:11,086 --> 00:34:11,886
Right.

460
00:34:11,886 --> 00:34:12,846
That's good.

461
00:34:14,386 --> 00:34:15,286
Thumbs up.

462
00:34:15,286 --> 00:34:22,266
I think that's a very important thing that I think we see a lot of the leaders saying
throughout the industry that the human still matters here.

463
00:34:22,266 --> 00:34:30,146
I feel like it's sort of this meme curve where on one side, it's like those who don't know
anything say the human still matters.

464
00:34:30,146 --> 00:34:32,406
Then there's the middle where it's like, the human doesn't matter.

465
00:34:32,406 --> 00:34:36,306
And then on the far side, they realize, actually the human still does matter.

466
00:34:36,306 --> 00:34:40,806
We actually have a whole episode on this podcast where we were talking about productivity.

467
00:34:41,850 --> 00:34:44,292
did that a few, I think it was only a couple months ago.

468
00:34:44,292 --> 00:34:46,463
We went into that really deep.

469
00:34:46,463 --> 00:34:52,477
One of the things I wanna bring up is the agent loop though, because I think you may have
some unique insights here.

470
00:34:52,477 --> 00:35:01,734
I find that we're right now in this time where what was old is new again, and I'm sure
that XKCD comic already came back of like, what are you waiting for?

471
00:35:01,734 --> 00:35:08,014
My code is compiling, which used to be MS build taking forever, uh or Java on Eclipse.

472
00:35:08,014 --> 00:35:12,196
Uh, 1.5 or six, and now we're back to like, we're waiting.

473
00:35:12,196 --> 00:35:20,141
And one of the advice that I see in the industry going around a lot is, well, aren't you
running like seven agents and like a hundred projects all at the same time?

474
00:35:20,141 --> 00:35:28,426
And I'm like, no, actually I can't handle that because the change of our cost is too high
retorts, guess would be my question.

475
00:35:29,187 --> 00:35:35,002
Well, going back to the running multiple agents thing, um this, it frightens me.

476
00:35:35,002 --> 00:35:36,566
And I'm going tell you why.

477
00:35:36,566 --> 00:35:50,475
ah I do run multiple agents, but only enough, they're only creating enough that I can
actually keep up with because it makes, you could argue, I suppose, that I'm going to have

478
00:35:50,475 --> 00:35:56,352
all these agents run in the background and they're going to build feature XYZ, know, ABC,
whatever.

479
00:35:56,352 --> 00:35:58,483
Who's going to go through and validate all that?

480
00:35:58,483 --> 00:36:00,675
ah I'm playing with a project right now.

481
00:36:00,675 --> 00:36:01,796
It's a personal project.

482
00:36:01,796 --> 00:36:12,853
I won't go into details, but I would say every four hours I check in with what I'm doing
because, you know, the loop will wrap up and then I got to evaluate it, do my review.

483
00:36:12,853 --> 00:36:15,255
I'll run it and I'll go, what the heck?

484
00:36:15,255 --> 00:36:16,856
How did you miss this?

485
00:36:16,856 --> 00:36:20,288
Like this should have been so obvious and it totally missed it.

486
00:36:20,288 --> 00:36:20,879
Right.

487
00:36:20,879 --> 00:36:26,462
And I think that's where the we'll go back to the human concept I made earlier, the human
in the loop.

488
00:36:26,526 --> 00:36:30,768
like knowing what you're doing, I think now matters more than ever.

489
00:36:30,768 --> 00:36:39,992
ah Because if you don't, what kind of security issues are going to crop up when you're
just like, yeah, I totally trust everything that's being spit out here.

490
00:36:40,692 --> 00:36:44,204
I'm just going to go with it as is like not even going to do a review on it.

491
00:36:44,204 --> 00:36:44,894
It's great.

492
00:36:44,894 --> 00:36:46,334
It's fantastic.

493
00:36:46,435 --> 00:36:54,542
And I'll be honest when, you know, when Opus four to six came out, GPT five to four just
recently came out.

494
00:36:54,542 --> 00:36:57,102
They're like a whole level above what we had.

495
00:36:57,102 --> 00:37:08,042
I just saw an internal comment earlier where somebody was using, I think it was, yeah, was
Opus 4.6 and it literally made a comment, something along the lines of, I'll summarize it

496
00:37:08,042 --> 00:37:10,522
in my terms, like who wrote this crap?

497
00:37:10,982 --> 00:37:15,682
And the who wrote this crap was Opus 4.5.

498
00:37:16,062 --> 00:37:20,002
And that's how far we've come just in that one iteration.

499
00:37:20,262 --> 00:37:24,270
So getting back to the agents and the loop and running all these agents,

500
00:37:24,270 --> 00:37:27,050
Here's kind of a couple scenarios I've been thinking through.

501
00:37:27,050 --> 00:37:32,090
I think you're going to have leadership in companies who realize that yes, we're way more
productive.

502
00:37:32,090 --> 00:37:37,910
We can ship way more, which means we still need these people to evaluate that.

503
00:37:37,910 --> 00:37:43,884
Like they have to constantly be going through and doing some good reviews and all that on
it, which takes time.

504
00:37:43,884 --> 00:37:51,920
And then I think you're going to have the leaders who are like, no, AI is so good that I
think we could just cut back our workforce huge and it'll be fine.

505
00:37:51,920 --> 00:37:54,722
Any predictions Warren on how that's going to go?

506
00:37:54,966 --> 00:38:01,708
Unfortunately, I'm not an optimist here that those companies will still last probably
about 10 years before crumbling.

507
00:38:01,708 --> 00:38:04,885
I think they'll last for a while until they're bit by something huge.

508
00:38:04,885 --> 00:38:07,369
No, some are going to get lucky.

509
00:38:07,410 --> 00:38:07,811
Sure.

510
00:38:07,811 --> 00:38:08,813
And it'll be fine.

511
00:38:08,813 --> 00:38:13,250
And then there's going to be those who just went, my gosh, I had no idea.

512
00:38:13,250 --> 00:38:16,611
Yeah, I mean, I'm, I'm, like your optimism.

513
00:38:16,611 --> 00:38:25,684
I just, I, we already see today that from the security domain, that lots of companies are
just winging it when it comes to building stuff, especially with the loop on startups and

514
00:38:25,684 --> 00:38:35,236
the money coming from the different flavors of venture capitalists to go and deliver
something that is as half baked as possible and basically scam users out of money until

515
00:38:35,236 --> 00:38:38,937
they get billions of dollars or in this case, guess trillions.

516
00:38:38,938 --> 00:38:40,800
And at no point is

517
00:38:40,800 --> 00:38:46,434
a requirement that their solution actually be secure, let alone even be good for humanity.

518
00:38:46,554 --> 00:38:55,390
don't think magically there's something that's going to change now that the
hypothetically, not that I agree, but if I were to agree that the cost of doing software

519
00:38:55,390 --> 00:39:04,747
development is less now, that is going to all of a sudden make them take that extra money
and put it to actually reviewing what they have and reducing those potential security

520
00:39:04,747 --> 00:39:05,327
vulnerabilities.

521
00:39:05,327 --> 00:39:06,488
There is something to be said here.

522
00:39:06,488 --> 00:39:09,294
And that's if it's easier to do the software development.

523
00:39:09,294 --> 00:39:20,074
If turning out a product is easier, is possible that if as a humanity all the source code
that we've ever generated is more secure on average than what people were turning up

524
00:39:20,074 --> 00:39:23,934
before, then the security does actually increase a little bit.

525
00:39:23,934 --> 00:39:29,894
And I know it's something that bothers me because I see any issue as a hugely problematic
one.

526
00:39:29,894 --> 00:39:33,418
We look at OWASP top 10, top 10 for...

527
00:39:33,418 --> 00:39:40,350
APIs, top 10 for authorization and authentication, top 10 for AI based stuff, and they're
really bad things.

528
00:39:40,350 --> 00:39:48,402
maybe the solutions we've had all along were already hugely problematic with like S3
buckets being completely exposed to the internet for most companies out there.

529
00:39:48,402 --> 00:39:55,925
ah from a Microsoft standpoint, publishing your API keys publicly in your .env file,
because people do that.

530
00:39:55,925 --> 00:40:00,686
And then having some sort of issue when it comes to payment time for your cloud provider.

531
00:40:01,164 --> 00:40:08,210
I think there's a couple of different angles there and I keep seeing, well, this is really
bad because, you know, insert all these other things.

532
00:40:08,210 --> 00:40:12,994
It's hard for me to really effectively evaluate where is the 50 % mark?

533
00:40:12,994 --> 00:40:15,206
You know, where is the average?

534
00:40:15,206 --> 00:40:16,527
Is it increasing?

535
00:40:16,527 --> 00:40:19,500
Is it going up over time or is it coming down?

536
00:40:19,500 --> 00:40:27,056
And I think from my standpoint, my challenge is that when I'm looking at the code, I feel
like I'm paying attention to it less, right?

537
00:40:27,056 --> 00:40:29,474
And in order to actually review it, I need to pay attention.

538
00:40:29,474 --> 00:40:31,676
to it more than I was doing before.

539
00:40:31,676 --> 00:40:38,747
And while I feel like I'm in the right position for that, I think we're going to start to
see a lot of engineers out there who are being pushed even more by their organizations,

540
00:40:38,747 --> 00:40:44,164
you these bad leaders that you're talking about, that will not review what they have.

541
00:40:44,164 --> 00:40:51,739
Because I think we have this sort of saying in security a lot, if the solution to the
security problem is training, that's not a real security answer.

542
00:40:51,739 --> 00:40:55,291
And I feel like when it comes to the LM generated code, it's very similar.

543
00:40:55,291 --> 00:40:56,632
yeah, for sure.

544
00:40:57,233 --> 00:40:59,274
And people still make mistakes, right?

545
00:40:59,274 --> 00:41:01,844
even if you review it, you could be missing something.

546
00:41:01,844 --> 00:41:05,144
And so I worry that that's just gonna keep happening more and more.

547
00:41:05,144 --> 00:41:10,978
Well, and to kind of put a positive spin on what we're talking about, here's one thing I
will say.

548
00:41:10,978 --> 00:41:22,996
When it comes to reviewing code, so one thing that I use a lot, and all the CLI-based AI
coding assistants have something like this, but is like in copilot CLI, I'll do slash

549
00:41:22,996 --> 00:41:26,128
review and then give it my prompt and what to do.

550
00:41:26,128 --> 00:41:34,134
And I am almost always pleasantly surprised how it will catch things that I honestly
wouldn't have thought of.

551
00:41:34,274 --> 00:41:41,939
And uh then security things I've also had come up where, do a review, but I want you to
look for X, Y, Z and whatever.

552
00:41:41,939 --> 00:41:47,342
And I will say, because you have different skill sets, of course, with developers.

553
00:41:47,342 --> 00:41:54,046
And so you've probably worked with the folks who are just expert in this one area, but
they don't know this other area at all.

554
00:41:54,046 --> 00:42:00,152
And that's where I think AI really can help out is it kind of, I don't want to say levels
the playing field.

555
00:42:00,152 --> 00:42:00,572
Totally.

556
00:42:00,572 --> 00:42:09,286
Because I think the people that have, for instance, architecture skills and know how to
deploy to clouds and things like that are always going to be like the go-to skill set.

557
00:42:09,286 --> 00:42:13,649
Because without that knowledge, like how do even know what to look for?

558
00:42:13,649 --> 00:42:15,079
It'd be like being a doctor.

559
00:42:15,079 --> 00:42:25,166
If somebody comes in with a heart issue and you have no heart experience at all, like you,
I don't know, you work on, you you're an orthopedist or something who just

560
00:42:25,166 --> 00:42:25,908
Cardiologist.

561
00:42:25,908 --> 00:42:26,891
Yeah, for sure.

562
00:42:26,891 --> 00:42:32,204
My, my, uh my, my, CEO Dorota, she like keeps on saying the same.

563
00:42:32,204 --> 00:42:33,446
uh

564
00:42:33,450 --> 00:42:40,036
really brings the same idea to the story, is that LLMs raise the floor and not the ceiling
usually.

565
00:42:40,036 --> 00:42:50,606
And so it does give people the capability of delivering the same level of stuff, but it
doesn't say anything about delivering it to a satisfactory level ah or what's necessary

566
00:42:50,606 --> 00:42:59,384
for say a real enterprise organization or to release a product that's going to be safe for
both the company to manage and for their users.

567
00:42:59,384 --> 00:43:00,673
Totally agree, totally agree.

568
00:43:00,673 --> 00:43:02,862
That's good analogy, I like that raises the floor.

569
00:43:02,862 --> 00:43:14,582
I have not loved my using LLMs and agents to generate code, but in forcing myself to do
it, I sort of realized one valuable aspect, which was what if I had to ask the question,

570
00:43:14,582 --> 00:43:22,182
okay, I'm going to go out on a limb here and say that I am not the worst software engineer
that ever existed, which means that I have some good points.

571
00:43:22,622 --> 00:43:32,332
And the things that I can utilize and pass on to an LLM to do the right thing and the
things that I get mad when the LLM doesn't do, I need to be better at articulating because

572
00:43:32,332 --> 00:43:34,724
that actually is maybe the value that I can bring.

573
00:43:34,724 --> 00:43:44,471
And so if I can transfer that to say another person that I'm mentoring, someone else on my
team or to an LLM, in order to do that realistically, I need to be able to articulate it.

574
00:43:44,471 --> 00:43:47,192
And so thinking about that is something that I've had to do specifically.

575
00:43:47,192 --> 00:43:49,924
Like, why do I like this pattern over that pattern?

576
00:43:49,924 --> 00:43:53,336
Why is there a special case here versus a special case there?

577
00:43:53,357 --> 00:43:57,109
It's been a very long time since I've actually thought about what those things are.

578
00:43:57,109 --> 00:43:58,771
And I think this does bring back to the story.

579
00:43:58,771 --> 00:44:00,884
And the more you think about that, I think,

580
00:44:00,884 --> 00:44:12,040
is an interesting area and as software engineers, as technical people, one of the things
we love to do is be non-logical and have philosophical discussions about principles rather

581
00:44:12,040 --> 00:44:16,242
than actually doing the real work because that's where the enjoyment is, right?

582
00:44:16,323 --> 00:44:18,864
And I do feel like I'm a little bit back there now.

583
00:44:18,864 --> 00:44:22,646
I mean, I'm doing that instead of doing the real work, but it is interesting.

584
00:44:22,646 --> 00:44:29,952
And I think this is where the of the start of agents MD or Claude MD or whatever you're
utilizing to drive the agents or the skills that you pull in.

585
00:44:29,952 --> 00:44:33,885
are so relevant, but I think people misunderstand what the purpose is.

586
00:44:33,885 --> 00:44:37,297
It isn't just like go out on the internet and copy something someone else says.

587
00:44:37,297 --> 00:44:41,810
This is the best thing for rust or C sharp or JavaScript.

588
00:44:41,810 --> 00:44:50,756
It should be something that you're generating because realistically the whole goal is to
automate your activity rather than just have something be generated that someone else did.

589
00:44:50,756 --> 00:44:53,798
And I think that's why reviewing the skills is so important.

590
00:44:53,798 --> 00:44:59,542
The other reason I sort of come to this and I want to get your take on it is right now I
actually like

591
00:44:59,542 --> 00:45:02,205
that I run out of tokens on doing work.

592
00:45:02,205 --> 00:45:04,227
It forces me to stop.

593
00:45:04,227 --> 00:45:09,862
I'm actually happier at that moment that I no longer have to work and I have to go do
something else and come back later.

594
00:45:09,862 --> 00:45:20,082
And I fear that we're gonna keep going down the longer agent loops rather than shorter
ones and token consumption, uh which I feel like will make software engineering even

595
00:45:20,082 --> 00:45:20,453
harder.

596
00:45:20,453 --> 00:45:24,446
And I don't know if I have this fully thought out, but it's something like that.

597
00:45:24,696 --> 00:45:32,513
I don't disagree at all actually because m the longer the loops run um and you know we're
talking they can go for days now.

598
00:45:32,513 --> 00:45:43,072
Okay that's fantastic like you could generate even more features right and I'm going to go
back to who's doing the security checks who's doing the code reviews do the features

599
00:45:43,072 --> 00:45:45,544
actually work as advertised.

600
00:45:45,544 --> 00:45:52,448
mean yeah you could have a good you know PRD document or whatever you're doing for this
because I'm one of my big big

601
00:45:52,448 --> 00:46:03,763
like go to things, which I'm sure you do as well, is I have by far the biggest success
when I switch into plan mode first, plan it out, and then I work with it to get to the

602
00:46:03,763 --> 00:46:06,414
final state where I'm like, okay, that looks pretty good.

603
00:46:06,414 --> 00:46:08,765
Because otherwise it's just too vague sometimes.

604
00:46:08,765 --> 00:46:13,867
my whole point of this is if the agent loop runs even longer, to me that calls for a
couple things.

605
00:46:13,867 --> 00:46:20,354
I already talked about, yeah, you still got to do the security checks, the code reviews,
all that, but you need even more guardrails.

606
00:46:20,354 --> 00:46:26,577
Because if you're going to let it run for days or whatever and you don't have any
guardrails, next thing you know, you just crashed off the canyon wall.

607
00:46:26,577 --> 00:46:32,959
And yeah, you have a feature and it's a horrible feature that, you know, who knows what
it's doing.

608
00:46:32,959 --> 00:46:41,023
But that's where you to go back to what you're saying about like sharing with colleagues,
you know, your knowledge and things like that.

609
00:46:41,023 --> 00:46:44,534
That's where I think skills like and I know this will sound weird.

610
00:46:44,534 --> 00:46:49,876
Well, maybe it won't sound weird, but you know, like critical thinking skills like are you
able to question

611
00:46:49,876 --> 00:46:54,551
yourself and how you look at things without being like, no, I'm always right.

612
00:46:54,551 --> 00:46:55,252
You know what I mean?

613
00:46:55,252 --> 00:47:04,481
Because in the AI world, I feel like we have to move into this question everything for the
first time in a long, maybe ever in human history.

614
00:47:04,481 --> 00:47:13,069
If you have ideas, you're like superpowers now, um, with these coding agents, which is
fantastic, except for all those other things we talked about.

615
00:47:13,539 --> 00:47:18,897
Maybe that's a good place to leave off on the episode and switch over to PIX.

616
00:47:18,897 --> 00:47:21,058
So Dan, what did you bring for us today?

617
00:47:21,058 --> 00:47:34,732
Yeah, I talked quite a bit about skills earlier, and one of the challenges I've had is uh
like if I'm on, let's just say Codex, for example, and I'm coding or Cloud Code or Copilot

618
00:47:34,732 --> 00:47:37,061
CLI, whatever it is, they all have skills.

619
00:47:37,061 --> 00:47:44,084
The problem is you'll get a skill for one of those and then you'll switch because I use
multiple agents at same time a lot of times.

620
00:47:44,084 --> 00:47:48,265
I will do code reviews, for example, and I'll run all three of those.

621
00:47:48,265 --> 00:47:50,686
I actually use all three of those and

622
00:47:51,118 --> 00:47:59,178
The reason is I'll get consensus across the three and then I'll identify the biggest
issues, you know, and it actually works out really well to do that versus just one.

623
00:47:59,178 --> 00:48:08,858
And so the problem is, though, like you start getting these skills installed and what ends
up happening is they get out of sync because I don't know about you, but like I have a VM

624
00:48:08,858 --> 00:48:12,598
I work with sometimes directly, which is a little more sandboxed.

625
00:48:12,598 --> 00:48:17,998
And I'm less like I can just let things run freely on that VM.

626
00:48:17,998 --> 00:48:20,806
But then I also have like I work on a Mac.

627
00:48:21,082 --> 00:48:24,673
And I have, you know, like Copilot CLI, for example, on there.

628
00:48:24,673 --> 00:48:27,524
And now now everything's out of sync on the skills.

629
00:48:27,524 --> 00:48:32,825
So there's this uh it's called Skillshare and uh it's a GitHub repo.

630
00:48:32,825 --> 00:48:38,207
If you just search Skillshare one word and there's I've seen some others pop up lately, by
the way, that do this.

631
00:48:38,207 --> 00:48:42,028
But what it does is you can install this Skillshare.

632
00:48:42,028 --> 00:48:44,088
It's kind of like a skill sinker.

633
00:48:44,088 --> 00:48:46,729
It'll actually sync them all in one place up to GitHub.

634
00:48:46,729 --> 00:48:50,552
And then on any machine, I can say Skillshare poll.

635
00:48:50,552 --> 00:48:57,190
push sync and what sync will do is automatically sync them across all my agent harnesses
that use skills.

636
00:48:57,190 --> 00:48:57,951
So it's pretty cool.

637
00:48:57,951 --> 00:48:59,713
So anyway, yeah, it's a technical one.

638
00:48:59,713 --> 00:49:00,576
So sorry, Warren.

639
00:49:00,576 --> 00:49:02,308
No, I actually totally get it.

640
00:49:02,308 --> 00:49:10,355
I think it will be interesting, especially if we look at the sorts of problems that the
listeners are dealing with.

641
00:49:10,355 --> 00:49:20,024
if you have, especially in a professional environment where you may have agents running
both on the development side for every single engineer or team member you have present, as

642
00:49:20,024 --> 00:49:26,978
well as agents running in any sort of cluster on behalf of users, where you may actually
either want to provide

643
00:49:26,978 --> 00:49:35,961
customers their own sort of thinking or even across what you're deploying like I feel like
we're back in the same world as you know early 2000s where it's how do I make sure all my

644
00:49:35,961 --> 00:49:45,264
build servers have the exact same versions of all of the dependencies so that when I build
it doesn't say yeah you know this doesn't work seg fault or whatever because it doesn't

645
00:49:45,264 --> 00:49:50,546
actually have the right version of the Microsoft redistributable C++ libraries on it.

646
00:49:51,471 --> 00:49:54,886
I think we're going to see that with agents, with skills, with all these things.

647
00:49:54,886 --> 00:49:58,734
And yeah, it's funny how things are just a big circle, aren't they sometimes?

648
00:49:58,734 --> 00:50:00,474
What is old is new again.

649
00:50:00,474 --> 00:50:02,594
Like everyone going to spec-based development.

650
00:50:02,594 --> 00:50:11,454
I'm just already thinking about the world where we're back in agile software development
with LLMs and I'm just putting my head down, ignoring spec-based for now, praying that we

651
00:50:11,454 --> 00:50:12,894
don't stay there forever.

652
00:50:12,894 --> 00:50:14,734
So, but I like the pick.

653
00:50:14,734 --> 00:50:17,814
I think it's super relevant and topical too.

654
00:50:17,814 --> 00:50:22,454
Okay, so I'll share what I brought, which is maybe a little relevant, hits too much at the
heart maybe.

655
00:50:22,454 --> 00:50:28,702
There's a paper called the Ironies of Automation from 1983 by Lee Zane Bainbridge.

656
00:50:28,702 --> 00:50:32,765
And she talks about, it's only five pages long, but I think it's absolutely great.

657
00:50:32,765 --> 00:50:39,210
The tasks left after automation are actually, ends up being the ones that are still
manual.

658
00:50:39,210 --> 00:50:47,296
And they're the parts that maybe an automation designer, or in this case in 2026, an LLM
company's company couldn't figure out how to automate.

659
00:50:47,296 --> 00:50:51,459
And so they're actually the most complicated parts of the tasks that are still left over.

660
00:50:51,459 --> 00:50:55,462
And the other aspect is what has been automated still needs to be monitored.

661
00:50:55,462 --> 00:50:59,285
And when something goes wrong, like you can't just have like you look at a nuclear
reactor.

662
00:50:59,285 --> 00:51:03,249
If you automate every part of it, how do you make sure the system is doing the right
thing?

663
00:51:03,249 --> 00:51:06,682
Well, I suppose you could have a monitor that pops up says everything's green.

664
00:51:06,682 --> 00:51:08,414
But how do know everything is green?

665
00:51:08,414 --> 00:51:15,400
Like do you just trust the same system that's doing the automation also that the monitor
that is generating and the alerts that is generated are also correct?

666
00:51:15,400 --> 00:51:17,282
Well, you probably need to dig into that.

667
00:51:17,282 --> 00:51:18,312
It's like, well, why is it green?

668
00:51:18,312 --> 00:51:19,764
OK, these stats are also green.

669
00:51:19,764 --> 00:51:23,176
Does that mean that all of the detectors are operating correctly?

670
00:51:23,176 --> 00:51:31,631
I guess we need to check to make sure that the input signals are all there and that it's
actually detecting those things correct and that those things are still in operating uh

671
00:51:31,631 --> 00:51:32,611
bands.

672
00:51:32,611 --> 00:51:38,255
All of that extra work on top is work that you wouldn't have to have done if you were just
doing the manual labor.

673
00:51:38,255 --> 00:51:40,856
And so this is why we can see companies.

674
00:51:40,856 --> 00:51:43,958
There was actually a paper not too long ago released by

675
00:51:44,090 --> 00:51:51,046
McDonald's and Wendy's after the installation of their automated cashiers that the level
of training was actually increased.

676
00:51:51,046 --> 00:51:54,319
The number of technical staff had to increase overall in the organization.

677
00:51:54,319 --> 00:51:59,843
Adding self-service ordering machines cost the company money, not necessarily made them
money.

678
00:51:59,843 --> 00:52:06,989
Now, you could argue that automation, the whole point is not to improve the process, but
to make it scale.

679
00:52:06,989 --> 00:52:13,548
And I think this is where this sort of an interesting duality uh analogy comes in with the
LLMs where

680
00:52:13,548 --> 00:52:15,799
I think you have to really look at what your bottlenecks are.

681
00:52:15,799 --> 00:52:20,621
And if an LLM can solve your bottleneck, then it's a good thing to actually introduce in
your process.

682
00:52:20,621 --> 00:52:29,585
But if you're just introducing it because why not, and you're not solving a bottleneck,
you could actually be requiring more complex understanding of your system to do the

683
00:52:29,585 --> 00:52:31,726
software development lifecycle or wherever you're sticking it in.

684
00:52:31,726 --> 00:52:34,397
So it may not necessarily be the right thing on the forefront.

685
00:52:34,397 --> 00:52:42,230
And as you pointed out, it could be even decades before companies that go down this route
actually see the impact of their decisions.

686
00:52:42,414 --> 00:52:44,554
So I don't know, I really liked this paper.

687
00:52:44,554 --> 00:52:52,534
It's like 1983, it's like, wow, whatever was a whole new, what was old is new again.

688
00:52:52,534 --> 00:52:55,474
So I think this has been a great episode, Dan.

689
00:52:55,474 --> 00:53:01,114
Thank you for coming on the show and gracing us with your presence and great stories for
the audience.

690
00:53:01,114 --> 00:53:02,246
Thank you so much.

691
00:53:02,542 --> 00:53:03,142
Well, thank you.

692
00:53:03,142 --> 00:53:04,422
I appreciate you having me.

693
00:53:04,544 --> 00:53:11,852
And thanks to all the listeners for tuning in for this episode of Adventures in DevOps,
and I hope to see everyone back next week.