1
00:00:07,832 --> 00:00:09,565
Welcome back to Adventures in DevOps.

2
00:00:09,565 --> 00:00:18,318
It certainly seems most of the tech industry is running around like headless chickens, and
we're hoping to bring some sensibility to the current state by focusing on observability.

3
00:00:18,318 --> 00:00:21,904
For that, we brought in co-founder of Grafana Labs, Anthony Woods.

4
00:00:21,904 --> 00:00:23,106
Welcome to the show.

5
00:00:23,106 --> 00:00:23,734
Thanks, Walter.

6
00:00:23,734 --> 00:00:26,157
It's it's a pleasure to be here and thank you for inviting

7
00:00:26,157 --> 00:00:34,763
been trying to get you on for for months and obviously it was one thing and then another
first Gryphonicon and then the recent source code exfiltration attacks, which uh we will

8
00:00:34,763 --> 00:00:35,884
definitely get to.

9
00:00:36,002 --> 00:00:36,263
Great.

10
00:00:36,263 --> 00:00:36,564
Yes.

11
00:00:36,564 --> 00:00:37,514
Definitely happy to get into it.

12
00:00:37,514 --> 00:00:38,319
It's definitely busy.

13
00:00:38,319 --> 00:00:44,385
Uh I mean obviously the observability industry's uh got a lot of work cut out for it.

14
00:00:44,385 --> 00:00:48,010
is like the most top of mind thing regarding trying to help customers today?

15
00:00:48,010 --> 00:00:54,199
Like I mean, obviously Otel is almost old at this point since twenty nineteen, but you
know, what is old is new again.

16
00:00:54,199 --> 00:00:56,794
So but maybe there's something else particularly on your mind.

17
00:00:56,794 --> 00:00:57,336
Yeah.

18
00:00:57,336 --> 00:01:03,190
I mean, I think it's just a lot of customers that we're working with are just going
through revamping their observability strategies.

19
00:01:03,190 --> 00:01:05,062
and there's a lot of reasons for that.

20
00:01:05,062 --> 00:01:11,647
you know, the two things that, you know, their engine engineering leadership, you know, is
focused on is reducing costs, right, and improving productivity.

21
00:01:11,647 --> 00:01:11,897
Right.

22
00:01:11,897 --> 00:01:14,619
Like b basically that's that's the things that they care about most.

23
00:01:14,619 --> 00:01:18,282
Um and you know, what we've seen is observability is a big part of that.

24
00:01:18,282 --> 00:01:23,425
Observability, you know, for a lot of our customers, you know, is a very large line item
on their bills.

25
00:01:23,425 --> 00:01:24,406
And so they want to

26
00:01:24,406 --> 00:01:26,400
work out ways where they can optimize those costs.

27
00:01:26,400 --> 00:01:32,746
So they're trying balance both deliver more observability to the developing teams, at the
same time do it cheaper.

28
00:01:32,746 --> 00:01:35,490
Um, you know, so it's a little bit of conflicting priorities.

29
00:01:35,490 --> 00:01:42,069
I think everyone's familiar with Coinbase reporting pre pandemic, the sixty five million
dollar spend on data dog.

30
00:01:42,114 --> 00:01:42,454
Yeah.

31
00:01:42,454 --> 00:01:42,794
Yeah.

32
00:01:42,794 --> 00:01:43,015
Yeah.

33
00:01:43,015 --> 00:01:43,495
It's interesting.

34
00:01:43,495 --> 00:01:45,116
There's there's it's not unusual.

35
00:01:45,116 --> 00:01:48,057
You know, we do see a lot of customers that have very, very large bills.

36
00:01:48,057 --> 00:01:53,390
Um and the thing that we're hearing from those customers is that they just feel that
there's just misalignment between the cost and the value.

37
00:01:53,390 --> 00:01:58,763
You know, back in the days when um you know, interest rates were zero dollars, right, and
and money was free, um, it didn't matter, right?

38
00:01:58,763 --> 00:02:04,076
Like you could just go and spend money, uh, and all you really cared about was that
velocity, that engineering velocity, go and get things done.

39
00:02:04,076 --> 00:02:09,379
Um, but then when things changed and uh we started caring about money again, uh it matters
now, right?

40
00:02:09,379 --> 00:02:11,510
People want to still um

41
00:02:11,510 --> 00:02:16,751
You have their observability, but they really want to align that that cost and value and
make sure that it's making sense from a a business.

42
00:02:16,751 --> 00:02:26,132
How have you been able to overcome that sort of duality there where it does seem like
fundamentally, since if nothing goes wrong, it's a lot like security, that's the best you

43
00:02:26,132 --> 00:02:26,722
can hope for.

44
00:02:26,722 --> 00:02:32,250
And then when there is an issue, uh you are actually reporting on that effectively, so you
can dive in and solve that.

45
00:02:32,250 --> 00:02:33,272
I I think

46
00:02:33,272 --> 00:02:37,746
Companies historically that have lots of problems are easy to justify a spend and
observability.

47
00:02:37,746 --> 00:02:43,142
But the ones who need it the most tend to have uh fewer issues uh reported, right?

48
00:02:43,142 --> 00:02:44,693
The ones that have to have higher reliability.

49
00:02:44,693 --> 00:02:55,604
So especially the changeover in since we've left the money is free era, how have you been
able to sort of make that justification or uh the mindset shift for those customers to

50
00:02:55,604 --> 00:02:57,945
really understand what they're getting out of the stack?

51
00:02:58,370 --> 00:03:02,213
comes down to is you know really focusing on developer productivity.

52
00:03:02,213 --> 00:03:04,484
You know, our engineers are very busy, right?

53
00:03:04,484 --> 00:03:08,727
We've had this, you know, shift over the last decade or so a little bit more, right, to
this DevOps motion, right?

54
00:03:08,727 --> 00:03:13,240
Where more and more is being put on the developers to go uh they have to be responsible
for, right?

55
00:03:13,240 --> 00:03:17,503
Not only are they writing the code, they're testing it, they're deploying it, and they're
operationally responsible for it.

56
00:03:17,503 --> 00:03:24,668
When you have them operationally responsible for things, you want them to feel comfortable
that they're not going to be getting constantly paged at night, you know, that they're

57
00:03:24,668 --> 00:03:26,579
that things are working reliably.

58
00:03:26,579 --> 00:03:27,714
And if they're not

59
00:03:27,714 --> 00:03:31,828
the first thing that they're gonna do is they're gonna slow down uh the velocity of
deployment, right?

60
00:03:31,828 --> 00:03:34,451
Just to get back to how do we keep things sane and safe, right?

61
00:03:34,451 --> 00:03:40,076
But for some organizations, you know, the risk is the, you know, reliability, right, and
the customer impact of unreliable systems.

62
00:03:40,076 --> 00:03:43,774
For other customers, the risk is uh the lack of velocity in in

63
00:03:43,774 --> 00:03:54,394
On the engineering side, how historically observability tools, monitoring, alerting, logs,
traces, et cetera, are this sort of unfortunate situation that you get into only once you

64
00:03:54,394 --> 00:03:56,486
actually have an incident in production.

65
00:03:56,486 --> 00:04:06,216
Uh, how are you seeing that be a shift forward or earlier in a way so that people are
involved in understanding how their stuff will be available when those incidents happen?

66
00:04:06,216 --> 00:04:15,155
Or when you talk about productivity, my thought is, are you seeing a fundamental shift in
how development is being done or engineering is being done so that they have those tools

67
00:04:15,155 --> 00:04:16,558
earlier in the process?

68
00:04:16,558 --> 00:04:18,710
I think it's still very similar to how it's always been, right?

69
00:04:18,710 --> 00:04:20,601
There's no, you know, motivator like pain.

70
00:04:20,601 --> 00:04:25,985
Once you're your first you have your first incident, suddenly then you realize the
importance of of observability.

71
00:04:25,985 --> 00:04:26,395
Um, right.

72
00:04:26,395 --> 00:04:30,909
There's nothing worse than, you know, a system being offline and you finding about it
because a customer has told you.

73
00:04:30,909 --> 00:04:35,332
Or even worse, it's broken, but you don't know why and you don't have any data to
understand what went wrong.

74
00:04:35,332 --> 00:04:35,655
Right.

75
00:04:35,655 --> 00:04:41,446
The the the best you can do is turn it off and turn it back on again and hope things start
working again, but you just don't have the the data that you need.

76
00:04:41,446 --> 00:04:44,012
Um but what we're seeing happen now is just

77
00:04:44,012 --> 00:04:46,453
you know, with the the tools that we have available, right?

78
00:04:46,453 --> 00:04:54,042
Certainly with the AI tools for being able to build code, it is so much easier for people
to, you know, to create new p features and and new software.

79
00:04:54,042 --> 00:04:57,259
Um and so we are seeing that velocity is really accelerating.

80
00:04:57,259 --> 00:05:02,912
The level of expertise you need, right, to be able to ship software has, you know, has
dropped significantly.

81
00:05:03,373 --> 00:05:07,945
and then there's also questions about um, you know, how what is the quality of that code
that is being shipped, right?

82
00:05:07,945 --> 00:05:09,336
How reliable is it?

83
00:05:09,336 --> 00:05:13,538
and so certainly there's a a need, right, to understand

84
00:05:13,538 --> 00:05:15,792
You know, how these systems are working and what's happening.

85
00:05:15,792 --> 00:05:20,878
but it's still a case of, you know, sometimes you have to wait until you know comes back
to bite you first before you realise, you know, how important.

86
00:05:20,878 --> 00:05:28,704
if engineers are writing code faster, or if we can maybe we shouldn't call them engineers
anymore, if code is being written faster by organizations and there is something to be

87
00:05:28,704 --> 00:05:35,849
said about the average level of code potentially going down or getting closer to the mean
in any organization, that means more bugs will show up.

88
00:05:35,849 --> 00:05:41,403
It seems right on cue that we really need to be talking about observability, honestly,
here.

89
00:05:41,403 --> 00:05:43,904
Uh because what we're saying is

90
00:05:43,906 --> 00:05:46,266
worse code potentially could be getting to production faster.

91
00:05:46,266 --> 00:05:55,503
I I guess I'm waiting for that magical Grafana report to come out that reviews all the
customers' logs and how many alerts they've got and and whatnot and actually can point to

92
00:05:55,503 --> 00:06:00,676
some real data about this because there's lots of people standing up and saying, we're
faster, but we're better.

93
00:06:00,676 --> 00:06:06,089
And from my own personal unfortunate experiences, it seems like, yeah, sure, faster seems
like it's the opposite of better.

94
00:06:06,089 --> 00:06:08,760
Uh there is a inverse relationship here.

95
00:06:08,760 --> 00:06:10,393
So what's what's the solution?

96
00:06:10,393 --> 00:06:12,052
Is there something that we should be looking at?

97
00:06:12,052 --> 00:06:13,975
Is there a different way that we should be thinking about it?

98
00:06:13,975 --> 00:06:15,945
Is it just a matter of having the right tools?

99
00:06:16,078 --> 00:06:23,490
And so there's a couple of interesting things, you know, certainly we're we're going
through now, you know, in this new AI phase is that one, it's been much easier to go and

100
00:06:23,490 --> 00:06:28,181
ship code, but now, you know, there's a lot of questions around, you know, are they
actually valuable?

101
00:06:28,181 --> 00:06:28,391
Right?

102
00:06:28,391 --> 00:06:30,162
Are they delivering value to the business?

103
00:06:30,162 --> 00:06:30,472
Right.

104
00:06:30,472 --> 00:06:32,782
And this just comes down to classic observability, right?

105
00:06:32,782 --> 00:06:35,003
Do we understand what the system is doing?

106
00:06:35,003 --> 00:06:37,084
Do we understand how our users are interacting with it?

107
00:06:37,084 --> 00:06:39,274
Um, do we understand how much it costs to run this thing?

108
00:06:39,274 --> 00:06:41,831
And so all of these things are becoming more and more important, right?

109
00:06:41,831 --> 00:06:44,736
You know, people have have now realized it's not just about shipping features, right?

110
00:06:44,736 --> 00:06:45,836
It's understanding like

111
00:06:45,836 --> 00:06:47,108
Are we shipping the right features?

112
00:06:47,108 --> 00:06:49,711
You know, and are they doing what we want them to be doing?

113
00:06:49,711 --> 00:06:52,594
Are they having the the desired impact on our business?

114
00:06:52,594 --> 00:07:01,335
If we've gone through the the hype cycle of of everyone just well, let's just ship as many
things as we can, uh, to now where we're realizing, well, actually let's focus on things

115
00:07:01,335 --> 00:07:02,686
that actually are delivering value.

116
00:07:02,686 --> 00:07:03,107
Right.

117
00:07:03,107 --> 00:07:05,950
And success is not just about how many tokens am I birthed.

118
00:07:05,950 --> 00:07:08,651
I think measuring the widgets is sort of the the problem here.

119
00:07:08,651 --> 00:07:11,402
Like what is the appropriate metric for for use?

120
00:07:11,402 --> 00:07:21,446
I think historically the knowledge industry has moved off of measuring lines of code,
although it seems like we're back to that as the only reliable metric left.

121
00:07:21,446 --> 00:07:23,977
but I always like pointing to to the original Dora metrics.

122
00:07:23,977 --> 00:07:32,261
It does feel like there's something that was always missing from them, and I think you
really hi ran into it here, is that it's about the business impact, realistically.

123
00:07:32,261 --> 00:07:35,278
And we've already seen a huge impact to GitHub, the

124
00:07:35,278 --> 00:07:39,119
product and organization as a result of lots more code being generated.

125
00:07:39,119 --> 00:07:40,489
I don't think anyone's brought up so far.

126
00:07:40,489 --> 00:07:51,052
There's maybe I'll say even a competitive advantage of using an open source technology as
your standard, or an open standard as your interface, because you have access to or

127
00:07:51,052 --> 00:07:55,214
everyone now has access to models which are repeating that information.

128
00:07:55,214 --> 00:08:00,645
And if you do something fundamentally different, then you're actually going to be at a
competitive loss against the rest of the industry.

129
00:08:00,645 --> 00:08:04,078
Open tracing has been around for a while, but uh the OTEL it's I think it's

130
00:08:04,078 --> 00:08:05,088
twenty nineteen or so.

131
00:08:05,088 --> 00:08:10,987
Uh how did Grafana Labs sort of port what they had to that?

132
00:08:10,987 --> 00:08:18,057
Was it an easy transition because it was something you were always involved with, or was
there a lot of stuff that had to change under the hood in order to make that an effective

133
00:08:18,057 --> 00:08:19,668
first class integration?

134
00:08:19,668 --> 00:08:27,694
Again, we're just kind of lucky with timing where, you know, certainly we started out, you
know, Grafana, you know, really big focus on the infrastructure side with, you know,

135
00:08:27,694 --> 00:08:28,526
Prometheus.

136
00:08:28,526 --> 00:08:30,757
Um, actually, you know, even Graphite before that, right?

137
00:08:30,757 --> 00:08:33,089
So actually I'll tell a bit of a story here that's interesting, right?

138
00:08:33,089 --> 00:08:39,425
So like when we started Grafana back in uh 2014, right, the the telemetry database at the
time was Graphite, right?

139
00:08:39,425 --> 00:08:42,937
That was the the dominant one and that's what Grafana was originally written for.

140
00:08:42,937 --> 00:08:46,811
Um but from day one, we really wanted to focus on integration, right?

141
00:08:46,811 --> 00:08:48,312
And so be able to support other different

142
00:08:48,312 --> 00:08:51,813
Time series databases on another different data source in Grafana.

143
00:08:51,813 --> 00:08:59,815
And one of the nice things that we had is, you know, if part of the open source software,
we'd have some anonymous phone home data that would come back from all these Grafana

144
00:08:59,815 --> 00:09:03,546
instances around the world and we could see which data sources people were using.

145
00:09:03,546 --> 00:09:09,098
Um so that really helped us kind of see this massive explosion of the use of Prometheus,
right?

146
00:09:09,098 --> 00:09:15,702
Which is why we shifted uh back in 2017 to

147
00:09:15,874 --> 00:09:18,145
change and go and focus on the Prometheus ecosystem, right?

148
00:09:18,145 --> 00:09:23,689
Because we just saw this massive adoption of it and all of that driven by the adoption of
Kubernetes um as well, right?

149
00:09:23,689 --> 00:09:28,126
Which was the you know Prometheus and uh kind of a Kubernetes paired really well together.

150
00:09:28,126 --> 00:09:35,810
you you almost sort of lucked out as well because Prometheus initially is more of a
pull-based system, which means it doesn't matter so much what the underlying protocol was

151
00:09:35,810 --> 00:09:43,853
going to be or the standard that was going to be used because you controlled the whole
mechanism on your side rather than being a first class API where people had to push data

152
00:09:43,853 --> 00:09:54,248
to, all it really required in a lot of areas is maybe flush out the push gateway or some
other piece of technology as a intermediary between the technology stack or SDKs or third

153
00:09:54,248 --> 00:09:58,040
party products and this sort of proxy gateway into allowing Prometheus

154
00:09:58,040 --> 00:09:59,155
To pull from it.

155
00:09:59,298 --> 00:10:02,529
Yeah, I mean, uh we we certainly can't take credit for that ourselves.

156
00:10:02,529 --> 00:10:07,853
I mean we we certainly have a a large portion of the maintainers of the Prometheus
project, um, but we certainly didn't create it.

157
00:10:07,853 --> 00:10:12,711
And and a lot of it was inspired from um you know from the internal Good Google project
Borg, right?

158
00:10:12,711 --> 00:10:13,516
Borgmon, right?

159
00:10:13,516 --> 00:10:20,480
So Kubernetes was the real implementation of Borg, and then Prometheus was the kind of
re-implementation of Borgmon.

160
00:10:20,480 --> 00:10:26,433
interestingly enough, Otel then came out of the new monitoring system inside Google, which
is Monarch, right?

161
00:10:26,433 --> 00:10:28,374
And so there's a lot of uh

162
00:10:28,374 --> 00:10:30,795
I guess a competition between the two, right?

163
00:10:30,795 --> 00:10:32,856
And it still is, you know, basically a religious war.

164
00:10:32,856 --> 00:10:33,296
What's better?

165
00:10:33,296 --> 00:10:35,997
Push or versus pull for for metric collection.

166
00:10:35,997 --> 00:10:39,058
I think both have value, right, in certain situations.

167
00:10:39,058 --> 00:10:41,419
Um there's no, you know, perfect one, right?

168
00:10:41,419 --> 00:10:43,730
They're all good for the, you know, the right use case, right?

169
00:10:43,730 --> 00:10:46,141
And I'm a big believer of like pick the right, right tool for the job.

170
00:10:46,141 --> 00:10:50,913
Um, but uh it's great to be able to see we're seeing these kind of ecosystems kind of come
together.

171
00:10:50,913 --> 00:10:56,255
We don't want everyone reinventing the wheel and building their own, you know, uh
observability architecture and solutions, right?

172
00:10:56,255 --> 00:10:58,078
We want to be able to take that decision away from.

173
00:10:58,078 --> 00:11:02,210
think it's like a lot of things, uh, Microservice versus Monolith is a canonical one.

174
00:11:02,210 --> 00:11:06,433
It doesn't matter which one you pick, you're going to find a way to mess it up in the long
term anyway.

175
00:11:06,433 --> 00:11:08,464
Yeah.

176
00:11:09,605 --> 00:11:19,591
I do want to ask though, if you do think that a particular architecture is going to see uh
higher popularity in the years to come, potentially, because of the pushes that AI stack

177
00:11:19,591 --> 00:11:25,506
are are coming to, like not just what it's recommending, not if you just code something up
and you ask an LLM.

178
00:11:25,506 --> 00:11:26,546
You know, how should I do this?

179
00:11:26,546 --> 00:11:36,565
But realistically, what is actually needed by those stacks that are doing either data
pipelines or model distillation or whatever have you their uh inference at the end of the

180
00:11:36,565 --> 00:11:36,975
day?

181
00:11:36,975 --> 00:11:40,172
Is there something that is better fitted for that?

182
00:11:40,172 --> 00:11:49,746
Uh is there a particular uh strategy or tech stack or just ideology that comes with it, or
has everything related to AI, would you say it's very similar to what we've been doing all

183
00:11:49,746 --> 00:11:50,356
along?

184
00:11:50,664 --> 00:11:53,677
Um, I think it's similar, uh, but there certainly are some differences.

185
00:11:53,677 --> 00:12:00,943
You know, we definitely certainly from an observability perspective, we feel as though
this new agentic kind of phase that we're in is like a new software architecture phase.

186
00:12:00,943 --> 00:12:07,309
As big a change as moving from monoliths to microservices, you know, from from what we see
because it is a a new way of writing software.

187
00:12:07,309 --> 00:12:09,821
It's it's a new group of people writing software in some cases.

188
00:12:09,821 --> 00:12:12,546
Uh but also it has some new observability channels.

189
00:12:12,546 --> 00:12:19,931
One thing that I haven't been able to figure out fundamentally is whether or not the long
term trajectory though is accelerating or if it's still constant.

190
00:12:19,931 --> 00:12:31,099
And what I mean by that is we have seen a lot of technologies or ideas or paradigms pop up
in the last uh six years or so where they are very quick to be spun up and then very

191
00:12:31,099 --> 00:12:35,332
quickly get annihilated or removed because they aren't the optimal long term solution.

192
00:12:35,332 --> 00:12:41,022
And I'm wondering if from those major milestones one to another, if that is shrinking in
time.

193
00:12:41,022 --> 00:12:51,256
Or if you wait on understanding what the industry actually needs and then delivering
something, if you say you just released AI observability you know in 2026, is that is that

194
00:12:51,256 --> 00:12:51,586
late?

195
00:12:51,586 --> 00:12:53,487
I mean, from my standpoint, I'd say no.

196
00:12:53,487 --> 00:13:03,952
That's actually ahead of the curve, like you are riding the innovators wave here because
most organizations don't have even running their own models or or inference in their own

197
00:13:03,952 --> 00:13:07,173
data centers or even interacting with providers to do that.

198
00:13:07,173 --> 00:13:09,452
So I'm curious how you think about

199
00:13:09,452 --> 00:13:19,272
whether or not it's better to be supporting the innovators and early adopters here or
whether or not it'd be better to wait, see us actual standards pop up that make sense, and

200
00:13:19,272 --> 00:13:21,984
then build products to support those.

201
00:13:22,050 --> 00:13:24,152
Yeah, it's a that's a great question.

202
00:13:24,152 --> 00:13:27,154
Um, so it's very challenging right now.

203
00:13:27,154 --> 00:13:29,996
I mean, I think there's certainly some foundations that we want to focus on.

204
00:13:29,996 --> 00:13:33,740
Um, and that's where you know, we see things like open telemetry being really important,
right?

205
00:13:33,740 --> 00:13:35,761
Like, you know, that's a foundation technology, right?

206
00:13:35,761 --> 00:13:37,362
Should be part of the architecture.

207
00:13:37,362 --> 00:13:45,359
Because no matter what happens, no matter how the world around us changes, we still need
to have visibility into what our software is doing.

208
00:13:45,359 --> 00:13:46,732
That's always going to be

209
00:13:46,732 --> 00:13:47,202
True.

210
00:13:47,202 --> 00:13:50,304
And who we means is certainly going to change, right?

211
00:13:50,304 --> 00:13:58,379
you know, historically that has been our SREs and our developers, but more and more of
that is now our agents who still wanna have visibility into what our software's doing,

212
00:13:58,379 --> 00:13:58,599
right?

213
00:13:58,599 --> 00:14:00,220
But they still need that telemetry data.

214
00:14:00,220 --> 00:14:06,208
So for us, I think, you know, what we're understanding is that those foundations now are
really important in in getting it right.

215
00:14:06,208 --> 00:14:13,620
If everyone believes there's a bubble and everyone believes it's going to pop, that you
really have to question whether or not that is the collective y wisdom that will follow.

216
00:14:13,620 --> 00:14:18,502
Uh but th there there's a whole uh economic philosophy there that we're not gonna go into.

217
00:14:18,502 --> 00:14:27,815
But the thing I will ask is whether or not you feel like the the curve that was originally
identified in crossing the chasm, that you start with these innovators and early adopters,

218
00:14:27,815 --> 00:14:31,326
early majority, late majority, and then laggards, are you focused

219
00:14:31,326 --> 00:14:33,989
in a different part of the market from where you were before.

220
00:14:33,989 --> 00:14:41,447
Whereas maybe one could say, yeah, we're totally early majority, late majority, those are
the that's the best place to look for, you know, getting growth or customers or really

221
00:14:41,447 --> 00:14:42,428
just any sort of revenue.

222
00:14:42,428 --> 00:14:51,417
Uh, but if we think that it's changing, that maybe you're focused now in pushing to a
different spot because you see that growth or the amplitude of the number of customers or

223
00:14:51,417 --> 00:14:54,250
the market segment there to be much bigger than it was before.

224
00:14:54,380 --> 00:14:55,121
Yeah, definitely.

225
00:14:55,121 --> 00:14:59,139
I mean, we definitely are playing in the the early majority, um, right now.

226
00:14:59,139 --> 00:15:00,864
Like we saw that change

227
00:15:00,864 --> 00:15:02,814
Certainly for for our business a few years ago.

228
00:15:02,814 --> 00:15:05,606
Um and a lot of that is around open source, right?

229
00:15:05,606 --> 00:15:08,017
And we're like, hey, we've got this great open source technology stack.

230
00:15:08,017 --> 00:15:15,531
Um, you know, you can kind of mix and match and do what you want and you can choose how
you go and deploy our software and and you know have all these flexibility and choice.

231
00:15:15,531 --> 00:15:19,993
And then we were discovering, you know, the customers we were talking to were like, I we
don't want this choice thing, right?

232
00:15:19,993 --> 00:15:23,345
We want you to just tell us how to do observability and just do it properly.

233
00:15:23,345 --> 00:15:24,545
And so we're like, okay.

234
00:15:24,545 --> 00:15:28,757
So we then took, you know, our learnings and we went and built, you what is our Grafinal
Cloud platform today, right?

235
00:15:28,757 --> 00:15:30,788
Which is more opinionated, kind of out of the box.

236
00:15:30,788 --> 00:15:31,418
experience, right?

237
00:15:31,418 --> 00:15:36,280
Where we focus less on building the technology and focus on building solutions, right?

238
00:15:36,280 --> 00:15:43,743
What's interesting is with those users, like you know, even though they're, you know, the
early majority, they're typically a little bit later to adopt technology, um, they're

239
00:15:43,743 --> 00:15:47,186
still adopting, you know, this new AI identic model just as fast as everybody.

240
00:15:47,186 --> 00:15:53,452
People want to get into it because they think they're gonna miss out and that just drives
more and more the cycle uh of of adoption.

241
00:15:53,452 --> 00:16:03,211
What I do want to ask you is in the new product line changes that what you're delivering
now that you said the new features that are come out with uh the Barcelona conference this

242
00:16:03,211 --> 00:16:11,160
year, was there just a shift in what you're tracking or also how you're exposing that data
back for agents to consume it as well?

243
00:16:11,160 --> 00:16:14,752
There's a big change in how the data is consumed.

244
00:16:14,752 --> 00:16:17,564
Um, you know, which is very interesting for us, right?

245
00:16:17,564 --> 00:16:23,127
As a as a company that's founded itself on being the, you know, the dashboarding company,
right?

246
00:16:23,127 --> 00:16:28,190
To discover in a you know, the new world that we're moving into, it's like how important
is the dashboard anymore?

247
00:16:28,190 --> 00:16:28,700
Right.

248
00:16:28,700 --> 00:16:37,452
Um, so many are developers where they're sitting in Claude Code or Codex or um Copilot or
whatever they're using, and from that they're just asking the questions, right?

249
00:16:37,452 --> 00:16:38,626
And they're saying, hey.

250
00:16:38,626 --> 00:16:41,599
what's happening in production right now, you know, what is my system doing?

251
00:16:41,599 --> 00:16:47,466
And so we need to we've had to kind of change, right, and make sure that, you know, we're
building a platform that that can do both, right?

252
00:16:47,466 --> 00:16:55,893
Um, you know, one, we want to support this new model of of agents being able to consume
the telemetry data and and look at it and understand it and they can, you know, make some

253
00:16:55,893 --> 00:16:57,624
some good decisions around it.

254
00:16:57,635 --> 00:16:59,158
but at the same time we still want to be able to

255
00:16:59,158 --> 00:17:01,902
you know, support dashboards, but 'cause they are they are still useful.

256
00:17:01,902 --> 00:17:02,924
Um, right.

257
00:17:02,924 --> 00:17:04,516
Like especially when you wanna have them up on a screen.

258
00:17:04,516 --> 00:17:08,715
Uh you can see what's happening or you want to generate report that you can share with
leadership, for example.

259
00:17:08,715 --> 00:17:16,189
I guess there's a two part question here, which is like, you know, is the branding should
it still be Grafana Labs if the dashboards are for humans and you're transitioning to, you

260
00:17:16,189 --> 00:17:20,505
know, a more agentic, you know, interactive layer there and you know the

261
00:17:20,586 --> 00:17:30,265
The second part is is realistically this problem where do we think that even the
dashboards may be the better solution for models if they are multimodal and how they're

262
00:17:30,265 --> 00:17:31,276
consuming data?

263
00:17:31,276 --> 00:17:34,319
Is there something captured in the way the dashboard was created?

264
00:17:34,319 --> 00:17:39,123
Or fundamentally do you need to change APIs to actually expose the data in a different
way?

265
00:17:39,123 --> 00:17:41,025
Or is it you've already had this data all along.

266
00:17:41,025 --> 00:17:42,056
It's already right there.

267
00:17:42,056 --> 00:17:46,680
It's just a matter of having the right MCP server set up or having the user set up some

268
00:17:46,680 --> 00:17:50,041
configuration for access and so it's an access control situation.

269
00:17:50,041 --> 00:17:53,196
It's not necessarily something fundamentally has to be different.

270
00:17:53,196 --> 00:17:57,569
Yeah, I mean there's definitely changes that we're making to accommodate these new use
cases.

271
00:17:57,569 --> 00:18:04,482
There's a couple of different things and the couple of different challenges, you know,
that we faced early on with AI and and one of it is that I guess for most of us, right,

272
00:18:04,482 --> 00:18:06,854
observability data is a lot of unstructured data.

273
00:18:06,854 --> 00:18:09,910
Um and we know LLMs don't really work well with unstructured data.

274
00:18:09,910 --> 00:18:11,366
It doesn't make a lot of sense to them.

275
00:18:11,366 --> 00:18:14,308
it doesn't make a lot of sense to individuals, right, when we when we look at it, right?

276
00:18:14,308 --> 00:18:18,433
We just you typically have context to kind of know what it means.

277
00:18:18,433 --> 00:18:21,890
And so one of the things that we've focused a lot on is, you know, how do we

278
00:18:21,890 --> 00:18:23,373
build that context around it, right?

279
00:18:23,373 --> 00:18:27,611
The use of these open standards and open ecosystems around open climate should really
help, right?

280
00:18:27,611 --> 00:18:30,518
Because it's really well documented, you know, data that we're collecting.

281
00:18:30,518 --> 00:18:37,646
you doing something complex like setting up some sort of semantic layer for it to
understand the data that you've actually collected, or are you using some sort of third

282
00:18:37,646 --> 00:18:41,314
party product or something built into Grafana to actually utilize this?

283
00:18:41,314 --> 00:18:46,847
Yeah, so we have this capability inside Grafina Cloud where one, we're leveraging uh open
telemetry, right?

284
00:18:46,847 --> 00:18:50,480
Where, you know, it does have a very, you know, well structured semantic layer.

285
00:18:50,480 --> 00:18:52,911
and then it's a um extensible system, right?

286
00:18:52,911 --> 00:18:58,929
So if you're using different data um you know outside of open telemetry, you can then go
and define rules um and build upon it.

287
00:18:58,929 --> 00:19:01,142
You can say, Hey, you know, these labels mean these things.

288
00:19:01,142 --> 00:19:08,908
training your own models to work through those business cases or are you pulling an open
weight model or another provider to actually help you through those internal challenges?

289
00:19:08,908 --> 00:19:11,299
Yeah, I mean, so right now we're just using kind of the frontier models.

290
00:19:11,299 --> 00:19:19,574
What we discovered, you know, that was really valuable for us is that um the reason why
the frontier models work so well with, you know, our use cases and our technology is

291
00:19:19,574 --> 00:19:21,795
because of our open source, you know, ecosystem.

292
00:19:21,795 --> 00:19:21,935
Right?

293
00:19:21,935 --> 00:19:29,079
We've we've had this more than a decade of you know building open source technology, but
it's not just about the technology, it's the the community and the ecosystem that we've

294
00:19:29,079 --> 00:19:29,439
built, right?

295
00:19:29,439 --> 00:19:36,163
There's so many people that blog about you know how to use Grafana or they write tutorials
or they've got public dashboards or they've got you know all this content that's just on

296
00:19:36,163 --> 00:19:38,104
the public internet about both what

297
00:19:38,104 --> 00:19:41,398
they're doing with our technology and and how to how to actually use it to do that.

298
00:19:41,398 --> 00:19:44,122
So uh the models just know, right?

299
00:19:44,122 --> 00:19:47,606
We haven't had to go and train our own models to to get to success.

300
00:19:47,606 --> 00:19:49,017
Um they just work out of the box.

301
00:19:49,017 --> 00:19:51,078
And that's a very big competitive advantage that we

302
00:19:51,078 --> 00:19:59,565
I'm very excited about asking questions to the frontier models and getting back relevant
context about your your products or how to implement things from a technology standpoint.

303
00:19:59,565 --> 00:20:06,941
Or I you know, for us recently it was like actually understanding our business case and
being able to explain it, you know, fitting in the market segment, especially when you're

304
00:20:06,941 --> 00:20:08,022
doing something very nuanced.

305
00:20:08,022 --> 00:20:13,057
So I I mean, I don't think I've ever heard anyone stand up and say, Yeah, I can't wait to,
you know, build our own models.

306
00:20:13,057 --> 00:20:15,878
I I have a lot of extra cash I want to throw away and burn.

307
00:20:15,878 --> 00:20:17,989
And so this is gonna give us a great opportunity.

308
00:20:17,989 --> 00:20:20,832
Uh there is something to be said about small models.

309
00:20:20,886 --> 00:20:28,943
that can run faster and are trained on a very specific data set, but when you are
utilizing open standards or you have such a huge prevalence in the market, y you may not

310
00:20:28,943 --> 00:20:30,644
ever need to actually do that.

311
00:20:30,926 --> 00:20:33,629
I gonna say, especially with the the how fast the models are are evolving, right?

312
00:20:33,629 --> 00:20:37,304
Like it's very difficult to compete against Anthropic and OpenAI and Google, et cetera,
right?

313
00:20:37,304 --> 00:20:40,678
Like they have a a a much bigger budget, right?

314
00:20:40,678 --> 00:20:44,281
And an infrastructure footprint to go and train models than anybody else.

315
00:20:44,418 --> 00:20:51,861
Well, you know, if anyone is actually concerned about that, I I would say that
realistically, a lot of different ways in which you can utilize a model are easily

316
00:20:51,861 --> 00:20:56,383
utilized via the non frontier models by the private model providers.

317
00:20:56,383 --> 00:21:04,827
There's a lot of great open weight models where if you and I hate to say this, if you're
just prompting it right, y you can get a good response without a problem.

318
00:21:04,827 --> 00:21:10,890
But I I I mean I do understand what you're saying, you if you're not doing anything, what
I have found is that for a lot of use cases.

319
00:21:10,890 --> 00:21:18,696
the system prompts that the model providers, Gemini, Anthropic, OpenAI have do a l huge
disservice to the value that they're offering.

320
00:21:18,696 --> 00:21:29,023
So if you already are on the path of managing your own system prompt and what how the
agents are really focused on and the data that is exposed to them through MCP or or tool

321
00:21:29,023 --> 00:21:35,967
calling, I think you can get really far today and probably a lot farther if you are just
running your own uh smaller model off the stack.

322
00:21:35,988 --> 00:21:39,520
Something that I I I saw and I want to get your perspective on this is that

323
00:21:39,520 --> 00:21:48,365
recently, especially with the increase in open source slot, is that we've seen models and
people utilizing models to create a lot of pull requests on open source libraries.

324
00:21:48,365 --> 00:21:57,110
And on the flip side, we've seen this mistake that'll that some companies have made, most
notably HashiCorp or Local Stack transitioning their license models to some sort of

325
00:21:57,110 --> 00:21:57,920
business license.

326
00:21:57,920 --> 00:22:02,843
I really appreciate how Gravana's used the AGPL version three.

327
00:22:02,843 --> 00:22:04,994
And I I just don't understand why anyone would ever switch.

328
00:22:04,994 --> 00:22:08,866
And maybe after this call you'll be like, we're we're we're gonna be switching.

329
00:22:10,595 --> 00:22:21,600
I mean, you basically have said how important it is to contribute publicly open source,
have out people out there who are generating for their own benefit docs and guides on

330
00:22:21,600 --> 00:22:26,142
using Grafana who you pay zero to basically and get all the benefit from.

331
00:22:26,142 --> 00:22:28,163
And you lose all that by switching off.

332
00:22:28,163 --> 00:22:33,622
But what I do wanna ask is since the now huge amount of

333
00:22:33,622 --> 00:22:42,086
capability or quote unquote productivity that we see open source developers have in integr
interacting or creating pull requests to open source projects.

334
00:22:42,086 --> 00:22:50,079
Uh have you had to deal with any of the blowback from that on your side, how have you have
there been any challenges to deal with potentially increased pull requests or spam or

335
00:22:50,079 --> 00:22:51,970
noise on issue tickets, etcetera?

336
00:22:51,970 --> 00:22:52,940
Yeah, I mean definitely.

337
00:22:52,940 --> 00:22:56,322
I mean the first thing I can say, yes, AGPL, right?

338
00:22:56,322 --> 00:23:01,025
We have no no plans to uh to move to a a closed source licensing model.

339
00:23:01,025 --> 00:23:02,206
Uh in fact we're really excited.

340
00:23:02,206 --> 00:23:11,401
I mean, like you know, we've seen uh a number of you know big open source well known open
source you know companies and projects that move to uh a a closed source license, you

341
00:23:11,401 --> 00:23:14,152
know, typically SSPL, um, who are now coming back, right?

342
00:23:14,152 --> 00:23:20,666
Who are now adopting AGPL, right, just because they've you know and they're referencing us
as as as a reason for that, which you know we we're very proud of.

343
00:23:20,994 --> 00:23:23,725
But you know, open source for us, you know, it's it's important, right?

344
00:23:23,725 --> 00:23:25,245
Like it's a deliberate business decision, right?

345
00:23:25,245 --> 00:23:28,085
We see a lot of value in building healthy ecosystems, right?

346
00:23:28,085 --> 00:23:31,297
It's not just about um, you know, the code itself, right?

347
00:23:31,297 --> 00:23:35,728
It's about building vibrant ecosystems and a community of people that can kind of share
knowledge and kind of come together.

348
00:23:35,728 --> 00:23:38,489
And that's what's really important to us and and that's working really well.

349
00:23:38,489 --> 00:23:46,331
And obviously, you know, we were Apache licensed and then we moved to AGPL because, you
there is still that threat of, you know, the hyperscalers, you know, taking your open

350
00:23:46,331 --> 00:23:50,232
source technology and and, you know, c uh you know, getting all the revenue from it,
right?

351
00:23:50,232 --> 00:23:50,786
So

352
00:23:50,786 --> 00:23:52,036
You know, you do want to go and protect that.

353
00:23:52,036 --> 00:23:55,908
And but we do feel feel like the AGPL model does give us the protections that we need.

354
00:23:55,908 --> 00:24:01,189
We are seeing, you know, a lot of, you know, pull requests coming through and and it is
problematic.

355
00:24:01,189 --> 00:24:07,381
And, you know, I think we're having a similar approach, um, you know, where we're we're
kind of just advising people just don't send it, right?

356
00:24:07,381 --> 00:24:16,139
One of the challenges is that for large uh, you know, open source projects and large code
bases, um, one thing that's really important to make sure it's is to well, one thing

357
00:24:16,139 --> 00:24:18,316
that's really important is to make sure the code is maintainable.

358
00:24:18,316 --> 00:24:23,761
And so there's a lot of work that we go and do to enforce our own coding standards and our
own way of doing things.

359
00:24:23,761 --> 00:24:30,758
And so as we're making changes, we want to make sure that the changes that are getting
introduced kind of match the rest of the code base, right?

360
00:24:30,758 --> 00:24:33,661
And are aligned with those standards that that we've got internally.

361
00:24:33,661 --> 00:24:38,305
And a lot of the AI uh code that's generated, they're not generated based on our
standards, right?

362
00:24:38,305 --> 00:24:43,289
They're generated based on whatever standards they've decided uh is common, for example.

363
00:24:43,289 --> 00:24:44,020
And so

364
00:24:44,020 --> 00:24:48,114
obviously there's a big work to go through that review cycle and have to go and change
things, right?

365
00:24:48,114 --> 00:24:55,900
So that, you know, even though even though the pull request may implement the change and
it and it may, you know, be technically correct, right?

366
00:24:55,900 --> 00:24:58,132
Like we still want to make sure that the code's maintainable.

367
00:24:58,132 --> 00:25:07,170
That is a big challenge where yeah, it's very difficult to accept pull requests that are
just AI generated, um, 'cause they don't they don't deliver a huge amount of value to the

368
00:25:07,170 --> 00:25:07,584
open

369
00:25:07,584 --> 00:25:08,015
projects.

370
00:25:08,015 --> 00:25:08,765
Yeah, for sure.

371
00:25:08,765 --> 00:25:16,782
I is there any sort of technical implementation or technical solution that you've put in
place or non technical implementation to help combat this in any way?

372
00:25:16,782 --> 00:25:21,195
And I know there's some conversations, certainly in the Open Telemetry project, right,
where they've they've had some pain around this.

373
00:25:21,195 --> 00:25:28,089
And I think they just, you know, had the approach of just like, you know, saying, you
know, we're just not accepting, you know, AI generated pull requests.

374
00:25:28,089 --> 00:25:29,359
Um, right, for one.

375
00:25:29,359 --> 00:25:31,471
Um, you know, they just won't get reviewed.

376
00:25:31,471 --> 00:25:34,492
Um, you know, unl if it unless it looks like it's a person.

377
00:25:34,492 --> 00:25:38,435
Um I think that's just the the easier uh approach.

378
00:25:38,435 --> 00:25:43,842
Um what's interesting is you know, we see this challenge in in the industry, right, for
software development where

379
00:25:43,842 --> 00:25:45,703
Yeah, we don't need new grads anymore, right?

380
00:25:45,703 --> 00:25:50,086
Uh AI agents can do the work that you would normally go and have a new grad do.

381
00:25:50,167 --> 00:25:53,970
and so then we run the problem of we don't need new grads, we only need experienced
engineers.

382
00:25:53,970 --> 00:26:03,516
How does how do we in 10 years time have experienced engineers if if no one's letting new
grads today go and develop their skills to, you know, to become experienced.

383
00:26:03,897 --> 00:26:07,330
so for me, I see that's where, you know, there's a big opportunity in open source, right?

384
00:26:07,330 --> 00:26:10,282
Where, you know, we still want those, you know,

385
00:26:10,434 --> 00:26:12,805
hands-on people we don't want AI to be used, right?

386
00:26:12,805 --> 00:26:15,126
Where they can go and uh and write code.

387
00:26:15,126 --> 00:26:16,817
Uh and it is a great ecosystem, right?

388
00:26:16,817 --> 00:26:19,168
Like you can go and write code and you can contribute it.

389
00:26:19,168 --> 00:26:22,890
And if you're a real person, uh, you know, the maintainers will support you, right?

390
00:26:22,890 --> 00:26:27,412
They'll coach you, they'll guide you on how to go and build, you know, a better software
and and you know improve things.

391
00:26:27,412 --> 00:26:36,035
So I see as open source as being the saving you know grace for you know how we're going to
get you know new grads to develop their own you know their skills over time so that they

392
00:26:36,035 --> 00:26:37,518
can you know be productive in the environment.

393
00:26:37,518 --> 00:26:44,281
I think that's very interesting perspective that open source loves artisanal software
development, you know, hand curated by by humans.

394
00:26:44,281 --> 00:26:53,245
Uh and it will set you apart as uh inexperienced engineer to actually go and do that
because maintainers want more human approach to it.

395
00:26:53,245 --> 00:27:00,117
I do think there is a little bit of the Silicon Valley's the the show, Jin Yang's hot dog
or not a hot dog problem though.

396
00:27:00,117 --> 00:27:06,722
Like ha I see a lot of these repositories popping up saying we only accept non-AI
generated code, but in practice.

397
00:27:06,722 --> 00:27:07,362
How do you know?

398
00:27:07,362 --> 00:27:15,578
Like if you know what code is being generated by an AI or an LLM, I feel like that is in
its own impossible problem to solve that lots of people have been trying to work on for, I

399
00:27:15,578 --> 00:27:16,969
suppose, now only four years.

400
00:27:16,969 --> 00:27:24,774
So I I I actually wonder in practice, like what like is this just a human going around
being like, yep, uh, that's LLM slot, close, close, close, close.

401
00:27:24,774 --> 00:27:31,338
Or if there's an automatic system that someone's actually figured out to correctly
identify LM generated stuff.

402
00:27:32,502 --> 00:27:34,002
I mean, I think it's challenging, right?

403
00:27:34,002 --> 00:27:41,104
As soon as you build a tool that can identify uh what looks like, you know, AI content,
the AI changes and and, you know, works around it.

404
00:27:41,104 --> 00:27:43,925
uh, you know, that's always gonna be the case.

405
00:27:43,925 --> 00:27:49,317
Um, I think really what it comes right now, it's certainly a um almost like a knee jerk
reaction, right?

406
00:27:49,317 --> 00:27:53,418
Like we're just getting overwhelmed with, you know, a bunch of of you know AI slop and and
noise.

407
00:27:53,418 --> 00:27:56,629
And so, you know, we're we're taking an approach to kind of filter that out.

408
00:27:56,629 --> 00:28:01,570
I think, you know, over time I think we're gonna be able to kind of focus on

409
00:28:02,048 --> 00:28:05,731
Not saying just no, right, but articulating what it is that we want, right?

410
00:28:05,731 --> 00:28:11,196
Because there there is probably a future where there is some AI generated code, um, you
know, where it's going to be valuable, right?

411
00:28:11,196 --> 00:28:15,971
So it's really about us, you know, uh, you know, as projects, really understanding what is
the the key things that we want, right?

412
00:28:15,971 --> 00:28:19,452
The main thing, you know, certainly for us that we understand is we want maintainable
change.

413
00:28:19,452 --> 00:28:23,993
I think I'm first of all, I think that's a super mature perspective compared to like we
don't accept any of this.

414
00:28:23,993 --> 00:28:32,346
Because the r the realistic r result is, well, we actually would accept it if it adhered
to all of our, you know, culture expectations and coding expectations.

415
00:28:32,346 --> 00:28:39,678
And I think just a lot of these repositories don't have that mentality of like what sort
of features are actually the right thing to build and what is the right way to do it.

416
00:28:39,678 --> 00:28:48,066
And then you may be actually motivated to uh add in some automation to codify your
expectations on which features get in.

417
00:28:48,066 --> 00:28:50,087
How big should the features be, et cetera, et cetera?

418
00:28:50,087 --> 00:28:56,449
Because I think we are very quickly going to end up in a world where you why would you
reject LLM generated pull requests?

419
00:28:56,449 --> 00:28:57,380
That's not what you care about.

420
00:28:57,380 --> 00:29:04,893
You care about LLM generated pull requests that are useless and have a long term
detrimental effect to the longevity of the project, et cetera.

421
00:29:04,893 --> 00:29:13,196
One of the challenges with open source technology has always been, and I sort of hinted at
this before, and I'm gonna try to get you to say something on the record, is uh the

422
00:29:13,196 --> 00:29:14,097
security aspect.

423
00:29:14,097 --> 00:29:17,430
There's a lot of thought that what was open source had in an

424
00:29:17,430 --> 00:29:21,953
inherent improved security mechanism because more people could look at that code.

425
00:29:21,953 --> 00:29:28,097
But I feel like over time, especially with the model usage increasing, everything is being
looked at for vulnerabilities.

426
00:29:28,097 --> 00:29:31,419
And one of one of these things actually turned out to be a problem.

427
00:29:31,419 --> 00:29:40,485
I don't recall exactly what Grafana was using, but um was susceptible some of their open
source uh repositories and closed store stuff to Shihalude, um, which is just this nasty

428
00:29:40,485 --> 00:29:42,216
worm out there that

429
00:29:42,242 --> 00:29:48,992
takes its name from Dune and basically just replicates itself by stealing MPM credentials,
et cetera, and asserting itself.

430
00:29:48,992 --> 00:29:54,820
And I'm sort of curious, um, how did you actually maybe you know, how did you actually
catch that this was happening?

431
00:29:54,820 --> 00:29:59,316
And the result was basically you're like, we don't need to pay any ransom for this.

432
00:29:59,448 --> 00:30:02,893
What we're seeing is is just like the supply chain uh attacks, right?

433
00:30:02,893 --> 00:30:05,847
Is is the the path that's being exploited right now.

434
00:30:05,847 --> 00:30:14,650
And you know, the reality is is everybody's using uh you know open source code, you know,
even if it's a commercial, you know, proprietary system, the reality is they're still, you

435
00:30:14,650 --> 00:30:16,032
know, vendoring in a lot of

436
00:30:16,032 --> 00:30:18,543
you know, open source um dependencies, right?

437
00:30:18,543 --> 00:30:20,013
Um, you know, it's a common thing, right?

438
00:30:20,013 --> 00:30:21,704
Like why do we want to go and reinvent the wheel?

439
00:30:21,704 --> 00:30:24,714
So it's a problem that's affecting everybody, um, right?

440
00:30:24,714 --> 00:30:27,475
Certainly not unique to to open source uh users.

441
00:30:27,475 --> 00:30:33,977
And um, you know, and and certainly, you know, we were very transparent with um, you know,
the incident that that affected us a couple of weeks ago.

442
00:30:33,977 --> 00:30:43,149
The summary of it for us is yes, supply chain attack, they were able to get a credential
from a developer um and use that to to access our GitHub uh repos.

443
00:30:43,149 --> 00:30:45,666
All they were able to get was our code bases, right, which

444
00:30:45,666 --> 00:30:48,437
For us, you know, Motro, that is open source anyway.

445
00:30:48,437 --> 00:30:49,907
Um, right.

446
00:30:49,907 --> 00:30:57,920
Um, but you know, because of the controls that we have in place, um, you know, at no time
were they able to access any of our production infrastructure or or access our customer

447
00:30:57,920 --> 00:30:58,470
data.

448
00:30:58,470 --> 00:31:02,422
Um and so um, you know, that's great, right?

449
00:31:02,422 --> 00:31:06,483
We're really proud of the, you know, the fact that we've got these controls in place that
we're able to kind of protect our data.

450
00:31:06,483 --> 00:31:09,074
There's certainly more that we're doing now to

451
00:31:09,282 --> 00:31:10,793
uh to protect things even further.

452
00:31:10,793 --> 00:31:12,383
Um but certainly yes.

453
00:31:12,383 --> 00:31:15,444
Um, you know, when they came to us and today we've we're gonna release all of your code.

454
00:31:15,444 --> 00:31:22,647
If you don't pay us, you know, like it was an easy choice for us to say, well we're not
gonna pay you one because, you know, how do how do we guarantee you're not gonna just

455
00:31:22,647 --> 00:31:23,437
release it anyway.

456
00:31:23,437 --> 00:31:26,549
But also like it's mostly you know open source code, right?

457
00:31:26,549 --> 00:31:30,270
Like uh you know that's not where our

458
00:31:30,530 --> 00:31:32,301
you know, differentiation lies, right?

459
00:31:32,301 --> 00:31:39,096
Like, you know, much to the displeasure of our sales team, um, you know, most of the
technology that we're building these days goes into our open source.

460
00:31:39,096 --> 00:31:40,517
Um or your lawyers.

461
00:31:40,517 --> 00:31:42,428
And so yeah.

462
00:31:42,428 --> 00:31:45,005
And so, you know, that that was a an easy choice for us to do.

463
00:31:45,005 --> 00:31:46,175
But certainly it was a lot of work.

464
00:31:46,175 --> 00:31:52,305
It was very impactful for the business to be able to, you know, have to go through and
audit everything, every single change on every single repository.

465
00:31:52,305 --> 00:31:54,358
Like what, you know, has there been any compromises?

466
00:31:54,358 --> 00:32:01,020
love the perspective and the article that was released, honestly, because it was like
quick and timely, realistically and a very

467
00:32:01,046 --> 00:32:09,159
methodical approach to what was actually reasonable done rather than hiding behind, you
know, secrecy for for months on end and saying, well, you know, something did happen

468
00:32:09,159 --> 00:32:13,342
because someone else reported it from a security agency or a, you know, a security tester.

469
00:32:13,342 --> 00:32:21,526
Uh no, I always just think it's interesting what happens and I I'm not sure what the
future will will be here, especially because a lot of different package managers are

470
00:32:21,526 --> 00:32:23,767
attempting to take different precautions in place.

471
00:32:23,767 --> 00:32:31,080
And I'm hoping still that I can get someone on the record who knows a lot about uh package
manager security on the on the

472
00:32:31,080 --> 00:32:31,660
on the show.

473
00:32:31,660 --> 00:32:33,271
Uh I'm just waiting for the right person.

474
00:32:33,271 --> 00:32:38,424
Uh so if you think that's you, you know, please come and ping and ping me.

475
00:32:39,225 --> 00:32:41,046
and uh we'll we'll get you on.

476
00:32:41,046 --> 00:32:51,613
What I do want to still quickly jump into is you did mention the challenge with some of
the other hyperscalers basically utilizing the solution.

477
00:32:51,613 --> 00:33:00,618
And we have seen in the last few years or so Grafana actually partnering directly with
them uh to offer a managed version at a cost reasonably

478
00:33:00,618 --> 00:33:09,859
And I'm wondering whether or not, given the current world views, whether or not there'll
also be similar partnerships we should expect to pop up for other similar cloud providers,

479
00:33:09,859 --> 00:33:11,094
especially ones in Europe.

480
00:33:11,094 --> 00:33:14,438
to be, you know, everywhere and available for everyone so that they're familiar with it.

481
00:33:14,438 --> 00:33:17,181
Um one, because it's great technology and why wouldn't you want to use it?

482
00:33:17,181 --> 00:33:19,464
Um, you know, but also from a business.

483
00:33:20,626 --> 00:33:25,523
you know it's great for us to, you know, for people to be familiar with our technology and
and our tool set, right?

484
00:33:25,523 --> 00:33:26,153
for sure.

485
00:33:26,153 --> 00:33:28,545
And so yeah, we certainly did the the partnerships.

486
00:33:28,545 --> 00:33:29,727
You know, Amazon was the first one.

487
00:33:29,727 --> 00:33:30,764
Um

488
00:33:30,764 --> 00:33:35,337
where you know we license our technology so they could deliver the the Amazon managed
Grafana.

489
00:33:35,368 --> 00:33:41,964
we obviously did similar uh partnership with uh with Microsoft you know the Azure platform
as well as Azure managed Grafana.

490
00:33:41,964 --> 00:33:43,195
We're always looking at other partners.

491
00:33:43,195 --> 00:33:50,501
You know what's what we're finding interesting is that you know certainly for a lot of our
users, you know, Grafana Cloud is going to be their preferred method uh just because

492
00:33:50,501 --> 00:33:52,963
that's where there's much richer capabilities, right?

493
00:33:52,963 --> 00:33:54,534
And it is a a more

494
00:33:54,652 --> 00:33:57,845
kind of out of the box experience and kind of solve sort of the observability problems.

495
00:33:57,845 --> 00:34:01,515
And we certainly, you know, have built a solution that supports all the different cloud
providers, right?

496
00:34:01,515 --> 00:34:04,421
So it doesn't matter who you're running, it'll be, you know, it'll work for you.

497
00:34:04,421 --> 00:34:06,232
And we see many customers that are multi cloud, right?

498
00:34:06,232 --> 00:34:09,165
That are are in different providers and and are using different systems.

499
00:34:09,165 --> 00:34:13,339
And so they need, you know, an observability product that can work across all of those,
right?

500
00:34:13,339 --> 00:34:15,801
And be able to correlate across, you know, all the different providers.

501
00:34:15,801 --> 00:34:17,326
So um yeah.

502
00:34:17,326 --> 00:34:18,957
It's hard it's hard it's hard to do it any other way.

503
00:34:18,957 --> 00:34:25,233
I mean otherwise you you build it and no users come or you know you waste a lot of money
on on something that doesn't make s a lot of sense for sure.

504
00:34:25,233 --> 00:34:31,318
I think at this point it probably would be a good opportunity to switch over to picks for
the episode.

505
00:34:31,318 --> 00:34:35,852
So uh Anthony, what did you bring for the audience today?

506
00:34:36,575 --> 00:34:37,365
Yeah, yeah.

507
00:34:37,365 --> 00:34:37,766
So that's great.

508
00:34:37,766 --> 00:34:42,878
Well, so um obviously for me, yeah, it was a bit of a challenge, you know, like what uh
what am I interested in?

509
00:34:42,878 --> 00:34:46,669
Um, you know, the reality is Grafana Labs, that's my my passion.

510
00:34:46,669 --> 00:34:50,181
Um, but um we've already spoken enough about that.

511
00:34:50,181 --> 00:34:55,443
but one thing that I thought was uh really timely was uh a an essay that was brought to my
attention.

512
00:34:55,443 --> 00:35:01,016
So Raj, our CEO, actually brought it up last week when we were talking, which is um it's
called The Bitter Lesson, right?

513
00:35:01,016 --> 00:35:06,238
So it was put together by by Rich Sutton, right, AI researcher out of Google and

514
00:35:06,538 --> 00:35:09,869
it's really a a very uh an interesting read, timely read.

515
00:35:09,869 --> 00:35:12,780
It was written a long time ago, back in 2019.

516
00:35:12,780 --> 00:35:22,983
But it's really a kind of a focus around, you know, the looking back at the you know
evolution of AI over the last, you know, 70 years, right?

517
00:35:22,983 --> 00:35:29,934
For as long as we've been doing this and kind of understanding where um you know what what
we're seeing playing out, right?

518
00:35:29,934 --> 00:35:36,186
Where the reality is is that um, you know, trying to build more kind of domain specific

519
00:35:36,186 --> 00:35:47,191
um you know tools, uh they'll they'll always get uh beaten by generic uh tools where you
are just throwing larger and larger volumes of data at larger and larger volumes of

520
00:35:47,191 --> 00:35:47,851
compute.

521
00:35:47,851 --> 00:35:48,441
Um right.

522
00:35:48,441 --> 00:35:51,252
So we've seen this with the rise of LLMs, right?

523
00:35:51,493 --> 00:35:59,076
you know for years everyone was trying to build you know domain-specific um kind of AI
tools, whether it was in you know whatever say the medical field, right?

524
00:35:59,076 --> 00:35:59,846
Um right.

525
00:35:59,846 --> 00:36:03,540
But now we're seeing that the large language models, right, where you just throw

526
00:36:03,540 --> 00:36:12,975
all of the data at it, right, and you just process it with more and more compute power,
you end up getting better results um, you know, than these uh these unique tools, right?

527
00:36:12,975 --> 00:36:16,357
the good examples that it brings up is around um like even chess, right?

528
00:36:16,357 --> 00:36:23,931
Like where, you know, started off they'll, you know, encoding chess strategies, right,
into, you know, these I tools to make it think like a person.

529
00:36:23,931 --> 00:36:31,365
Um, but then the way they made the most successful ones is where they just said, actually
let's just build big neural network and just have you play chess against each other as

530
00:36:31,365 --> 00:36:33,196
many times as possible and you'll just

531
00:36:33,196 --> 00:36:35,528
you know, with the volumes of data, you'll end up being better.

532
00:36:35,528 --> 00:36:38,550
Um and that's, you know, that's the world that we're living in now.

533
00:36:38,550 --> 00:36:46,727
There's companies, right, you know, that have just disappeared overnight because they
built these, you know, domain specific AI technologies and now, you know, these generative

534
00:36:46,727 --> 00:36:50,000
AI tools have just come out and just completely just, you know, overrun them.

535
00:36:50,000 --> 00:36:55,114
Um and it's always going to be the case where, you know, right now we talked about earlier
today, right?

536
00:36:55,114 --> 00:36:59,868
Where, you know, sometimes building those more domain specific kind of things works for
some things, right?

537
00:36:59,868 --> 00:37:01,229
For a while, right?

538
00:37:01,229 --> 00:37:03,074
Um, but then what's happening is then

539
00:37:03,074 --> 00:37:08,511
the larger models are then getting better at it, uh, you know, inevitably over time
anyway.

540
00:37:08,511 --> 00:37:11,243
And so, you know, we're always having to constantly kind of iterate.

541
00:37:11,243 --> 00:37:16,022
And so I think it was really good um, you know, kind of read to go through that and kind
of reflect on that.

542
00:37:16,022 --> 00:37:16,202
Right.

543
00:37:16,202 --> 00:37:21,764
The original deep mind approach that solved chess, I guess, realistically, wasn't wasn't
trained just on chess.

544
00:37:21,764 --> 00:37:24,484
It was completely unsupervised training there.

545
00:37:24,484 --> 00:37:33,917
I do think there's an interesting perspective here, which is that realistically, the
reason that companies are getting overtaken by the I hate to call it Gen AI solutions

546
00:37:33,917 --> 00:37:41,689
instead, is because they didn't build or innovate on their space of having an LLM model
that solves that problem.

547
00:37:41,689 --> 00:37:44,070
They just added an LLM to their space.

548
00:37:44,070 --> 00:37:45,442
And so you

549
00:37:45,442 --> 00:37:52,694
They basically doubled down on having the best LLM, but for that area and they weren't
doing anything to d to get around it.

550
00:37:52,694 --> 00:37:57,536
And I think there's a question of like where is the end of innovation for LM design?

551
00:37:57,536 --> 00:38:01,307
And a lot of people are saying, yeah, it happened in Opus four point six or or whatever.

552
00:38:01,307 --> 00:38:06,999
And as long as you can beat that on a benchmark in your particular area, maybe we'll come
back to this particular space.

553
00:38:06,999 --> 00:38:10,720
But unless you're gonna innovate and buy the cutting line.

554
00:38:11,024 --> 00:38:18,766
GPUs to do the model training or be able to fine-tune existing models out there, I do
think a lot of those companies are going to fail just because it's not necessarily that

555
00:38:18,766 --> 00:38:19,437
the models are better.

556
00:38:19,437 --> 00:38:29,119
It's just like you don't have the technology to train to the level of capacity um or I
reasoning that the companies that are dedicated to the actual hardware to work on.

557
00:38:29,119 --> 00:38:38,124
So I I I hope long long term we see an opportunity here where smaller models exist because
it will be cheaper to maintain them and maybe increase the accuracy.

558
00:38:38,124 --> 00:38:44,554
But I think we have to get much closer to wherever the end of innovation for the LLM
creation to actually apply.

559
00:38:44,554 --> 00:38:45,236
I agree.

560
00:38:45,236 --> 00:38:47,565
Uh so what did I bring?

561
00:38:47,565 --> 00:38:49,017
had to rack my brain over this.

562
00:38:49,017 --> 00:38:55,079
Uh since since I started running engineering teams, I kept getting into popular culture.

563
00:38:55,079 --> 00:38:57,159
How good is this leader on screen?

564
00:38:57,159 --> 00:39:05,702
So think of your favorite character and your favorite television show or movie and see how
they actually perform leadership qualities in the gambit.

565
00:39:05,702 --> 00:39:06,112
And

566
00:39:06,112 --> 00:39:07,203
I can't look away now.

567
00:39:07,203 --> 00:39:11,887
Like every single time I see a group of people on the screen, I'm questioning like, who is
the leader?

568
00:39:11,887 --> 00:39:13,028
How are they working?

569
00:39:13,028 --> 00:39:14,088
Is this even a good leader?

570
00:39:14,088 --> 00:39:18,022
And you go back and you'll watch some old stuff and you're like, wow, this is not a team.

571
00:39:18,022 --> 00:39:22,284
This is just four random individuals with no organizational structure whatsoever.

572
00:39:22,284 --> 00:39:25,147
And the person who's called leader is just doing the worst job ever.

573
00:39:25,147 --> 00:39:28,240
Uh, I'm sorry if I broke anyone with this realization.

574
00:39:28,240 --> 00:39:31,883
Uh and so over time I'm just like, well, where is the best leader?

575
00:39:31,883 --> 00:39:36,098
The the one, the character that seems like it's the best emulation of a leader.

576
00:39:36,098 --> 00:39:45,039
And it makes sense because uh I think leaders aren't uh consulted on the set, like you get
medical doctors or military personnel or engineers when they want to design something or a

577
00:39:45,039 --> 00:39:45,621
process.

578
00:39:45,621 --> 00:39:52,342
But how often do you see like reference consultant for leadership on the set of some show
before they're like, Would a leader even say this?

579
00:39:52,342 --> 00:40:00,184
Uh and so my pig my my big is gonna be the best leader that I've ever seen on the show,
which is Captain Pike of Star Trek.

580
00:40:01,165 --> 00:40:05,246
and it it's not the best Star Trek show, but it's definitely my favorite leader.

581
00:40:05,246 --> 00:40:05,880
Excellent.

582
00:40:05,880 --> 00:40:07,984
Yeah, I mean I think it's uh it's important, right?

583
00:40:07,984 --> 00:40:11,140
But if you want to be a manager, right, if you it's a very different skill, right?

584
00:40:11,140 --> 00:40:15,478
It's a very different you know, problem set, but it's a a very, very, very different role.

585
00:40:15,478 --> 00:40:18,132
Uh very different set of problems.

586
00:40:18,540 --> 00:40:26,259
I I think just the single point of failure for someone doing everything in the highest
leadership bot in an organization is just so problematic.

587
00:40:26,259 --> 00:40:32,787
But anyway, so this has ruined uh so many television shows for me that, you know, I'm
watching it, I'm just like, This is so unrealistic.

588
00:40:32,787 --> 00:40:35,450
This it doesn't represent reality in any way.

589
00:40:35,450 --> 00:40:38,488
Uh I guess my other theory uh or perhaps it

590
00:40:38,488 --> 00:40:38,918
That's right.

591
00:40:38,918 --> 00:40:43,212
It's just they just they could be a terrible leader and that's there's plenty of those
that exist, right?

592
00:40:43,620 --> 00:40:44,931
yeah, yeah, for sure.

593
00:40:44,931 --> 00:40:47,672
I I don't remember what it was, and I probably should find a reference to this.

594
00:40:47,672 --> 00:40:55,254
A long time ago I read this thing like if aliens came to Earth, what would be the thing
about humans that would be most surprising or shocking for them?

595
00:40:55,254 --> 00:40:58,115
And for for me, you know, it just goes along this angle.

596
00:40:58,115 --> 00:41:05,458
I think it's the uh great attention spent in the last a hundred years on how to manage
humans effectively.

597
00:41:06,082 --> 00:41:08,725
But that's that's a whole different philosophical topic.

598
00:41:08,725 --> 00:41:14,470
So um in this episode, we we we actually reference a lot about productivity and we have a
separate episode dedicated to productivity.

599
00:41:14,470 --> 00:41:21,537
So if anyone is actually interested in that, that will be linked in the description, along
with anything else that we've we've talked about, and there was quite a few number of

600
00:41:21,537 --> 00:41:22,097
things.

601
00:41:22,097 --> 00:41:24,739
So thank you, Anthony, for joining us today.

602
00:41:25,400 --> 00:41:27,682
it's been honestly a great episode.

603
00:41:28,824 --> 00:41:31,648
And here's a short reminder to everyone who's listening to Click.

604
00:41:31,648 --> 00:41:35,062
subscribe uh to the Adventures in DevOps or leave a comment in the episode.

605
00:41:35,062 --> 00:41:36,804
You have no idea what that means to me.

606
00:41:36,804 --> 00:41:43,051
I if you want me to make the right content and get the right guess on, make a suggestion
or, you know, say what you like or don't like.

607
00:41:43,051 --> 00:41:46,304
And I hope to see everyone again back next.