1
00:00:07,874 --> 00:00:09,715
Welcome back to Adventures in DevOps.

2
00:00:09,715 --> 00:00:12,397
Every episode is a deep dive with an expert guest.

3
00:00:12,397 --> 00:00:17,320
Today's adventure focuses on heterogeneous compute and auto scaling for data pipelines.

4
00:00:17,320 --> 00:00:23,903
The expert had a huge impact on Android early days at Google and is now currently the head
of engineering at any scale.

5
00:00:23,903 --> 00:00:25,005
Welcome to the show, J.

6
00:00:25,005 --> 00:00:25,896
Kumar, J.K.

7
00:00:25,896 --> 00:00:26,540
Ganesh.

8
00:00:26,540 --> 00:00:27,957
So excited to have this conversation.

9
00:00:27,957 --> 00:00:29,206
Thanks for the warm intro, Arun.

10
00:00:29,206 --> 00:00:35,762
You have a different paradigm for how you need to execute a particular program or code.

11
00:00:35,762 --> 00:00:45,320
it's not like, historically we can say containers are terrible as the unit of a worker or
job that needs to be done, which is realistically what Kubernetes is using.

12
00:00:45,320 --> 00:00:49,885
And in especially the ML space, it may not map one to one with what should be optimized.

13
00:00:49,885 --> 00:00:51,660
So we need a different paradigm.

14
00:00:51,660 --> 00:00:57,456
you have fast iteration loops, training loops, reinforcement learning loops, you need much
more finer grain control.

15
00:00:57,456 --> 00:00:59,422
And that's how Ray came to be.

16
00:00:59,422 --> 00:01:05,986
One interesting aspect is it does seem similar to the idea of having serverless running on
top of container orchestrators.

17
00:01:05,986 --> 00:01:17,122
So just except for the ML world and realistically the interface isn't the terrible OCI
standard we have for deciding how to interact with specifically how much virtual memory

18
00:01:17,122 --> 00:01:21,834
and how big the containers are, et cetera, and having to depend on everything and be
specific about it.

19
00:01:21,834 --> 00:01:28,630
But really, if you look at AWS's Lambda or GCP, you know, cloud run functions, you are
defining like memory usage.

20
00:01:28,630 --> 00:01:30,762
and size maybe, essentially.

21
00:01:30,762 --> 00:01:38,951
And I can understand that there are some things that you would have to configure for Ray,
but the end goal realistically is to remove that complexity from someone who's gonna be

22
00:01:38,951 --> 00:01:40,034
running an ML job.

23
00:01:40,034 --> 00:01:41,236
Yes, you're spot on.

24
00:01:41,236 --> 00:01:43,700
There's also some complexity as to where the data lives.

25
00:01:43,700 --> 00:01:51,822
So there are some people who are fine with serverless and there are some who says, no, no,
no, I have my own AWS or an ABS contract.

26
00:01:51,822 --> 00:01:53,915
So I want you to run in my environment.

27
00:01:53,915 --> 00:01:56,046
have seen a lot of customers do that.

28
00:01:56,046 --> 00:01:59,726
Well, I think Lambda was like, I'm going to get this number wrong, but I think it was
2012.

29
00:01:59,826 --> 00:02:08,746
And now we're looking at 14 years later, they're finally potentially releasing
improvements on top of that, that may actually give you fine-grained control.

30
00:02:08,746 --> 00:02:17,826
So I think one of the aspects here is, I think you're totally right, that at the end of
the day, you need to not only control the underlying layer wherever you're running, it

31
00:02:17,826 --> 00:02:24,454
doesn't matter if you're running Kubernetes on-prem or on a cloud provider, it's not the
right interface for...

32
00:02:24,643 --> 00:02:28,472
distributing control over that to individual workloads that are being run.

33
00:02:28,472 --> 00:02:31,143
And so you're installing a platform on top.

34
00:02:31,143 --> 00:02:38,002
At the end of the day, most developers honestly care about time to market, how quickly can
they get the solution and what is the efficient.

35
00:02:38,729 --> 00:02:41,512
Well, I don't actually know if all developers care about them.

36
00:02:41,512 --> 00:02:45,023
Well the business definitely cares about it and so it passes on to them

37
00:02:45,775 --> 00:02:54,711
Yeah, no, I mean, I totally agree that it's definitely 100 % of business concern and it's
better when ML developers understand or also engineers really understand what their

38
00:02:54,711 --> 00:03:01,956
constraints are for what they're creating because they're the ones at the end of the day
who have to make the determination of what technology to pull into their stack and solve

39
00:03:01,956 --> 00:03:05,609
their specific problems to handle the business needs.

40
00:03:05,609 --> 00:03:13,622
I think in practice, we see a lot of companies historically have just given an unlimited
budget to quote unquote data science teams and

41
00:03:13,622 --> 00:03:21,366
let them go wild and the result is a complete mess of Python packages and code that no
software engineer would ever approve of.

42
00:03:21,366 --> 00:03:23,227
Yeah, yeah, true, that's true.

43
00:03:23,227 --> 00:03:30,423
That used to happen a lot more maybe seven, eight years back and then, know, the software
engineering practices have come to the ML world and many of the software engineers have

44
00:03:30,423 --> 00:03:31,283
become ML engineers.

45
00:03:31,283 --> 00:03:34,836
So you've actually seen that change a lot too when models are checked into Git.

46
00:03:34,836 --> 00:03:37,368
I remember seven, eight years back, people were not checking in models.

47
00:03:37,368 --> 00:03:38,779
They were like, this is an iteration.

48
00:03:38,779 --> 00:03:39,759
Here's my Google doc.

49
00:03:39,759 --> 00:03:47,435
And no one else can reproduce the results because it's like, you know, although the
features used were not documented well.

50
00:03:47,435 --> 00:03:49,366
So that has changed quite a bit.

51
00:03:49,586 --> 00:03:57,370
There were so many horrifying stories I have from my past, like when I worked at one
aerospace company, they for sure were putting their source control, wrapping it up,

52
00:03:57,370 --> 00:04:03,874
whatever the files were, into a zip archive and uploading it to Confluence as their source
control with like the version there.

53
00:04:03,874 --> 00:04:05,564
that's actually, that wasn't even that long ago.

54
00:04:05,564 --> 00:04:09,947
And that was pure software engineering, but I know where those engineers came from.

55
00:04:09,947 --> 00:04:10,918
So it makes a lot of sense.

56
00:04:10,918 --> 00:04:14,620
I do have to ask, who is building models today still?

57
00:04:14,620 --> 00:04:19,315
Like I do see like there is this aspect of a fixed small number of companies that are
making

58
00:04:19,315 --> 00:04:23,247
the state-of-the-art models, we'll call them foundational models.

59
00:04:23,247 --> 00:04:24,758
And it seems like a small set.

60
00:04:24,758 --> 00:04:29,431
And there was a little bit of time where this idea of fine-tuning could be considered.

61
00:04:29,431 --> 00:04:34,674
But realistically, I feel like that's been eliminated, unfortunately, in a way, because
the fine-tuning is expensive.

62
00:04:34,674 --> 00:04:40,077
And it also couldn't keep up with the improvements to models that were being created by
larger companies.

63
00:04:40,077 --> 00:04:47,734
So I think my question really is, what specific problems do these companies have that
they're turning to Ray in any scale for a solution?

64
00:04:47,734 --> 00:04:51,857
You have your Anthropics and OpenAI's and your Google's creating the foundation models.

65
00:04:51,857 --> 00:05:00,802
And then there are certain other companies which create the next year of foundation models
for their specific vertical, whether it was health, whether it was finance, et cetera.

66
00:05:00,802 --> 00:05:04,714
And yes, fine tuning was and rag based systems were popular.

67
00:05:04,714 --> 00:05:05,504
Rag is still popular.

68
00:05:05,504 --> 00:05:06,755
Fine tuning has gone down a bit.

69
00:05:06,755 --> 00:05:14,850
And like, you know, now with agents, text engineering is more key that you actually
provide the right context of your internal system to the models, right?

70
00:05:15,032 --> 00:05:21,747
Where we are seeing a lot of users is like, as many times, it's easy to get started with
your OpenAI model um when you're a startup.

71
00:05:21,747 --> 00:05:29,422
But as your business scales, or if you're already a scaled digital native business, you
are not wanting to be fully dependent on Anthropic and OpenAI.

72
00:05:29,422 --> 00:05:31,062
You have your data stream sets there.

73
00:05:31,062 --> 00:05:36,737
So what we are seeing is that people are reading large amounts of data and then creating
embeddings out of it.

74
00:05:36,737 --> 00:05:39,163
And then they want to serve those embeddings.

75
00:05:39,163 --> 00:05:44,510
These scenarios where I think in a lot of them it really depends on having a lot of data
how

76
00:05:44,510 --> 00:05:51,016
What is the go-to mechanism here for the capacity for storage and then interacting with
the platform?

77
00:05:51,016 --> 00:06:00,053
You see that a lot of these companies are running Kubernetes and they're running large, I
don't know, Postgres instances directly on their Kubernetes clusters and you're sourcing

78
00:06:00,053 --> 00:06:08,841
the data from there when they're executing or are they using some sort of cloud provider
and the data is being stored and that mechanism, are we seeing on-prem versus cloud

79
00:06:08,841 --> 00:06:09,421
instances?

80
00:06:09,421 --> 00:06:12,453
Where are most of the companies building their models today?

81
00:06:12,767 --> 00:06:15,119
storage, some with data breaks, some with snowflakes.

82
00:06:15,119 --> 00:06:21,196
Just read the data from wherever the data is stored and where the data does not leave
their environment over there.

83
00:06:21,196 --> 00:06:30,615
On-prem, yes, there are some on-prem customers, but then, you know, they're on-prem for a
reason and they want a lot more control.

84
00:06:30,615 --> 00:06:38,626
One thing I do struggle with a bit here is understanding specifically when to use the CPU
versus switch off to GPU.

85
00:06:38,626 --> 00:06:41,902
And I think you're the right person to answer this question for me.

86
00:06:41,902 --> 00:06:42,142
Yeah.

87
00:06:42,142 --> 00:06:43,562
So let's take an example.

88
00:06:43,562 --> 00:06:55,122
Suppose you have podcasts, okay, a bunch of podcasts and newsletters, and then you want to
have a simple search interface, which says, I am interested in a topic, say about growth

89
00:06:55,122 --> 00:07:02,342
hacking, and it should go pick the right newsletter, write your transcript, and take you
exactly to that point.

90
00:07:02,342 --> 00:07:06,402
So you have this corpus of data and you now want to read that data.

91
00:07:06,402 --> 00:07:08,582
Like usually the reading stuff happens on the CPU.

92
00:07:08,682 --> 00:07:11,458
Now you need to like, you know, create chunks of this data.

93
00:07:11,458 --> 00:07:15,141
And then you need to create an embedding model of this data.

94
00:07:15,141 --> 00:07:18,673
That is a GPU efficient process.

95
00:07:18,673 --> 00:07:20,044
You use GPUs for that.

96
00:07:20,044 --> 00:07:23,486
And then you want to write the results somewhere, which is, again, a CPU process.

97
00:07:23,486 --> 00:07:35,054
So your standard pipeline is like CPU, GPUs, like two GPUs, one for, say, chunking,
another GPU for segmentation or an embedding model, and then another CPU for actually

98
00:07:35,054 --> 00:07:35,964
writing it.

99
00:07:36,054 --> 00:07:39,315
And once you have written this results, you need to serve these results.

100
00:07:39,315 --> 00:07:42,206
And now for serving the results, again, you need to say either read it.

101
00:07:42,206 --> 00:07:44,646
If it's a two-stage pipeline, then you read it back from the disk.

102
00:07:44,646 --> 00:07:47,297
You can obviously stream it too, but let's make it simple.

103
00:07:47,297 --> 00:07:49,438
And you're reading it back from the CPU.

104
00:07:49,438 --> 00:07:51,658
And now you need to use an LLM inference provider.

105
00:07:51,658 --> 00:07:53,299
That's also another GPU.

106
00:07:53,299 --> 00:08:00,341
And so you need to have a GPU for the LLM inference provider so that you can now serve it
on a page and people can type queries and you can stream in the results.

107
00:08:00,341 --> 00:08:04,716
Usually what happens in such pipelines is that when you're reading images,

108
00:08:04,716 --> 00:08:12,339
your GPU is just sitting there waiting for all these videos, images, newsletters to
actually be read.

109
00:08:12,339 --> 00:08:17,341
And if you have a massive corpus, that's a good amount of time it's actually waiting it
for, right?

110
00:08:17,341 --> 00:08:21,413
you need to keep running this pipeline again and So as the first image is read, it is sent
to the GPU.

111
00:08:21,413 --> 00:08:25,074
And now that GPU has already done the chunks and then it's sent to the second GPU for the
embedding model.

112
00:08:25,074 --> 00:08:26,555
And then it's written to the CPU.

113
00:08:26,555 --> 00:08:31,147
And then you can keep reading the next set of data, the next set of data kind of stuff.

114
00:08:31,147 --> 00:08:33,837
So overall utilization of this

115
00:08:33,837 --> 00:08:35,522
pipeline is much more efficient.

116
00:08:35,522 --> 00:08:41,006
the GPUs are utilization is higher, the time it takes for the entire processing is much
more reduced.

117
00:08:41,006 --> 00:08:52,712
How do you know that some code that's being executed is the inference or some code that's
executing is going to be image generation so that scheduling it on a GPU-specific

118
00:08:52,712 --> 00:08:59,825
container that has access to the GPUs from the underlying Kubernetes cluster is being used
versus one that's optimized for compute?

119
00:08:59,825 --> 00:09:05,778
Maybe the underlying containers are all the same, but the real trouble is you have to
know...

120
00:09:05,778 --> 00:09:08,839
What you're doing is you're figuring out what capacity is still available.

121
00:09:08,839 --> 00:09:10,039
Let's schedule there.

122
00:09:11,724 --> 00:09:15,010
program is deciding what is actually necessary.

123
00:09:15,010 --> 00:09:19,454
This is where a little bit of the ML developers work comes in.

124
00:09:19,454 --> 00:09:23,757
If they say, hey, yes, you can specify some computer, say it says use GPUs.

125
00:09:23,757 --> 00:09:25,599
So I don't care about the numbers.

126
00:09:25,599 --> 00:09:27,871
I know this requires GPUs, use GPUs.

127
00:09:27,871 --> 00:09:30,103
And rest is taken care of.

128
00:09:30,103 --> 00:09:34,547
Now, if the system only provides CPUs, then they can't do anything.

129
00:09:34,547 --> 00:09:38,822
So we do need some amount of hint from the user saying, hey, use GPUs kind of stuff.

130
00:09:38,822 --> 00:09:39,423
It makes sense.

131
00:09:39,423 --> 00:09:47,574
The only alternative I can think of is that somehow you would be collecting the actual
code that would be executed, hashing it, storing that hash and seeing how it dynamically

132
00:09:47,574 --> 00:09:51,238
performs under different utilization curves, like how many GPUs.

133
00:09:51,263 --> 00:09:51,634
Correct.

134
00:09:51,634 --> 00:09:54,698
So we can figure out, the CPU is really busy right now.

135
00:09:54,698 --> 00:09:56,522
Then we should add a GPU to the system, right?

136
00:09:56,522 --> 00:09:59,658
So we can actually do all those interesting things.

137
00:09:59,658 --> 00:10:01,291
And some of that is in the work.

138
00:10:01,291 --> 00:10:04,162
Some of that is in our plans to actually continue doing that.

139
00:10:04,162 --> 00:10:05,023
Yeah, for sure.

140
00:10:05,023 --> 00:10:15,077
So my one question I have here is that my 20 years ago knowledge said that if I needed
serialized work, use a CPU and if it's very parallelizable, use a GPU.

141
00:10:15,077 --> 00:10:22,851
But obviously with hyper threading and multiple cores, you know, that statement went out
the window and I actually don't I haven't followed the GPU architecture in a long time.

142
00:10:22,851 --> 00:10:30,592
So is this still an accurate statement or are there specific things that GPUs have been
optimized for to be actually be able to handle in some specific way?

143
00:10:30,592 --> 00:10:35,847
Yeah, I think in the ML world, for the transformer models, GPUs have been optimized for a
lot.

144
00:10:35,847 --> 00:10:49,018
And you know, you keep getting new generations of GPUs, GP to GPU transfer, speed has
increased and you know, your interconnects are becoming a lot more efficient so that the

145
00:10:49,018 --> 00:10:52,001
nodes transfers doesn't have to go through CPUs.

146
00:10:52,001 --> 00:10:58,496
So in the last three, four years, lot of the GPU architecture is focused on making weight
transfer efficient.

147
00:10:58,540 --> 00:11:05,153
the ML world, post-training world, et cetera, are much more efficient, which is probably
not the case during the graphics time when GPUs started and then they started going for

148
00:11:05,153 --> 00:11:06,956
the crypto world and now it's for the ML world.

149
00:11:06,956 --> 00:11:12,401
Well, there was this strategy with graphics where it's, you know, maybe you can just
render this part of the screen.

150
00:11:12,401 --> 00:11:14,202
And so it's very easy to break that down.

151
00:11:14,202 --> 00:11:23,149
And for the crypto world, of course, it doesn't matter if you have any sort of alignment
on what's being processed, because realistically, it was all random in the proof of work

152
00:11:23,149 --> 00:11:28,775
world, pull a random number from who cares, and then calculate and see if it's a useful
result.

153
00:11:28,775 --> 00:11:30,348
And if so, you know, great.

154
00:11:30,348 --> 00:11:34,631
You don't have any coordination required there, but now we're definitely at the point
where it is required.

155
00:11:34,631 --> 00:11:42,197
So it is an interesting insight basically that one of the things that has been
significantly improved in the last few years is the ability to scale up the coordination

156
00:11:42,197 --> 00:11:45,098
between individual GPUs, which I'm totally with you.

157
00:11:45,098 --> 00:11:54,347
Who needs more than a couple of them max for personal usages, but for commercial
strategies, stacking them in parallel, they need to collaborate in some way.

158
00:11:54,347 --> 00:11:56,214
So the old style CPU...

159
00:11:56,214 --> 00:11:58,515
or memory aspect was like a Beowulf cluster.

160
00:11:58,515 --> 00:12:06,910
Now, obviously we need something much more complex and having the companies who make the
technology actually care about this use case means that they are investing in actually

161
00:12:06,910 --> 00:12:07,530
trying to improve it.

162
00:12:07,530 --> 00:12:13,174
that's an interesting insight that I wouldn't have guessed how they're actually improving.

163
00:12:13,174 --> 00:12:24,674
One thing that comes to mind with the CPU GPU breakdown is I feel like there has been this
hypothetical that will break through this unnecessary aspect where we have

164
00:12:24,674 --> 00:12:31,074
two completely separate pieces of technology which sort of do similar things and are both
useful in computing.

165
00:12:31,074 --> 00:12:35,720
I remember there being lots of releases like, no, you will only ever need a GPU going
forward.

166
00:12:35,720 --> 00:12:46,865
Are we missing a hardware primitive that can do everything in a way where I know there's a
lot of talk of these hypothetical AI chips, but I don't think it's actually a thing that

167
00:12:46,865 --> 00:12:48,598
really exists so much as.

168
00:12:48,598 --> 00:12:57,042
we understand different requirements for how the hardware needs to process things and have
dedicated logical units for actually executing those specific areas.

169
00:12:57,042 --> 00:13:04,145
Maybe it's a, instance, in the cryptography world, performing this particular hash or
signature function is optimized all the way at the CPU level.

170
00:13:04,145 --> 00:13:06,476
So my question is, where do you see this going?

171
00:13:06,476 --> 00:13:07,660
Is it that...

172
00:13:07,660 --> 00:13:15,037
we are going to keep on living in this world where some companies produce CPUs, some
companies produce GPUs, and everyone has to pay a ton of money for both of these things,

173
00:13:15,037 --> 00:13:24,147
or is some company going to come out there and be like, you only need one of these things,
and it's not just a secret CPU plus GPU combination, but really something that is

174
00:13:24,147 --> 00:13:27,330
innovation when it comes to the hardware in computers.

175
00:13:27,330 --> 00:13:35,997
I think Nvidia just announced it recently called the Vera chips where, you know, the CPUs
and GPUs are in a single chip kind of stuff.

176
00:13:35,997 --> 00:13:40,731
I see it's more in that frame of reference where CPU, GPU get integrated.

177
00:13:40,731 --> 00:13:47,027
I see some parallels to like, you know, what used to happen, say in the Bluetooth and the
wifi world on mobile phones.

178
00:13:47,027 --> 00:13:48,808
Bluetooth was a separate chip.

179
00:13:48,808 --> 00:13:52,491
Wi-Fi was a separate chip and GPS was a separate chip.

180
00:13:52,491 --> 00:13:55,928
And then Broadcom and Qualcomm started integrating it all in a single chip.

181
00:13:55,928 --> 00:13:57,580
but each one had their own use case.

182
00:13:57,580 --> 00:14:01,083
That started causing problems because Bluetooth and Wi-Fi was on the same frequency.

183
00:14:01,083 --> 00:14:09,620
And so many times when you're in a Wi-Fi, just getting off your car and your phone
connects to the home Wi-Fi, your audio in your car will just drop off for a second because

184
00:14:09,620 --> 00:14:10,491
of the interference.

185
00:14:10,491 --> 00:14:14,113
So there is the chip to chip connectivity that is there.

186
00:14:14,954 --> 00:14:19,392
in these newer systems, the CPU and GPU combined integrated chips kind of stuff, right?

187
00:14:19,392 --> 00:14:22,948
So you will actually get much more higher performance.

188
00:14:22,948 --> 00:14:27,970
Though I do think the bigger problem that's coming up is just the shortage of electricity
for these data centers.

189
00:14:27,970 --> 00:14:40,679
hope this is a turnaround for green energy production strategies, fission reactors,
reinvigorated use and building up fusion reactors, given we know how bad wind is and

190
00:14:40,679 --> 00:14:41,100
solar.

191
00:14:41,100 --> 00:14:44,683
I actually think they're just going to keep on burning more gas.

192
00:14:44,683 --> 00:14:52,568
We talked with a couple of data center owners in the past and they told us there's lots of
reservations already still available for energy.

193
00:14:52,568 --> 00:14:54,880
think the goal of getting cheaper

194
00:14:54,880 --> 00:14:57,742
energy is just a ploy like, we're going to run out.

195
00:14:57,742 --> 00:14:59,774
You have to make it cheaper for us.

196
00:14:59,774 --> 00:15:01,756
Or those things aren't connected for me.

197
00:15:01,756 --> 00:15:11,463
I mean, I can totally believe at some point it's going to be an issue, but the way I see
it, and maybe it's super pessimistic, us lowly individual humans still have lights on in

198
00:15:11,463 --> 00:15:16,488
our homes and electricity working for our refrigerators and dishwashers, assuming you have
those.

199
00:15:16,488 --> 00:15:22,672
until that electricity has been commandeered by the hyperscalers, there's still, there's
still available capacity.

200
00:15:23,333 --> 00:15:24,652
I don't understand.

201
00:15:24,652 --> 00:15:26,793
and maybe this is my lax knowledge here.

202
00:15:26,793 --> 00:15:29,575
GPUs, why are they so much more expensive than CPUs?

203
00:15:29,575 --> 00:15:33,555
And maybe that's just not even a true statement, but that's how my understanding is.

204
00:15:33,555 --> 00:15:42,094
I'm like, I've seen the insides of the clean rooms for manufacturing CPUs and they always
seem like the technology is quite amazing.

205
00:15:42,094 --> 00:15:45,426
It's very, you know, precision manufacturing and everything.

206
00:15:45,426 --> 00:15:49,779
And I don't remember the last time I saw a video of the actual manufacturing of GPUs.

207
00:15:49,779 --> 00:15:54,506
And maybe it's because it's a closely guarded secret that, you know, Radeon and ATI and

208
00:15:54,506 --> 00:15:59,470
Nvidia have been keeping and they just haven't shared or maybe it is quite more
spectacular.

209
00:15:59,470 --> 00:16:00,650
That's a good question.

210
00:16:00,910 --> 00:16:13,570
So one of the reasons GPUs are more expensive is just that the dies are much more larger
for a GPU than a CPU.

211
00:16:13,610 --> 00:16:15,810
I think it's at least larger.

212
00:16:16,430 --> 00:16:18,770
That makes it harder to manufacture.

213
00:16:18,770 --> 00:16:22,270
If there is a single defect, then there is a problem.

214
00:16:22,310 --> 00:16:25,550
And there's also the integration components, right?

215
00:16:25,550 --> 00:16:26,990
need, like if you have a

216
00:16:26,990 --> 00:16:32,430
GPU machine as a result of which you need specialized Vram, you need cooling for it.

217
00:16:32,430 --> 00:16:37,250
So you've got a bunch of these things that are components which are very specific to the
GPUs.

218
00:16:37,250 --> 00:16:41,418
This will be a question for someone who actually does these die manufacturing to see

219
00:16:41,418 --> 00:16:45,449
Hint, hint, wink, wink for anyone who has the answer to this question.

220
00:16:45,449 --> 00:16:53,421
Okay, no, I just like, I've always seen the die manufacturing on the wafers and they're,
you know, the dies are cut specifically and they're in small form factor.

221
00:16:53,421 --> 00:17:02,304
oh And I think one of the challenges here is I think it's a similar fundamental challenge
in the quantum computing space is you can't just make it fundamentally bigger.

222
00:17:02,304 --> 00:17:09,750
Like an innovation, there won't help a lot because you still have to get data from one
point of the chip to a different point of the chip and...

223
00:17:09,750 --> 00:17:12,642
Size, you can't just, so there are things that prevent it from being bigger.

224
00:17:12,642 --> 00:17:18,356
And maybe what we're talking about challenges, just to focus it back on maybe the space
that you're more of the expert in.

225
00:17:18,356 --> 00:17:19,196
I am sort of curious.

226
00:17:19,196 --> 00:17:22,599
So you've built up this, the platform and the open source libraries.

227
00:17:22,599 --> 00:17:27,744
There have got to be some fundamental challenges that you and your team have faced in
actually spinning up this.

228
00:17:27,744 --> 00:17:36,078
It's actually a challenging because many open source packages are libraries and we have
multiple libraries and the core of the platform.

229
00:17:36,078 --> 00:17:39,879
And so the core of the platform have to work on different compute paradigms.

230
00:17:39,879 --> 00:17:50,164
They have to work on Nvidia chips, ARM chips, Intel chips, Kubernetes, VMs, Azure
Kubernetes, Google version of their Kubernetes, AKS their version of their Kubernetes.

231
00:17:50,164 --> 00:17:52,625
And now all the core weaves and all those things.

232
00:17:52,625 --> 00:17:55,936
And now we talked about GPUs, each of the GPUs are different.

233
00:17:56,058 --> 00:17:57,879
characteristics kind of stuff, right?

234
00:17:57,879 --> 00:18:00,001
And that's just on the core infrastructure itself.

235
00:18:00,001 --> 00:18:08,867
Then you've got your Python libraries and you have to actually make sure it works on a
different CUDA version, it works on a different Python thing, it works with the right

236
00:18:08,867 --> 00:18:13,229
NumPy library, it works with the right TensorFlow library, it works with the right PyTorch
library, right?

237
00:18:13,229 --> 00:18:20,586
So just making this whole ecosystem of Python repackaging work is...

238
00:18:20,586 --> 00:18:29,260
Really a lot of work in open source just more than I think it's easy to release a package
Just making sure it works in every single case and keeps working in different versions in

239
00:18:29,260 --> 00:18:35,663
different environments There'll be some developer who will have some custom environment
will change something and it'll not work for them and they'll say hey This is broken and

240
00:18:35,663 --> 00:18:41,046
we like we don't know why and then we spend time investigating it and and there's contact
for general these Packages.

241
00:18:41,046 --> 00:18:45,458
So yes, the open source requires investment and honestly, it's as a startup.

242
00:18:45,458 --> 00:18:45,920
It's

243
00:18:45,920 --> 00:18:48,581
And it's not just that it doesn't work, right?

244
00:18:48,581 --> 00:18:51,132
Because that's almost like an easy problem to identify.

245
00:18:51,132 --> 00:18:55,531
Like someone's like, I'm using, you know, version one of this, two of that, three of that,
and it doesn't work.

246
00:18:55,531 --> 00:18:57,514
You're like, okay, I can reproduce that problem.

247
00:18:57,514 --> 00:19:04,226
But then I think the real challenge is like, oh, we're offering a optimization solution
for capacity utilization.

248
00:19:04,226 --> 00:19:06,497
So it's not just, it doesn't work on these.

249
00:19:06,637 --> 00:19:09,710
Set it's it works, but it's not as a.

250
00:19:09,710 --> 00:19:12,350
big of reduction as we want it to be.

251
00:19:12,350 --> 00:19:14,790
And then you're like, well, how big should it actually be here?

252
00:19:14,790 --> 00:19:24,810
So it sounds like you have to basically be continually performing benchmarks on the
combinatorial nature of permutations really of all the sets of different things that you

253
00:19:24,810 --> 00:19:25,854
could be utilizing.

254
00:19:25,854 --> 00:19:34,958
I mean, from a process, operational process perspective, a bunch of regular release
testing on various cloud providers, various combinations that's actually there.

255
00:19:34,958 --> 00:19:45,163
And purely from an operational perspective, the release team becomes extremely important
because that's the team which is like, know, which is responsible for making sure these

256
00:19:45,163 --> 00:19:49,495
packages keep continuing to work all the time in all these combinational stuff.

257
00:19:49,495 --> 00:19:52,800
So we have actually got a very strong release team, probably.

258
00:19:52,800 --> 00:19:54,651
in percentage wise, higher proportion.

259
00:19:54,651 --> 00:19:56,572
And so that is, that is one part of the puzzle.

260
00:19:56,572 --> 00:20:03,877
The second part of the puzzle is we are still a startup to be honest, so we cannot, we
need to make sure we are focused and so we work closely with the customers.

261
00:20:03,877 --> 00:20:10,915
And customers also have their own unique environment saying, Hey, this networking setup is
in my layer seven stuff I need to connect over here.

262
00:20:10,915 --> 00:20:12,992
Or my DNS is having this issue.

263
00:20:12,992 --> 00:20:16,084
So I need to go talk to my security team, my security team, what is this Ray?

264
00:20:16,084 --> 00:20:18,155
I don't care about you wanting to do X, Y, Z.

265
00:20:18,155 --> 00:20:19,686
This is my policy adhere to it.

266
00:20:19,686 --> 00:20:20,424
you know, so.

267
00:20:20,424 --> 00:20:21,434
no, I totally get it.

268
00:20:21,434 --> 00:20:28,886
think something that is left often unsaid is that the company that you want to build and
specifically the product you want to make is very dependent on the people you hire.

269
00:20:28,886 --> 00:20:29,817
That's true.

270
00:20:29,817 --> 00:20:38,479
If you have a particular mindset you're going after and it is about, I say, reliability or
consistency in what you're pushing out, it sounds like you have a very close attention on

271
00:20:38,479 --> 00:20:40,210
how your release team is being built.

272
00:20:40,210 --> 00:20:47,414
And so my question for you maybe is any special tricks or anything that you're
specifically looking at when you're hiring into the release team that you're looking for?

273
00:20:47,414 --> 00:20:47,804
Yeah.

274
00:20:47,804 --> 00:20:51,535
Honestly, it just depends upon someone who has a true passion for this work.

275
00:20:51,535 --> 00:20:54,876
It's very hard to find people who love this kind of work.

276
00:20:54,876 --> 00:20:57,317
And you need at least one or two such people.

277
00:20:57,317 --> 00:20:57,837
Yeah.

278
00:20:57,837 --> 00:21:00,138
To be fair, it's actually a lot of grand work.

279
00:21:00,138 --> 00:21:02,138
And so you need to have an eye for automation.

280
00:21:02,138 --> 00:21:12,611
And especially now with all the coding tools, some parts of it becomes easier, but you
have to understand the full space and say, Hey, what are the ways so we can actually

281
00:21:12,611 --> 00:21:13,461
automate these things?

282
00:21:13,461 --> 00:21:15,062
How should we actually release these things?

283
00:21:15,062 --> 00:21:16,022
All that kind of stuff.

284
00:21:16,022 --> 00:21:16,396
Right.

285
00:21:16,396 --> 00:21:19,707
because people who are not closely in the details don't see it.

286
00:21:19,707 --> 00:21:22,449
They don't see the complexity that's actually required.

287
00:21:22,449 --> 00:21:31,464
They don't see the one developer who is complaining um and they don't see like, know, Hey,
this particular test is failing or we cannot release it.

288
00:21:31,464 --> 00:21:34,095
They don't see the actual grunt work that's happening.

289
00:21:34,095 --> 00:21:34,395
Right.

290
00:21:34,395 --> 00:21:40,154
So I think this is where it becomes a work of the leader to make sure that work is
highlighted.

291
00:21:40,154 --> 00:21:48,058
My CEO frequently says, especially for these sorts of teams, when everything is going
right and the teams are working effectively, which they're hiding everything else from the

292
00:21:48,058 --> 00:21:52,222
business, basically from the outside, it seems like nothing's happening.

293
00:21:52,222 --> 00:22:01,799
And so you're tempted to make a change to make it more splashy or, you know, wouldn't it
be great if things failed once in a while so that the team can get recognition for the

294
00:22:01,799 --> 00:22:02,369
work they're doing?

295
00:22:02,369 --> 00:22:03,420
But I definitely agree.

296
00:22:03,420 --> 00:22:10,048
It is a huge challenge there because to find those people that want to do that work, I
mean, you called it maybe

297
00:22:10,048 --> 00:22:17,703
grunt work or maybe thankless, but I'll say with 8 billion people in the world
statistically, there is definitely some people who absolutely love that work.

298
00:22:17,703 --> 00:22:21,218
There are people without this, companies and software packages don't exist.

299
00:22:21,218 --> 00:22:24,919
So maybe I'll ask you about that and you can feel free to say no comment.

300
00:22:24,919 --> 00:22:27,300
Every company needs to go through hiring at some point.

301
00:22:27,300 --> 00:22:37,343
And in today's world, I think that is an insurmountable challenge for many organizations,
given the verbosity of the number of candidate submissions you get.

302
00:22:37,343 --> 00:22:47,295
How have you been tackling that to not only find qualified candidates, but given the
nuance required to operate in these sort of special teams that have just a huge impact on

303
00:22:47,295 --> 00:22:49,630
the business, that sort of thing doesn't just...

304
00:22:49,630 --> 00:22:52,706
show up straight away on a resume that can be easily filtered out.

305
00:22:52,706 --> 00:22:54,367
I think it all depends on the stage of the company.

306
00:22:54,367 --> 00:22:58,051
Like two years back when we were in really scaling stage, we hired a lot.

307
00:22:58,051 --> 00:22:59,642
Now we are not hiring that many.

308
00:22:59,642 --> 00:23:04,446
say, suppose you are on core Ray, then the expertise that is required is different.

309
00:23:04,446 --> 00:23:08,109
So we look for people who have deep systems knowledge.

310
00:23:08,109 --> 00:23:10,130
lot of our hiring is like based on referrals.

311
00:23:10,130 --> 00:23:13,693
We kind of know the popular open source packages.

312
00:23:13,693 --> 00:23:17,366
This is where open source is really beneficial because many companies use Ray.

313
00:23:17,366 --> 00:23:19,988
So they come and apply to saying, Hey, I've already used Ray.

314
00:23:19,988 --> 00:23:21,580
I met you at your conference and stuff like that.

315
00:23:21,580 --> 00:23:21,954
Right.

316
00:23:21,954 --> 00:23:27,646
But then there are also some gems which come in just from cold outbound recruiting, et
cetera.

317
00:23:27,646 --> 00:23:29,209
And you just sometimes have to think creatively.

318
00:23:29,209 --> 00:23:31,971
So we were doing international growth at Uber.

319
00:23:31,971 --> 00:23:37,486
So growing Uber's business outside US in Latin America, Southeast Asia, India, China, et
cetera.

320
00:23:37,486 --> 00:23:43,530
And when we are growing China and India growth team here in San Francisco, we were...

321
00:23:43,530 --> 00:23:50,337
looking for people who had lived in those areas because then they understood the problems
on the ground because Uber was a very much, you know, it's a physical thing.

322
00:23:50,337 --> 00:23:52,769
You need to understand that local place, right?

323
00:23:52,769 --> 00:23:53,090
Yeah.

324
00:23:53,090 --> 00:23:54,341
We had operations team.

325
00:23:54,341 --> 00:23:55,272
So we are like targeting.

326
00:23:55,272 --> 00:24:03,554
So the we did it is we actually put an ad for hiring engineers in a movie in Indian and a
Chinese language movie at the AMC theater.

327
00:24:03,554 --> 00:24:11,986
we went to a social festival and we put a hiring booth and all our nearby booths were all
about foot stalls and we were the only hiring booth and we actually got good candidates.

328
00:24:11,986 --> 00:24:15,010
So what I meant is sometimes you just have to be creative in that.

329
00:24:15,564 --> 00:24:21,634
No, I mean, that's genius on honestly, I think it's the same research that applies to why
hiring.

330
00:24:21,634 --> 00:24:25,336
diverse teams actually guarantees you to get more talented people.

331
00:24:25,336 --> 00:24:34,020
think hiring that, I mean, it's still an aspect of diversity, but if there's no one else
from none of your competitors or even in the whole industry are hiring from particular

332
00:24:34,020 --> 00:24:43,174
venue, you being there gives you a statistically outsized chance on even a finding people,
but a finding expertly good people who would not have gotten picked up because they're not

333
00:24:43,174 --> 00:24:51,668
using whatever's linked in social media or other mechanisms because if they had, they
would have seen, you know, jobs by your competitors and got those up.

334
00:24:52,888 --> 00:24:53,808
I love that example.

335
00:24:53,808 --> 00:25:01,760
mean, I don't know if I would go through the process of buying an ad for a movie, but if
that's you, if there's an adjacency there, like, know, video stuff, video editing, I

336
00:25:01,760 --> 00:25:03,531
totally see the overlap.

337
00:25:03,531 --> 00:25:05,229
Seems like genius idea, honestly.

338
00:25:05,229 --> 00:25:13,633
I would be remiss if I didn't harass you a little bit on something related to Android,
given your historical experience there.

339
00:25:13,633 --> 00:25:17,882
And my question is going to be, I think, did it have to be Java?

340
00:25:17,882 --> 00:25:19,542
it have to be Java?

341
00:25:19,722 --> 00:25:26,742
I mean, it did not have, I think the, I remember right, initially it was JavaScript and
then we moved to a Java based VM at that point of time.

342
00:25:26,742 --> 00:25:28,662
Now even Kotlin is supported.

343
00:25:28,662 --> 00:25:32,282
Did it have some kind of a deep dislike for Java?

344
00:25:32,358 --> 00:25:34,640
Oh, since the university long time ago.

345
00:25:34,640 --> 00:25:46,950
If I had to pick a preference between submitting myself to the Oracle derived mindset and
language of the world or Microsoft, think unfortunately I'd pick Microsoft.

346
00:25:46,950 --> 00:25:52,271
Still not my preference for coding languages for sure, but I do like C Sharp more than...

347
00:25:52,271 --> 00:25:59,783
Yes, I mean there were a lot of controversies like you know between Google and Oracle over
like you know Because Android had created its own VM.

348
00:25:59,783 --> 00:26:03,989
I was not part of the Java VM team But my teammates were deeply involved in the lawsuit
etc.

349
00:26:03,989 --> 00:26:04,758
Wow

350
00:26:04,758 --> 00:26:16,499
Yeah, so I love this lawsuit because it's so ridiculous that the outcome was that open
source software is copyrightable, but Google didn't violate the copyright because their

351
00:26:16,499 --> 00:26:17,851
API didn't match.

352
00:26:17,851 --> 00:26:23,056
The API interface is copyrightable, but they didn't violate it because it was different.

353
00:26:23,056 --> 00:26:24,477
That's just so ridiculous to me.

354
00:26:24,477 --> 00:26:26,259
ah yeah, think so.

355
00:26:26,259 --> 00:26:28,791
Does that mean that you're a Java fan?

356
00:26:28,791 --> 00:26:29,966
You love the JVM.

357
00:26:29,966 --> 00:26:34,286
think every language has a purpose for its use case.

358
00:26:34,286 --> 00:26:39,026
So for example, Go was used in large scale distributed systems for a reason.

359
00:26:39,226 --> 00:26:41,806
And there is a reason why Python is used in the machine learning system.

360
00:26:41,806 --> 00:26:47,546
And because of the number of library packages and data scientists and ML engineers being
comfortable with Python and it's just easy to get started with.

361
00:26:47,546 --> 00:26:54,946
I mean, if you were to ask me my favorite language, would actually pick on C, because
that's what I grew up learning and I still love that.

362
00:26:57,226 --> 00:26:59,702
So maybe this is a diplomatic answer.

363
00:26:59,702 --> 00:27:04,325
But I'm also not one of those language ZL also says, everything else is terrible.

364
00:27:04,325 --> 00:27:06,325
My language is the best kind of stuff.

365
00:27:06,446 --> 00:27:07,987
It's programming languages.

366
00:27:07,987 --> 00:27:12,148
Honestly, I think all the languages conversations will just go away.

367
00:27:12,589 --> 00:27:17,321
AI coding agents, natural languages are the key.

368
00:27:17,321 --> 00:27:22,634
Long pool has moved from pure engineering to what to build.

369
00:27:22,840 --> 00:27:26,830
See, I don't think the poll was ever in engineering.

370
00:27:26,830 --> 00:27:29,456
I think it's always been in what to build.

371
00:27:29,456 --> 00:27:36,071
just think historically we sort of lied to ourselves that we knew what to build and pushed
it on engineering.

372
00:27:36,071 --> 00:27:45,989
And then since the cycle time was so long for engineering to build stuff, we could wait to
verify our hypothesis that we didn't spend any time thinking about.

373
00:27:46,156 --> 00:27:47,011
You're spot on there.

374
00:27:47,011 --> 00:27:47,705
You're spot on there.

375
00:27:47,705 --> 00:27:48,710
I don't disagree with that.

376
00:27:48,710 --> 00:27:49,130
Yeah,

377
00:27:49,130 --> 00:27:51,623
Now it's like, oh crap, we built that?

378
00:27:51,623 --> 00:27:53,892
I guess I have to take the next step now, so.

379
00:27:53,892 --> 00:27:54,518
It's true.

380
00:27:54,518 --> 00:27:56,923
I mean, yes, yes, I don't disagree with it.

381
00:27:56,923 --> 00:28:00,967
like your diplomatic answer because I think there's a couple different aspects to it.

382
00:28:00,967 --> 00:28:08,205
think one of them is that fundamentally each language has its deficiencies, but maybe
there's also its benefits.

383
00:28:08,205 --> 00:28:11,230
so matching it up with the use case is required.

384
00:28:11,230 --> 00:28:14,636
I disagree with the train of thought that...

385
00:28:14,636 --> 00:28:17,468
all the languages are completely isomorphic or interchangeable.

386
00:28:17,468 --> 00:28:26,883
Because I do see, like as you said, there's no alternative for Python for machine learning
because you needed something sitting on top of R to do all of the quantitative analysis.

387
00:28:26,883 --> 00:28:30,305
And so you got the interface with Python and then that's grown over time.

388
00:28:30,305 --> 00:28:37,639
it really only the last couple of years since Chachipiti did we start to see other
interfaces pop up for other languages.

389
00:28:37,639 --> 00:28:40,070
But no, I'm totally with you there.

390
00:28:40,070 --> 00:28:42,271
So I like that flavor.

391
00:28:42,271 --> 00:28:43,682
I think that's interesting.

392
00:28:43,682 --> 00:28:45,993
engineering leaders need to think also beyond engineering it.

393
00:28:45,993 --> 00:28:55,626
think it becomes a little bit of the onus on the engineering leaders to make sure your
sales team, your customer support team, your other teams are also using AI and getting the

394
00:28:55,626 --> 00:28:58,096
benefits out of AI and how that is actually set up.

395
00:28:58,096 --> 00:29:00,167
So many times you have to hold training sessions for them.

396
00:29:00,167 --> 00:29:09,720
Many times groups can get siloed and especially with AI and agents and the state of the
company, engineering leaders can actually play a bigger role than just being stuck in

397
00:29:09,720 --> 00:29:10,388
their book.

398
00:29:10,388 --> 00:29:19,721
I think you said something interesting here, which is it sounds like we're shifting back
the responsibility of building to where the decision can be made, where the knowledge is.

399
00:29:19,721 --> 00:29:28,963
So if it's the sales team that, or marketing team that wants to build something, they now
not only have the responsibility, but sort of the obligation to make that happen.

400
00:29:28,963 --> 00:29:37,856
And so my question is gonna be, how do you actually train them to build software in the
reliable way that we believe that has been a...

401
00:29:37,856 --> 00:29:40,231
sort of a lifeblood of engineering for such a long time.

402
00:29:40,231 --> 00:29:46,322
mean, I know there's so many engineers out there still who believe, no, no, no one else
can do it exactly as I can.

403
00:29:46,322 --> 00:29:50,798
I know how to do the special thing myself and no one else can really make that happen.

404
00:29:50,798 --> 00:30:00,698
a very good conversation because just two days back, my security engine leader came in and
said, Hey, this person in sales used $15,000 worth of token.

405
00:30:00,698 --> 00:30:03,278
I sat down with them and they just didn't have to spend the whole context.

406
00:30:03,278 --> 00:30:05,538
could have done the whole thing in like $15 kind of stuff.

407
00:30:05,538 --> 00:30:10,658
Maintainable systems is actually an art and this experience matters and agents will not
necessarily.

408
00:30:10,798 --> 00:30:12,498
So sales marketing, etc.

409
00:30:12,498 --> 00:30:13,122
can.

410
00:30:13,122 --> 00:30:17,064
have agents to improve their workflow, make it much more efficient kind of stuff.

411
00:30:17,064 --> 00:30:22,168
But then you also have to be very, very clear that if there is something they're building
in production, who's responsible for it?

412
00:30:22,168 --> 00:30:23,810
Many times it says, I built it.

413
00:30:23,810 --> 00:30:24,730
It's broken.

414
00:30:24,730 --> 00:30:25,371
please, can you help me?

415
00:30:25,371 --> 00:30:27,552
And then you, why did you build it without talking to us?

416
00:30:27,552 --> 00:30:34,327
And all of that stuff, But there are also platforms which are created to actually solve
this problem, help solve these problems, uh right?

417
00:30:34,327 --> 00:30:42,062
So where you don't have to make it easy to develop agentic software without having to
worry about the infrastructure, et cetera, kind of thing.

418
00:30:42,070 --> 00:30:45,490
I think I'm going to skip ahead a little bit and I really like your framing.

419
00:30:45,490 --> 00:30:49,067
I think it's that we'll eventually lose the engineering team with this mentality.

420
00:30:49,067 --> 00:30:51,580
There will be no one to double check what is happening elsewhere.

421
00:30:51,580 --> 00:30:54,573
Because if everyone can build things, what is engineering really doing for us?

422
00:30:54,573 --> 00:30:55,804
And maybe we'll change the name.

423
00:30:55,804 --> 00:30:59,909
Maybe we'll just lose that reliability that we have in our organizations.

424
00:30:59,909 --> 00:31:05,506
But the idea that comes to mind really based on what you're saying is it sounds like we
need to

425
00:31:05,506 --> 00:31:14,652
take the original idea of what DevOps meant, breaking down the silo between engineering
and release basically, and have everyone in the organization understand really

426
00:31:14,652 --> 00:31:18,655
fundamentally what DevOps is, that if you write this thing, you will run it.

427
00:31:19,436 --> 00:31:27,181
One struggle I could imagine is that the mindset of people who haven't historically built
software wasn't necessarily on reliability.

428
00:31:27,181 --> 00:31:31,234
And teaching them reliability, is it easy or could it be a challenge?

429
00:31:31,234 --> 00:31:34,136
Do you even have the right people in those positions to do both?

430
00:31:34,136 --> 00:31:43,526
the job they have been doing, sales and marketing or whatever, design, et cetera,
literally anything other than engineering to also be responsible for the reliability work.

431
00:31:43,534 --> 00:31:49,600
I don't think we should teach them reliability to be honest, because teaching reliability,
there's a teaching part and there's experience.

432
00:31:49,600 --> 00:31:50,520
Teaching is enough.

433
00:31:50,520 --> 00:31:53,963
You need to have the battle scars.

434
00:31:53,963 --> 00:31:56,045
Even with agents, you need to have the battle scars.

435
00:31:56,045 --> 00:32:00,438
Agents help you a bit, know, agents can cause other kinds of outages without
observability.

436
00:32:00,438 --> 00:32:04,782
So you need to have a strong observability team, strong production infrastructure team,
release and engineering team.

437
00:32:04,782 --> 00:32:11,458
think sales folks should create workflow automation tools that they run themselves, but
not production software.

438
00:32:11,458 --> 00:32:12,298
Yeah.

439
00:32:12,534 --> 00:32:21,297
And they should have a cloud automation tool that makes it easier for them to do research
on a customer and be able to generate the right insights for the customer.

440
00:32:21,297 --> 00:32:26,528
All the examples we talked about is data processing the problem for them, is serving the
problem for them, training the problem for them.

441
00:32:26,528 --> 00:32:30,140
Those kind of research, cloud or cursor, can easily do it for them.

442
00:32:30,140 --> 00:32:33,621
And that's what workflow automation is what they should be focusing on.

443
00:32:33,621 --> 00:32:38,382
I would strongly resist the urge for an engineering team to build a sales specific.

444
00:32:38,382 --> 00:32:40,962
tool and then making it reliable especially at a startup.

445
00:32:40,962 --> 00:32:42,782
It's the buy versus build sometimes.

446
00:32:43,022 --> 00:32:43,042
I...

447
00:32:43,042 --> 00:32:46,210
in these cases spending some money to buying the right software is much easier.

448
00:32:46,210 --> 00:32:54,810
It's so hard to convince startups or those companies with very little money not to just do
a thing that seems like if they're at the pinnacle of ignorance, you know, doing that

449
00:32:54,810 --> 00:32:56,022
thing and handing it off.

450
00:32:56,022 --> 00:33:03,420
And but I mean, I think you make two really good points that you have to think about the
job functions that are required or really roles responsibilities in your company and look

451
00:33:03,420 --> 00:33:04,701
at the tools that they're utilizing.

452
00:33:04,701 --> 00:33:07,192
And if you find people who need

453
00:33:07,192 --> 00:33:12,013
feel like they need to build something, they don't have the skills to really do that, then
look at what tool they're using.

454
00:33:12,013 --> 00:33:15,252
Like maybe the wrong tools are being evaluated and handled there.

455
00:33:15,252 --> 00:33:23,326
And you also shouldn't build those yourself, but you know, maybe switch off of the
Salesforce and SAPs to the little startup companies that are doing the exact thing that,

456
00:33:23,326 --> 00:33:26,127
you know, you want your people actually utilizing.

457
00:33:26,127 --> 00:33:35,588
So the second thing that I really spoke to me was that in order to understand how to build
reliable software, you need to have some sort of PTSD in your path.

458
00:33:35,588 --> 00:33:36,349
Yeah.

459
00:33:36,622 --> 00:33:41,182
you the trauma that you've seen building something and struggling and like being on call.

460
00:33:41,182 --> 00:33:47,502
And I think there are those that learn through, you know, others' experience and those
that learn through, you know, books and knowledge.

461
00:33:47,502 --> 00:33:51,842
And some of these things I think are very difficult to teach without having to deal with
it yourself.

462
00:33:53,422 --> 00:33:56,242
And now's the time to switch over to PIX.

463
00:33:56,242 --> 00:33:58,744
So JK, what did you bring for us today?

464
00:33:58,744 --> 00:33:59,255
Pics.

465
00:33:59,255 --> 00:34:06,220
right, so there's an interesting book I'm actually reading called The Explorer's Gene by
Alan Hutchison.

466
00:34:06,220 --> 00:34:09,763
So it goes into the fact as to when should you explore and when should you exploit?

467
00:34:09,763 --> 00:34:14,966
Why did humans adventure out, even in the early days, to newer lands?

468
00:34:14,966 --> 00:34:17,089
Why did some do it and some not?

469
00:34:17,089 --> 00:34:20,232
And many times after I was reached age, they stopped exploring.

470
00:34:20,232 --> 00:34:22,383
So Warren, how do you navigate?

471
00:34:22,383 --> 00:34:27,097
You're sitting in a car, maybe you're using Google Maps or Apple Maps, and you have to go
from point A to point B.

472
00:34:27,097 --> 00:34:28,302
How do you navigate?

473
00:34:28,302 --> 00:34:32,562
I'm so glad I haven't had to drive in almost a decade for real.

474
00:34:32,562 --> 00:34:34,842
So that's one of the benefits of living in Switzerland.

475
00:34:34,842 --> 00:34:36,702
But you're still planning a path.

476
00:34:36,702 --> 00:34:37,802
I totally get you.

477
00:34:37,802 --> 00:34:43,342
I mean, my strategy now is like try to memorize the directions beforehand, like look
visually at it.

478
00:34:43,342 --> 00:34:49,982
And then while I'm in the car driving, I constantly am second guessing myself like, wait,
I was supposed to turn already, right?

479
00:34:49,982 --> 00:34:50,962
Shouldn't I already turn?

480
00:34:50,962 --> 00:34:51,922
Shouldn't I already turn?

481
00:34:51,922 --> 00:34:55,062
So, you know, if I'm already in that position, I am, am stuck.

482
00:34:55,062 --> 00:34:57,022
I have the worst on there.

483
00:34:57,022 --> 00:34:57,344
So.

484
00:34:57,344 --> 00:35:06,353
Now, I just, you pull out your phone, you have it attached, you have GPS going with the
maps or whatever else you're using, and it will tell you the turn-by-turn directions.

485
00:35:06,353 --> 00:35:14,691
And it's still not good enough for me because I want to know the thing that happens
afterwards so I can already prepare myself mentally.

486
00:35:14,691 --> 00:35:18,360
I guess I'm just complaining about the state of map driving today.

487
00:35:18,360 --> 00:35:20,245
Do you zoom in and zoom out before starting your car?

488
00:35:20,245 --> 00:35:21,734
Do you zoom in and zoom out?

489
00:35:21,734 --> 00:35:25,427
so many times in like while I'm driving, I want to zoom in and like try to understand
what's there.

490
00:35:25,427 --> 00:35:26,579
Like how many lanes are there?

491
00:35:26,579 --> 00:35:27,720
Am I going to get in the right?

492
00:35:27,720 --> 00:35:34,186
It's like because there are so many times, especially in the US where you're like driving
and it's like, yeah, get in the right lane to turn right.

493
00:35:34,186 --> 00:35:38,651
But then the next instruction you don't tell you is like, and then get in the left lane to
turn left.

494
00:35:38,651 --> 00:35:41,484
And I'm like, I wish I knew that because there are like five lanes.

495
00:35:41,484 --> 00:35:46,158
I wouldn't have picked the right most lane to get in in a in dire traffic.

496
00:35:46,158 --> 00:35:46,778
Yeah.

497
00:35:46,778 --> 00:35:48,298
Why do some people do it and some don't?

498
00:35:48,298 --> 00:35:51,058
Par- So here's an interesting thing.

499
00:35:51,058 --> 00:35:55,738
Say for example, let's since you said Switzerland, suppose you're going on a trip from
Bern to Berlin, right?

500
00:35:55,738 --> 00:35:56,058
Yeah.

501
00:35:56,058 --> 00:35:57,378
This is my personal habit.

502
00:35:57,378 --> 00:36:02,098
If I'm going, Google Maps has given me direction and I'll zoom in and zoom out and said,
oh, these are the freeways.

503
00:36:02,098 --> 00:36:03,938
Oh, maybe there's an interesting route over here.

504
00:36:03,938 --> 00:36:04,318
Oh, okay.

505
00:36:04,318 --> 00:36:07,498
Even though I would not take it, you know, mentally I'm like, oh, here's the thing.

506
00:36:07,498 --> 00:36:07,698
Okay.

507
00:36:07,698 --> 00:36:09,258
Now I'm going close to this thing.

508
00:36:09,258 --> 00:36:12,198
And before I even start driving and my wife has a different style.

509
00:36:12,198 --> 00:36:14,158
And when I'm doing that, need to annoy her.

510
00:36:14,158 --> 00:36:15,378
And she's like, hey, here's the route.

511
00:36:15,378 --> 00:36:17,118
We are going point A to point B kind of stuff.

512
00:36:17,118 --> 00:36:19,318
And I was like, okay, maybe the different styles.

513
00:36:19,318 --> 00:36:23,678
Until I read this book and it actually says the different parts of your brain are
activated.

514
00:36:23,678 --> 00:36:26,778
In the first style, it's the hippocampus, which is doing the job.

515
00:36:26,778 --> 00:36:32,958
And in the second style, it's the claudate nucleus, if I'm pronouncing these words right
for any of the...

516
00:36:32,958 --> 00:36:34,496
In the audience.

517
00:36:34,496 --> 00:36:36,838
It's that part of the brain that's actually doing the work kind of thing.

518
00:36:36,838 --> 00:36:40,161
so which one is an exploration part, which is the exploitation part kind of stuff.

519
00:36:40,161 --> 00:36:41,973
So there's a fascinating section about this.

520
00:36:41,973 --> 00:36:43,254
And I was like, interesting.

521
00:36:43,254 --> 00:36:50,701
And so that kind of explained why some people pick this versus some people like always
want to do it.

522
00:36:50,701 --> 00:36:54,965
And you know, when I was a kid, I used to draw maps by hand.

523
00:36:54,965 --> 00:36:55,445
Here's it.

524
00:36:55,445 --> 00:36:56,066
This is the router.

525
00:36:56,066 --> 00:36:58,348
This is how I'm going to plan my city and stuff like that.

526
00:36:58,348 --> 00:36:59,629
it is, yeah.

527
00:36:59,629 --> 00:37:01,140
So highly recommend the book.

528
00:37:01,140 --> 00:37:03,352
are such an interesting topic as well.

529
00:37:03,352 --> 00:37:13,238
just feel like unless you actually went out and tried orienteering, you probably would not
survive in the wilderness today, especially given the level of technology.

530
00:37:13,238 --> 00:37:14,169
Imagine if it all went away.

531
00:37:14,169 --> 00:37:16,060
It sounds like a really fascinating book.

532
00:37:16,060 --> 00:37:17,781
actually now want to add it to my read list.

533
00:37:17,781 --> 00:37:18,691
So thank you for that.

534
00:37:18,691 --> 00:37:19,072
Yeah.

535
00:37:19,072 --> 00:37:23,064
So my pick, maybe it's just not as inspired as that, honestly.

536
00:37:23,064 --> 00:37:25,792
So there's one particular post in this

537
00:37:25,792 --> 00:37:28,814
whole collection called archers don't fire volleys.

538
00:37:28,814 --> 00:37:30,205
I guess that's sort of my pick.

539
00:37:30,205 --> 00:37:33,227
The collection is called the collection of unmitigated pedantry.

540
00:37:33,227 --> 00:37:39,812
It's an article series online by I think it's like a Greek Roman scholar basically.

541
00:37:39,812 --> 00:37:49,318
And the interesting thing is that it talks about how basically everything in popular
culture when it comes to references, medieval and Romans specifically regarding combat is

542
00:37:49,318 --> 00:37:50,629
just like so totally wrong.

543
00:37:50,629 --> 00:37:52,100
Archers don't fire volleys.

544
00:37:52,100 --> 00:37:55,502
There's no like get ready aim fire because

545
00:37:55,502 --> 00:38:04,628
Firing arrows is actually incredibly taxing and if you've ever done it, you don't sit
there with your, the arrow hooked back, holding it for minutes for the perfect

546
00:38:04,628 --> 00:38:05,449
opportunity.

547
00:38:05,449 --> 00:38:09,231
The other thing is that arrows actually don't, weren't used to like kill people.

548
00:38:09,231 --> 00:38:14,395
I mean like you wouldn't expect that there'd be a lot of deaths as a result of firing as
an archer.

549
00:38:14,395 --> 00:38:16,246
It was most of like disorientation.

550
00:38:16,246 --> 00:38:23,611
Maybe you, you know, hit people and their heads or they're wearing armor and legs and
they're like trying to guard themselves.

551
00:38:23,611 --> 00:38:24,948
And so it's

552
00:38:24,948 --> 00:38:32,884
delay tactics and after that now when I watch stuff it just one more of those things where
I'm like it always is like they're typing on the keyboard that's not how you hack stuff

553
00:38:32,884 --> 00:38:41,662
now I have to be annoyed whenever I see archers volleying I I love this collection though
there's like there's so many things in there and I'm sure one of these other articles will

554
00:38:41,662 --> 00:38:49,620
be my pick later and it's not like a short little post like there are 20 40 minute reads
basically that explain every aspect of

555
00:38:49,620 --> 00:38:57,536
of the architect, like the technology that they had at the time, why it was used, how the
battles were actually waged and why those things were wrong.

556
00:38:57,536 --> 00:39:06,552
I just, there's almost too much there to go through, but it's written in such a great
style that it almost makes you want to keep on going and become an expert in that topic.

557
00:39:06,552 --> 00:39:17,159
And I just think back, like if my teachers gave us reading material on different topics
like this, when I was in middle school or high school, I probably wouldn't have been a

558
00:39:17,159 --> 00:39:18,569
software engineer.

559
00:39:18,702 --> 00:39:20,042
All right, I have a different question for you.

560
00:39:20,042 --> 00:39:22,542
What's this painting behind you?

561
00:39:22,542 --> 00:39:23,847
I'm guessing it's a painting.

562
00:39:23,847 --> 00:39:24,708
It is a painting.

563
00:39:24,708 --> 00:39:28,851
It's acrylic with molding in it of different kinds.

564
00:39:28,851 --> 00:39:30,041
It's abstract art.

565
00:39:30,041 --> 00:39:31,432
It's actually...

566
00:39:31,432 --> 00:39:36,055
You are the first person, I think, on this podcast in years to ask me what this is.

567
00:39:36,055 --> 00:39:39,418
These pieces behind me were all done by my wife.

568
00:39:39,418 --> 00:39:41,369
And a painting is what it is.

569
00:39:41,369 --> 00:39:45,322
If it could be conveyed with words, then you wouldn't need to paint it.

570
00:39:45,322 --> 00:39:46,613
You could just write the words there.

571
00:39:46,613 --> 00:39:48,244
So it is what it is.

572
00:39:48,244 --> 00:39:49,254
And that's it.

573
00:39:49,254 --> 00:39:52,076
I mean, you could say there's like a dark side and a light side.

574
00:39:52,088 --> 00:39:56,930
filled and whatever, you know, it evokes a feeling or an emotion and that's, that's all
there is.

575
00:39:56,930 --> 00:39:59,655
Yeah, think the colours are fascinating at least, the way I see it in the video.

576
00:39:59,655 --> 00:40:01,785
I'd love to see it in real life.

577
00:40:01,994 --> 00:40:07,502
You know, maybe I got to take a picture of this and put it up on the podcast so people can
actually see what's on this wall behind me.

578
00:40:07,502 --> 00:40:16,044
I, to be fair, I went through a couple different ones, which I, we talked about, like
which ones was I comfortable having on the wall behind me for video calls.

579
00:40:16,204 --> 00:40:18,613
Your wife is going to be happy listening to this section.

580
00:40:20,624 --> 00:40:24,429
Or she'll be mortified that I brought this up.

581
00:40:24,429 --> 00:40:26,532
No, I actually love this painting.

582
00:40:26,532 --> 00:40:29,075
one of hers that is my absolute favorite.

583
00:40:29,075 --> 00:40:32,299
Well, thank you, JK, for coming on to this episode.

584
00:40:32,299 --> 00:40:33,771
It's been absolutely great having you.

585
00:40:33,771 --> 00:40:34,638
Thank you so much.

586
00:40:34,638 --> 00:40:40,478
Same here, it was fun talking to you about everything happening in AI landscape, release
processes, bunch of things.

587
00:40:41,547 --> 00:40:42,528
I'm glad to hear it.

588
00:40:42,528 --> 00:40:46,764
thanks for all the listeners for tuning in for today's episode of Ventures and DevOps.

589
00:40:46,764 --> 00:40:50,598
And hopefully we'll see everyone back again next.