1
00:00:07,918 --> 00:00:15,685
Welcome back to Ed Ventures and DevOps, where we try to solve our collective existential
crises we all have in software engineering.

2
00:00:15,685 --> 00:00:16,665
Crises?

3
00:00:16,665 --> 00:00:17,546
Crises?

4
00:00:17,546 --> 00:00:20,068
uh Can you have more than one at a time?

5
00:00:20,068 --> 00:00:28,816
Until we resolve the English grammar question, we'll have to switch to our backup for this
week, which is a deep dive into control planes, continuous integration, and build systems.

6
00:00:28,816 --> 00:00:36,598
And to answer all our impossibly tough questions, we have previously principal engineer,
early circle CI denizen, Haskellar.

7
00:00:36,598 --> 00:00:41,354
and currently staff engineer on the stability team at Mercury with us, Ian Duncan.

8
00:00:41,354 --> 00:00:42,825
Welcome to the show.

9
00:00:43,767 --> 00:00:46,754
Is it Haskellar or Haskellar?

10
00:00:46,754 --> 00:00:51,037
I would say Haskell-er is probably the more likely uh pronunciation you're going to hear.

11
00:00:51,037 --> 00:01:03,164
Yeah, I'd say Haskell is actually uh definitely one of my passions as far as programming
goes because it's a language where a lot of the research that's come out of it has led to,

12
00:01:03,194 --> 00:01:06,424
has informed a lot of design decisions for subsequent languages.

13
00:01:06,424 --> 00:01:12,920
The reason I actually sourced you for this episode is because of one of your blog posts on
the challenges of GitHub CI.

14
00:01:12,920 --> 00:01:19,607
And I've actually gone through quite some of your blog and it seems like every article is
just, honestly, it's a gold mine.

15
00:01:19,607 --> 00:01:20,948
They're really well done.

16
00:01:20,948 --> 00:01:21,989
It's not just the words.

17
00:01:21,989 --> 00:01:22,660
I saw some of them.

18
00:01:22,660 --> 00:01:25,533
They've got like clickable dynamic examples.

19
00:01:25,533 --> 00:01:31,488
Like you must have spent no small amount of effort putting these together and they're not
short articles either.

20
00:01:31,707 --> 00:01:40,224
Yeah, I'll say uh in the interest of total honesty, a lot of the little visualizations
have been somewhat vibe-coded to try to get a point across.

21
00:01:40,224 --> 00:01:44,257
It's not like I actually wrote every single one of those by hand.

22
00:01:44,802 --> 00:01:54,332
Well, so there is a great book called, blah, blah, blah by Dan Roem, which really talks
about the importance of conveying information, both as a written way and using visuals.

23
00:01:54,332 --> 00:02:02,941
And I feel like the moving visuals are a whole nother step on top of that, that I think
I've only ever seen one other blog out there that ever tries to include that.

24
00:02:02,941 --> 00:02:08,750
And this, you know, I put myself in that group, like I don't have any moving visuals in my
articles.

25
00:02:08,750 --> 00:02:12,501
Well, there's, yeah, you may be referencing the same uh individual.

26
00:02:12,501 --> 00:02:17,652
There's a guy out there who does some really fantastic infographic type blogs.

27
00:02:17,652 --> 00:02:22,834
He's written one that's about load balancing algorithms and a few other things.

28
00:02:22,914 --> 00:02:28,195
And that was sort of the inspiration that I wanted to go for was to kind of try to help
people visualize as they go.

29
00:02:28,195 --> 00:02:29,496
Like, what is the point?

30
00:02:29,496 --> 00:02:38,250
Because uh there's a lot of nuance to a lot of these concepts and kind of helping people
have something to fiddle around with until it clicks, find, can be quite helpful.

31
00:02:38,250 --> 00:02:38,941
yeah, no, I totally agree.

32
00:02:38,941 --> 00:02:49,260
It's very hard to get feedback on how well you're post land and so doing everything you
can to actually try to convey it effectively is definitely worthwhile something.

33
00:02:49,260 --> 00:02:56,033
Yeah, what's interesting, think, is I've been having a little bit more uptick on people
reading my blog lately.

34
00:02:56,033 --> 00:03:06,317
But by and large, it's a mixture of stuff that I've written for myself or else propaganda
for people at work to convince them to do something that I want them to do.

35
00:03:07,758 --> 00:03:16,422
So in particular, the GitHub actions rant that you referenced is largely because I'm
trying to convince work to switch to build-kite.

36
00:03:17,326 --> 00:03:25,266
You know, I was just working one day and just like, you know, GitHub actions glitched out
on me and it was just like the final straw.

37
00:03:25,266 --> 00:03:29,786
And I was just like, you know, I kind of crashed out a little bit, but here we are.

38
00:03:30,046 --> 00:03:37,006
A guy that I used to work with, his name's Mark Watton, wrote a really nice article about
what do you call the novelty budget?

39
00:03:37,006 --> 00:03:38,986
And it's a term that's really stuck with me.

40
00:03:38,986 --> 00:03:47,062
I'm sure maybe other people have used it as well, but you know, just the idea that you
have every business needs like competitive advantage.

41
00:03:47,062 --> 00:03:57,698
And so like if you're picking tooling or if you're, you know, whether it's databases or
languages or SaaS products that you're adopting to like achieve your goals, have a certain

42
00:03:57,698 --> 00:04:04,792
amount of tokens where you, can probably invest those into like, I think this technology
is really going to move the needle for me.

43
00:04:04,792 --> 00:04:08,384
But you know, there's, there's a lot of things that are like Postgres, right?

44
00:04:08,384 --> 00:04:10,175
Where it's like, everybody uses it.

45
00:04:10,175 --> 00:04:12,336
It's a really well understood tool.

46
00:04:12,517 --> 00:04:15,214
It's probably going to be here 30 years from now in some

47
00:04:15,214 --> 00:04:23,503
capacity and like do you really need to reinvent the database or do you really need to
like you know what I mean so I'd really say that I think I think this kind of ties into

48
00:04:23,503 --> 00:04:31,781
that concept of like you should really pick your battles uh or like limit the number of
and I'm saying this as a guy who's been doing Haskell for 20 years.

49
00:04:31,781 --> 00:04:33,233
I know right.

50
00:04:33,233 --> 00:04:34,413
Believe me.

51
00:04:34,978 --> 00:04:42,788
mean, there is definitely this aspect of being a hypocrite, but at the same time, I think
as you spend more time in engineering, you understand that there's like the business needs

52
00:04:42,788 --> 00:04:46,152
and then except like everything is risky in some regard.

53
00:04:46,152 --> 00:04:51,638
There is no such thing as the perfect software service except for the one that just
doesn't exist.

54
00:04:51,638 --> 00:04:56,014
And so since it can't be perfect and if you believe it can be perfect, you know,

55
00:04:56,014 --> 00:05:02,394
I want to invite you onto the show so we can have a conversation about that because it
will fail in some way.

56
00:05:02,394 --> 00:05:06,454
There is no way to have fully reliable stuff and so you're making a trade-off somewhere.

57
00:05:06,774 --> 00:05:13,994
And I love the post that you're talking about actually which is about like innovation
tokens and using boring technology basically.

58
00:05:14,154 --> 00:05:24,614
I don't know why I would have called Postgres boring that long ago but at this point it
has so many extra sort of angles to solve business problems that it's hard to justify

59
00:05:24,614 --> 00:05:25,676
picking up

60
00:05:25,676 --> 00:05:32,511
particular other database technology as a solution because it's just so natively
implemented in Postgres today.

61
00:05:32,511 --> 00:05:41,447
That being said, I sort of want to ask your perspective, especially being on the stability
team at Mark3, where you've used your innovation tokens versus where you felt like boring

62
00:05:41,447 --> 00:05:42,670
technology was better.

63
00:05:42,670 --> 00:05:48,250
Oh yeah, mean Haskell is easily our biggest innovation token, I would say.

64
00:05:49,770 --> 00:05:51,090
There's no doubt.

65
00:05:51,090 --> 00:05:57,030
But I think we kind of hit the functional programming nerd peak on a few other fronts too.

66
00:05:57,030 --> 00:06:03,050
We use Nix quite a lot for reproducible builds and for setting up development
environments.

67
00:06:03,050 --> 00:06:05,210
And that's gotten a little bit more mainstream traction.

68
00:06:05,210 --> 00:06:12,610
But in terms of how we do our operations at Mercury, we are pretty ruthlessly simple about
it.

69
00:06:12,610 --> 00:06:26,794
I would say up until a year, year and a half ago, we were still basically just deploying
to straight up EC2 instances using a bash script and operating a massive system with

70
00:06:26,794 --> 00:06:29,335
really simple deployment techniques.

71
00:06:29,335 --> 00:06:30,905
there was really nothing fancy there.

72
00:06:30,905 --> 00:06:33,536
was GitHub actions, build sort of like a Nix.

73
00:06:33,536 --> 00:06:37,617
Package isn't really the right word, but I'm going to use it just to keep the conversation
simple here.

74
00:06:37,617 --> 00:06:39,628
Build a package of our system.

75
00:06:39,628 --> 00:06:44,660
ship it to the boxes, switch over to the new package, restart the services, bots your
uncle, right?

76
00:06:44,660 --> 00:06:51,053
And we got super far with that with like three, three instances, five instances on EC2.

77
00:06:51,053 --> 00:06:56,785
You don't have to have like a lot of real fancy stuff until you get pretty far along as a
company.

78
00:06:56,785 --> 00:06:59,807
So yeah, I would say it doesn't take much.

79
00:06:59,807 --> 00:07:05,209
And in fact, we were really quite ruthless about trying to avoid adopting additional like
databases.

80
00:07:05,209 --> 00:07:07,502
Basically we had Postgres, we have...

81
00:07:07,502 --> 00:07:10,242
Up until very recently, we didn't even use Redis for anything.

82
00:07:10,242 --> 00:07:12,142
And we still don't even use that for caching.

83
00:07:12,142 --> 00:07:14,062
We just use it for some tertiary things.

84
00:07:14,062 --> 00:07:17,742
It was basically Postgres, Cloudflare, and some EC2 instances.

85
00:07:17,742 --> 00:07:22,422
And we're servicing a huge amount of customers and moving a lot of money around.

86
00:07:22,462 --> 00:07:23,862
And that's all it takes.

87
00:07:23,862 --> 00:07:27,022
And even the one thing I'll say is that it can be pushed too far.

88
00:07:27,022 --> 00:07:31,482
So we were also using Postgres as our queuing mechanism.

89
00:07:32,502 --> 00:07:33,822
you'll see this.

90
00:07:33,822 --> 00:07:36,790
I feel like I see about once every six months to a year.

91
00:07:36,790 --> 00:07:46,456
somebody will come onto Hacker News or Lobster or these other tech fora and be like, you
can just use Postgres as a queue and it's fine and it's actually not a big deal.

92
00:07:46,456 --> 00:07:53,110
But it turns out that it's remarkably hard to do that properly and it does scale for good
long amount of time.

93
00:07:53,110 --> 00:08:01,915
But once you get past the point where it doesn't scale, then it's kind of mission critical
that you get off of it because if you're running a huge amount of stuff through that, then

94
00:08:01,915 --> 00:08:03,874
it's going to be using a lot of your

95
00:08:03,874 --> 00:08:12,157
database resources, it's going to affect uh replication to your read replicas, it's going
to affect how long it takes to restore backups, all these things.

96
00:08:12,157 --> 00:08:14,597
So I guess what I'll say on that here.

97
00:08:14,617 --> 00:08:16,858
Yeah, no, there's lots of stories here.

98
00:08:17,878 --> 00:08:27,051
What I'll say on that front, before I get into any stories, just I'm really a strong
believer that you should be looking nine months to a year into the future on a pretty

99
00:08:27,051 --> 00:08:30,638
regular basis and forecasting what happens at that point.

100
00:08:30,638 --> 00:08:36,738
Because it's just one of those problems where if you get to the point where it's a
problem, you don't necessarily have enough time to solve it.

101
00:08:36,738 --> 00:08:45,818
And so you either have to spend a huge amount of money to scale vertically, or it becomes
an all hands on deck situation and product development screeches to a halt, things like

102
00:08:45,818 --> 00:08:46,518
that.

103
00:08:47,331 --> 00:08:57,190
Well, on that, don't you think having those predictive problems in your backlog would
allow engineering teams to remember what challenges are that are there and then just

104
00:08:57,190 --> 00:09:03,626
prioritize according to whatever the business initiatives are that are aligned with the
vision that goes out maybe to a year?

105
00:09:03,827 --> 00:09:06,520
What's the problem with relying on that as the strategy?

106
00:09:06,520 --> 00:09:09,302
You mean like filing a linear ticket and...

107
00:09:09,302 --> 00:09:14,285
...so prioritize that?

108
00:09:14,285 --> 00:09:15,856
Well, I think that's a cultural thing.

109
00:09:15,856 --> 00:09:19,699
It depends on the context of the company that you're working at.

110
00:09:19,699 --> 00:09:27,324
But as we know with a lot of the startup world, there's a huge amount of pressure to be
faster than your competitors, to ship features faster.

111
00:09:27,324 --> 00:09:36,226
uh You know, the rise of AI is, if anything, accelerating that sense of pressure like
innovate or...

112
00:09:36,226 --> 00:09:37,348
be left behind.

113
00:09:37,348 --> 00:09:48,703
And so I think there's a huge amount of pressure to deprioritize maintenance tasks or
preventative or work on these preventative measures in favor of just going full steam

114
00:09:48,703 --> 00:09:51,186
ahead on beating your competitors, I guess.

115
00:09:51,695 --> 00:10:00,602
It's like this thing, and I think you have to drive a car to sort of understand this
analogy, but it's like if you see two cars headed to an intersection way ahead of time

116
00:10:00,602 --> 00:10:07,067
where they're going to collide at some point, like they're not trying to veer off course
and it's going to happen.

117
00:10:07,067 --> 00:10:12,652
It's just like in slow motion, you can see and it's like, don't you want to do something
about it?

118
00:10:12,652 --> 00:10:20,792
And I just find that a lot of people uh have shifted their focus to more short term to not
even seeing the accident.

119
00:10:20,792 --> 00:10:26,447
coming head on, they're just not thinking about it, they're not paying attention, where
it's clearly going to happen.

120
00:10:26,447 --> 00:10:29,804
And I feel like even now, it's even more of a struggle.

121
00:10:29,804 --> 00:10:38,537
Well, there's this, I guess what I'd say on this front is the problem that you always have
to be looking to solve is making sure that uh incentives are aligned.

122
00:10:38,537 --> 00:10:50,490
If a person on a product team, like a PM or an engineer is not prioritizing the sort of
like stability or reliability issues, it's because they've got like an OKR or they've got

123
00:10:50,490 --> 00:10:54,861
some upper level manager breathing down their back about a particular problem.

124
00:10:54,861 --> 00:10:56,962
And if they say, well, I don't have time to...

125
00:10:56,962 --> 00:10:59,374
meet our quarterly objectives that you've established for me.

126
00:10:59,374 --> 00:11:02,407
I have to work on this thing that's blah, blah.

127
00:11:02,407 --> 00:11:06,020
It's like the Charlie Brown voice, like wah, wah, wah, wah.

128
00:11:06,020 --> 00:11:13,057
That engineer knows they're not doing their career any favors in that moment by pissing
off a person who's got other priorities.

129
00:11:13,057 --> 00:11:17,221
uh So it's kind of a self-preservation ah mechanism.

130
00:11:17,221 --> 00:11:18,284
unless you can...

131
00:11:18,284 --> 00:11:23,667
really work with your upper level engineering management to make sure that those
incentives are aligned.

132
00:11:23,667 --> 00:11:28,462
It's a remarkably hard thing for a single engineer in a company to accomplish on their
own.

133
00:11:28,462 --> 00:11:30,562
Yeah, absolutely.

134
00:11:30,702 --> 00:11:38,998
And now I'm going to grill you on what were you using if it wasn't SQS and AWS for your
queue management.

135
00:11:38,998 --> 00:11:43,819
yeah, we had a homegrown, well, we still have it to a degree.

136
00:11:43,819 --> 00:11:46,650
It's called Postgres queue, which is not super imaginative.

137
00:11:46,650 --> 00:11:56,473
it's, yeah, it's basically it's a table and it is really nice because it follows the
transactional outbox pattern where you can ensure that within a particular database

138
00:11:56,473 --> 00:12:02,904
transaction, you've got some stuff that you want to do, updates and rows, but you also
need to ensure that some side effect happens.

139
00:12:03,005 --> 00:12:08,268
And so having the queue within Postgres itself is really nice because you can just say,
all right.

140
00:12:08,268 --> 00:12:12,482
let's put this job into this table and then it'll be worked asynchronously eventually.

141
00:12:12,482 --> 00:12:17,546
And you don't have to worry about making sure that there's like any sort of delivery
coordination or anything like that.

142
00:12:17,546 --> 00:12:20,479
And there's a lot of ways you can do this, by the way.

143
00:12:20,479 --> 00:12:25,423
This is just how it was solved early on at Mercury and it just sort of persisted.

144
00:12:25,423 --> 00:12:28,655
But that in and of itself is kind of okay.

145
00:12:28,796 --> 00:12:33,741
But the problem is that our version of the queue kept creating all these new features.

146
00:12:33,741 --> 00:12:35,534
So first it was just, you know, as...

147
00:12:35,534 --> 00:12:41,494
just a bunch of insertions into a table that's like, here's the job name, here's the
payload.

148
00:12:41,514 --> 00:12:47,174
But then I was like, OK, well, some things need to be, some things can't run until some
other system has run.

149
00:12:47,174 --> 00:12:48,594
then we have to implement.

150
00:12:48,694 --> 00:12:49,934
FIFO keys.

151
00:12:50,414 --> 00:12:52,214
you have FIFO keys, all right?

152
00:12:52,214 --> 00:12:56,214
So anything that's got this key, gets processed from front to back.

153
00:12:56,214 --> 00:13:05,068
And then I was like, well, sometimes we can't run these things until, because like banking
requirements or whatever, we literally can't run this until it's next Monday.

154
00:13:05,068 --> 00:13:07,199
So then we added scheduling.

155
00:13:07,520 --> 00:13:09,301
and then what about the service going down?

156
00:13:09,301 --> 00:13:12,478
uh I guess we need to add retries.

157
00:13:12,478 --> 00:13:16,626
OK, but now it's retried a bunch, but it's still not working.

158
00:13:16,626 --> 00:13:18,362
So now we need to see what the error was.

159
00:13:18,362 --> 00:13:26,013
And there's kind of this gradual progression from what seems like a very simple system
into all these sort of exceptional cases.

160
00:13:26,013 --> 00:13:31,258
And each of those things does become kind of expensive for the database to maintain.

161
00:13:31,258 --> 00:13:33,080
And you end up having to implement.

162
00:13:33,080 --> 00:13:42,143
partial indexes and if you need to do some runtime debugging of what's going on and the
index isn't right and you've got a billion things in this table, well suddenly you've got

163
00:13:42,143 --> 00:13:46,555
a very slow query that times out before you can actually get meaningful information out.

164
00:13:46,555 --> 00:13:56,048
So yeah, it kind of turned into this thing where what we needed at some point really could
have been better serviced otherwise, but every single step along the way seemed reasonable

165
00:13:56,048 --> 00:13:57,580
to the person who was doing it.

166
00:13:57,580 --> 00:13:59,051
Yeah, for sure.

167
00:13:59,051 --> 00:14:02,053
It's the same way the state actors, they get you, right?

168
00:14:02,053 --> 00:14:13,051
It's like one small microcrime on top of another one until you're fully indebted and your
whole life will get tear to shreds if you decide otherwise.

169
00:14:13,051 --> 00:14:16,974
And it does seem reasonable at the time for what you know.

170
00:14:16,974 --> 00:14:26,370
And I think one of the struggles I've seen in my experience is the conversion to a
different technology or in our case, we're using SQS.

171
00:14:26,438 --> 00:14:30,641
It doesn't perfectly support every feature that you're currently implementing.

172
00:14:30,641 --> 00:14:39,888
so I feel like, and this is an engineering flawed problem, I think, where you always try
to have every feature in the new thing.

173
00:14:39,928 --> 00:14:47,804
And so you're like, well, if we can't migrate all of our queue dependencies from this
table to uh real service, then we're not going to do it at all.

174
00:14:48,118 --> 00:14:59,648
Yeah, well, I think for us, a big part of it was really people who had burned by sort of
earlier experiences with microservices and the like, or systems that just had a huge

175
00:14:59,648 --> 00:15:01,689
amount of stuff that you needed to run locally.

176
00:15:01,689 --> 00:15:05,623
There's a real heavy emphasis at Mercury on developer experience.

177
00:15:05,623 --> 00:15:12,879
So we've got a front-end developer experience, back-end developer experience as part of
sort of, well, uh we call it Side-UX.

178
00:15:12,879 --> 00:15:16,782
So it's stability, infrastructure, uh developer experience, basically.

179
00:15:16,782 --> 00:15:19,302
And that's sort of like all the non-product things.

180
00:15:19,782 --> 00:15:26,722
And yeah, there's really something to be said for like, all right, you're a growing
company, you've hired a new engineer, you shipped him a laptop.

181
00:15:26,722 --> 00:15:31,042
Can they just like have running environment in 20 minutes once they get the laptop?

182
00:15:31,042 --> 00:15:39,298
And that was really heavily prioritized, but it was really kind of perhaps overweight in
terms of the considerations at hand.

183
00:15:39,298 --> 00:15:40,899
No, I totally agree.

184
00:15:40,899 --> 00:15:45,650
I think that the reality is that neither extreme is accurate.

185
00:15:45,650 --> 00:15:56,113
The single monolith with a whole database cluster running to support all of the tables and
forcing an engineer to run basically the production system in order to do software

186
00:15:56,113 --> 00:15:57,373
development, that feels wrong.

187
00:15:57,373 --> 00:16:03,255
But it also feels wrong to run a thousand microservices in order to also do development at
the same time.

188
00:16:03,255 --> 00:16:06,722
And there's a perfect middle ground here that's different.

189
00:16:06,722 --> 00:16:12,322
per company where somehow every single company gets it wrong and makes this a challenge.

190
00:16:12,322 --> 00:16:20,733
Yeah, there's an interesting initiative on this front that we've got going on at Mercury
at the moment, which is adopting Buck 2, if you're familiar with it.

191
00:16:20,733 --> 00:16:22,291
Maybe you've heard of Basil.

192
00:16:22,291 --> 00:16:25,198
That's the monolith build system.

193
00:16:25,198 --> 00:16:28,978
It's like a monolithic build system that I think comes out of Google.

194
00:16:28,978 --> 00:16:33,598
Well, Buck and then subsequently Buck2 are basically Facebook's version of that.

195
00:16:33,598 --> 00:16:34,298
I see.

196
00:16:34,298 --> 00:16:42,918
So yeah, we've kind of entered the stage of the company where we're trying to stick with a
monolithic approach to things for the most part.

197
00:16:42,918 --> 00:16:47,518
basically, we've outgrown the standard kind of build system tools.

198
00:16:47,698 --> 00:16:54,574
I think that sort of gets to what you're talking about here, which is like, you don't want
to have to build stuff that you don't care about.

199
00:16:54,574 --> 00:16:56,274
or operate stuff that you don't care about.

200
00:16:56,274 --> 00:17:04,274
But you still kind of need global access to everything that your company could be doing or
would be doing, depending on the day.

201
00:17:04,554 --> 00:17:12,974
having the ability to sort of do these, have a monolith, but have it just build out
subsets of it instead of building the whole world, running all of the systems that it

202
00:17:12,974 --> 00:17:16,334
could possibly need to run, that's kind of what we're angling for.

203
00:17:16,334 --> 00:17:18,794
And it's a very hard thing to do.

204
00:17:18,794 --> 00:17:23,714
I think our development experience team has been working on this for two plus years.

205
00:17:23,714 --> 00:17:24,634
Oh, wow.

206
00:17:25,079 --> 00:17:30,882
where we're just kind of rounding the corner where it's like really starting to be like,
okay, this is actually usable for it.

207
00:17:30,882 --> 00:17:36,467
We're just two years away from the build system, for the developer experience, for
building on the machine.

208
00:17:36,467 --> 00:17:38,789
It's going to be solved in just two more years.

209
00:17:39,372 --> 00:17:42,763
I mean, that's what it's felt like for some time.

210
00:17:42,763 --> 00:17:45,024
actually, I mean, it does work now.

211
00:17:45,024 --> 00:17:46,264
So that's exciting.

212
00:17:46,264 --> 00:17:50,865
it's it's actually really remarkable once you have it.

213
00:17:50,865 --> 00:17:58,247
Because what's really great about it is that with Buck2, all of the build targets are sort
of hermetically sealed.

214
00:17:58,247 --> 00:18:02,729
They're reproducible in the same way that Nix kind of tries to achieve reproducibility.

215
00:18:02,729 --> 00:18:05,429
But it's on a much more granular level.

216
00:18:05,429 --> 00:18:07,854
So you can spin up like a really beefy.

217
00:18:07,854 --> 00:18:10,777
cluster of machines and fire off your build jobs.

218
00:18:10,777 --> 00:18:12,979
it can do massive parallelism.

219
00:18:12,979 --> 00:18:16,242
And because they're hermetic, they're reproducible.

220
00:18:16,242 --> 00:18:23,318
If one person builds a particular artifact and all the inputs are the same, then if you
try to build that same artifact, then it can just download it for you.

221
00:18:23,318 --> 00:18:24,409
You don't have to rebuild it.

222
00:18:24,409 --> 00:18:33,637
And those sorts of improvements in speed are ultimately going to be incredibly powerful
for any organization that's really not willing to descend into architecture madness that

223
00:18:33,637 --> 00:18:35,308
comes from microservices.

224
00:18:35,362 --> 00:18:42,495
So, you a long time ago, I was working at a company that had a similar process in their
custom build system to achieve this.

225
00:18:42,495 --> 00:18:49,929
It was running in Jenkins and actually was running something else before that, which was
also a mistake, ah but it was running in Jenkins at the time.

226
00:18:49,929 --> 00:19:01,134
And one of the challenges was that you did have jobs on top of jobs that would trigger
each other on builds and store the results in artifactory at that point to pull them back

227
00:19:01,134 --> 00:19:01,920
down.

228
00:19:01,920 --> 00:19:12,008
And rather than ending up with a nice, easily managed solution, which could have
individual components be cached without having to be rebuilt, we ended up with essentially

229
00:19:12,008 --> 00:19:21,846
a distributed monolith, where it was still all the benefits of the monolith and also all
of the disadvantages of microservices all wrapped into one.

230
00:19:21,886 --> 00:19:29,883
You couldn't really rebuild just the part you wanted, and you had to connect to the
artifactory to pull down individual components at runtime.

231
00:19:29,883 --> 00:19:31,552
And what you ended up with is

232
00:19:31,552 --> 00:19:36,354
a lot of individual functions basically were all being wrapped and published.

233
00:19:36,354 --> 00:19:47,059
So this one function in this one class would be its own sort of package which would get
pushed up and pulled down later and it just wasn't the right unit of granularity for what

234
00:19:47,059 --> 00:19:47,829
we were building.

235
00:19:47,829 --> 00:19:58,294
And it also encouraged this aspect where you would really define micro pieces of
functionality sort of like what you have today in any sort of mature package management

236
00:19:58,294 --> 00:19:59,052
system.

237
00:19:59,052 --> 00:20:07,127
where you end up with all these packages that depend on each other, but rather than having
development teams really focus on package management and dependency trees and what makes

238
00:20:07,127 --> 00:20:17,388
sense, you just ended up with, it was so easy to hook up to the system that you published
a lot of these micro packages without thinking about what makes sense as a library.

239
00:20:17,388 --> 00:20:19,180
Yeah, that's a really interesting point.

240
00:20:19,180 --> 00:20:23,663
So in terms of Buck2, I mean, it's sort of like if you write a make file, right?

241
00:20:23,663 --> 00:20:28,547
Let's think about a traditional C library that you're going to build or something like
that or executable.

242
00:20:28,547 --> 00:20:38,575
And you write a make file and it's got these targets and you're defining sort of a
dependency graph effectively where it's like in order to get a .o file, you have to

243
00:20:38,575 --> 00:20:41,257
compile a .c file and yada yada.

244
00:20:41,257 --> 00:20:43,329
And it kind of figures out how to resolve all that stuff.

245
00:20:43,329 --> 00:20:47,202
That's kind of the same thing that Buck2 does, but it just provides sort of like

246
00:20:47,202 --> 00:20:54,968
the hermetically sealed bit of it so that way you're guaranteed that, you know, basically
it's hashing the inputs and the outputs and it's kind of making sure that everything that

247
00:20:54,968 --> 00:20:56,500
needs to be rebuilt is rebuilt.

248
00:20:56,500 --> 00:20:58,391
And this has some really nice knock-on effects.

249
00:20:58,391 --> 00:21:08,930
So for example, if you have tests, like if you have a module that has a bunch of tests in
it for CI, uh if you change anything that causes that to be rebuilt, then it reruns the

250
00:21:08,930 --> 00:21:09,380
tests.

251
00:21:09,380 --> 00:21:16,588
But if like the targeted change that you're making doesn't actually affect that part of,
that other part of the tree at all, then

252
00:21:16,588 --> 00:21:21,082
you already know those test paths effectively, as long as your tests are sufficiently
isolated.

253
00:21:21,082 --> 00:21:24,705
there's some nice knock-on benefits there.

254
00:21:25,966 --> 00:21:35,334
in terms of packaging, that's actually the issue that we ran into, is that we have a bunch
of local packages that we have to manage currently with the traditional built-like

255
00:21:35,334 --> 00:21:37,916
packaging package manager for Haskell.

256
00:21:37,916 --> 00:21:44,598
So let's say that you've got a utility package that's way up in the dependency tree, and
you slightly tweak it.

257
00:21:44,598 --> 00:21:49,841
and then you've got 30 other packages that depend on that package, then those all have to
rebuild and everything underneath it.

258
00:21:49,841 --> 00:21:51,782
And so there's this huge cascade of rebuilds.

259
00:21:51,782 --> 00:21:53,483
And that's kind of the problem that you have to solve.

260
00:21:53,483 --> 00:21:59,167
Whereas if you're using something like buck two, that's doing literally module level
rebuilds.

261
00:21:59,167 --> 00:22:05,830
So like if you change one file, it's only having to rebuild everything that's transitively
dependent on that file.

262
00:22:05,891 --> 00:22:11,374
Then you don't have these huge like two hour coffee breaks because you're waiting for
your.

263
00:22:11,374 --> 00:22:13,516
for your local system to rebuild all these packages.

264
00:22:13,516 --> 00:22:18,261
It's able to just do a targeted traversal through what needs to actually be rebuilt.

265
00:22:18,462 --> 00:22:27,062
so breaking apart the concept of packages and going entirely granular gives you massive
improvements in terms of build parallelism.

266
00:22:27,062 --> 00:22:29,204
And that's a thing that you really have to solve now.

267
00:22:29,766 --> 00:22:40,863
I'm just having some trauma revisiting from the days of old where a similar thing had
attempted to be done, but really just ended up in a more complex diamond dependency

268
00:22:40,863 --> 00:22:45,917
problem, uh which is like, you have two different packages that depend on different
versions of the thing.

269
00:22:45,917 --> 00:22:48,758
And then at runtime, there is no way to resolve the right thing.

270
00:22:48,758 --> 00:22:54,642
But what we actually ended up with uh in practice was this scenario where

271
00:22:54,658 --> 00:23:03,541
You would have some shared dependency between two different services or components that
would change, as you mentioned, and it would get rebuilt.

272
00:23:03,541 --> 00:23:10,403
But we would determine that that function was only used in one place because statically it
was determined it was only used in one place.

273
00:23:10,403 --> 00:23:16,664
But it turned out that dynamically at runtime, it was used in the other place and that
thing wouldn't get rebuilt.

274
00:23:16,664 --> 00:23:22,796
So it was targeting a different binary compatibility than the one that was actually found
at runtime.

275
00:23:22,796 --> 00:23:32,245
And unlike some dependency management systems which allow you to solve this diamond
dependency problem by including both versions of the dependency at runtime, some languages

276
00:23:32,245 --> 00:23:38,761
like C, C++, C sharp, et cetera, most of them realistically only let you get one version
of the binary dependency.

277
00:23:38,761 --> 00:23:41,223
And so you would just end up with a runtime crash.

278
00:23:41,223 --> 00:23:48,669
Now I think that sort of thing is impossible with Haskell, but it definitely is possible
with dynamically linked libraries in say Java and C sharp.

279
00:23:54,424 --> 00:23:55,178
Yeah

280
00:23:57,510 --> 00:23:59,531
Yeah, no, no, for sure, for sure not.

281
00:23:59,531 --> 00:24:01,672
I can tell you this is a thing that did happen.

282
00:24:01,672 --> 00:24:12,126
Whether or not it's solved now is sort of a different question, but I found that this is
sort of the result of the primary builder, whatever that script is or executable for that

283
00:24:12,126 --> 00:24:21,200
language, not offering this functionality by default and trying to build it on top, like a
second level where it doesn't fully understand how the builds work and then trying to

284
00:24:21,200 --> 00:24:22,701
compensate for that.

285
00:24:22,701 --> 00:24:26,272
And I think this is, I don't know, my clue would be...

286
00:24:26,272 --> 00:24:31,255
potentially we should pick a different technology or whatnot, but maybe there it is a
solved problem in this case.

287
00:24:31,718 --> 00:24:32,719
I don't know.

288
00:24:32,941 --> 00:24:33,602
Could be.

289
00:24:33,602 --> 00:24:43,274
I think one of the interesting things is that, um at least the challenge that I found, is
that every ecosystem ends up having to build its own build system, right?

290
00:24:43,274 --> 00:24:43,742
Yeah.

291
00:24:43,742 --> 00:24:56,572
uh It's a remarkably challenging thing if you want to have a polyglot system because you
have that many package managers on top of it with their own quirks, their own dependency

292
00:24:56,572 --> 00:24:58,933
resolution algorithms, and yada yada.

293
00:24:58,933 --> 00:25:02,178
And so doing any sort of serious integration across

294
00:25:02,178 --> 00:25:06,100
multiple languages can become remarkably painful.

295
00:25:06,100 --> 00:25:18,647
And so there is a compelling aspect of saying, all right, in a monolith, we're using this
one build system and then letting teams write some rules, plugins, so to speak, that kind

296
00:25:18,647 --> 00:25:27,932
of map their particular linguistic concepts onto a single build system so that engineers
who you like, all right, for example, we send emails.

297
00:25:28,533 --> 00:25:30,197
Most companies send emails.

298
00:25:30,197 --> 00:25:31,051
Shots.

299
00:25:31,458 --> 00:25:32,838
They're templated, right?

300
00:25:32,838 --> 00:25:36,259
And um we want them to match our design system.

301
00:25:36,259 --> 00:25:41,111
so consequently, because we write our front end in React, engineers now react.

302
00:25:41,111 --> 00:25:43,641
So we're using this React email templating thing.

303
00:25:43,641 --> 00:25:45,082
I love where this is going.

304
00:25:45,082 --> 00:25:51,584
Yeah, but the thing is sending emails happens on the back end, which is where all our
Haskell developers live.

305
00:25:51,584 --> 00:25:56,225
And our Haskell developers really don't want to fucking deal with NPM, right?

306
00:25:56,225 --> 00:25:58,305
That's just like this whole can of worms.

307
00:25:58,305 --> 00:26:00,146
They don't want to have to mess with that.

308
00:26:00,770 --> 00:26:05,814
Yeah, this ability to provide a common interface across a monolith, I think, is quite
compelling.

309
00:26:05,814 --> 00:26:09,076
So that way you can just say, build me the email templates.

310
00:26:09,076 --> 00:26:10,107
And that's that.

311
00:26:10,107 --> 00:26:11,958
You don't have to think so hard about it.

312
00:26:11,958 --> 00:26:14,480
But I want to say this before we move on.

313
00:26:14,480 --> 00:26:15,941
I mentioned we use Nix.

314
00:26:15,941 --> 00:26:17,702
And Nix is remarkably powerful.

315
00:26:17,702 --> 00:26:19,774
um And now we're using Buck2.

316
00:26:19,774 --> 00:26:21,245
And Buck2 is remarkably powerful.

317
00:26:21,245 --> 00:26:24,270
They both provide incredible benefits to an organization.

318
00:26:24,270 --> 00:26:26,388
I still hate using them.

319
00:26:28,160 --> 00:26:33,652
Like, the actual uh user experience of these tools is really brutal.

320
00:26:33,652 --> 00:26:43,616
So I want to say, as a counterpoint to myself, that if you're not a company with 200-plus
engineers, or if you don't have existing experience in this stuff, like Knicks, there's

321
00:26:43,616 --> 00:26:48,699
probably an argument that you should just bite the bullet and use it, because it's
incredibly powerful.

322
00:26:48,699 --> 00:26:50,209
It still sucks to use.

323
00:26:50,209 --> 00:26:56,702
Any day where I have to mess with these tools for more than about five minutes is a day
poorly spent for me.

324
00:26:56,702 --> 00:26:57,142
Yeah.

325
00:26:57,142 --> 00:27:08,898
I guess on the subject of novelty budget, sometimes you have these things that have really
great payoffs when you actually have it all set up, but they're still painful.

326
00:27:09,039 --> 00:27:13,710
A lot of wasted time in your past is what I'm hearing here.

327
00:27:13,958 --> 00:27:16,057
I can appreciate that.

328
00:27:16,070 --> 00:27:18,793
I mean, it's the eat your Brussels sprouts thing, right?

329
00:27:18,793 --> 00:27:21,096
Like, no kid wants to their Brussels sprouts.

330
00:27:21,096 --> 00:27:25,941
uh It's uh good for you, but maybe not enjoyable.

331
00:27:27,234 --> 00:27:38,658
You know, I've tried to cook Brussels sprouts so many ways and at one point I thought I
liked them, but having migrated to so many other vegetables and root vegetables that I, or

332
00:27:38,658 --> 00:27:46,681
leaf vegetables in my, in the last 30 years or so, I have to say I still can't make
Brussels sprouts taste good.

333
00:27:46,681 --> 00:27:50,002
So maybe there's an argument for not eating them.

334
00:27:50,713 --> 00:27:56,272
Yeah, you know, um wow, we're gonna get really off topic here, but cooking is my other
passion, so.

335
00:27:57,614 --> 00:28:01,374
Maybe we can hold off the cooking one.

336
00:28:01,374 --> 00:28:02,134
we come back to it.

337
00:28:02,134 --> 00:28:10,520
I mean, you can make a cooking comment, but I need to hear it have some connection to the
conversation.

338
00:28:10,520 --> 00:28:12,890
It has no connection, so I guess that's me one.

339
00:28:12,906 --> 00:28:16,818
You can always try to connect this back in the end.

340
00:28:16,818 --> 00:28:20,279
Yeah, right.

341
00:28:20,279 --> 00:28:22,109
There's always a point here.

342
00:28:22,150 --> 00:28:23,930
You can make them taste good, maybe.

343
00:28:23,930 --> 00:28:33,154
I do want to get to one last thing, I think, before we close out the episode, and that's
your take on what I would call control planes.

344
00:28:33,678 --> 00:28:41,858
I think that you brought this up in your blog articles, I think a couple of them
realistically, and it's like one of those phrases from my experience where I didn't really

345
00:28:41,858 --> 00:28:44,238
pay a lot of attention to historically.

346
00:28:44,238 --> 00:28:51,758
So it's similar to like software-defined networking in practice, but once I got to the
cloud, I felt like actually understanding networking is now critical.

347
00:28:51,758 --> 00:28:53,218
I need to understand how DNS works.

348
00:28:53,218 --> 00:28:56,698
I need to understand how load balancers work and packet switching, et cetera.

349
00:28:56,698 --> 00:28:59,966
I may not love it, and software may make it easier, but...

350
00:28:59,966 --> 00:29:09,388
The other one is like control planes and it's that thing where if you're in a cloud or
running in some way, it suddenly becomes a useful phrase for communication.

351
00:29:09,388 --> 00:29:18,352
Yeah, so I think I said this in my blog posts as well, but the way that I use it is
probably some people, some purists are probably going to say that this is like a really

352
00:29:18,352 --> 00:29:19,592
twisted definition.

353
00:29:19,592 --> 00:29:27,126
But the way that I think of it is basically you've got your software, your main product,
your main system that's running.

354
00:29:27,126 --> 00:29:36,992
And it is easy and tempting to basically bake in all of the management that you need to
run your software into the software itself.

355
00:29:36,992 --> 00:29:46,477
So, you know, for example, a lot uh of businesses have like admin pages where you log in
and you do your admin things and it's like some portion of the executable that you built

356
00:29:46,477 --> 00:29:49,139
and it's like a subsystem of, you know, this thing that you ship.

357
00:29:49,139 --> 00:29:58,233
The problem with that is that when you have incidents, when you have outages that are
taking down your system, you run into a circular problem where the tools that you need to

358
00:29:58,233 --> 00:30:01,595
fix your system are not available because they're baked into your system.

359
00:30:01,595 --> 00:30:06,700
And so there's a lot of systems that you're probably using on a daily basis that

360
00:30:06,700 --> 00:30:07,561
have control planes.

361
00:30:07,561 --> 00:30:09,772
Like a lot of people use Kubernetes, for example.

362
00:30:09,772 --> 00:30:18,358
And Kubernetes, I think, is probably like the canonical example that everybody's familiar
with at this point, where there's some services that you're running through your product,

363
00:30:18,358 --> 00:30:21,570
but then you've got this thing that's in charge of resource allocation.

364
00:30:21,570 --> 00:30:29,986
It's in charge of making sure that the things that are supposed to talk to each other can,
and that it's in charge of how they connect to the rest of the world, all that.

365
00:30:29,986 --> 00:30:32,267
That is a control plane.

366
00:30:32,267 --> 00:30:36,184
But you probably often, also when I think about having a control plane,

367
00:30:36,184 --> 00:30:37,655
for your product itself.

368
00:30:37,655 --> 00:30:40,558
So as I mentioned, we got this thing called PostgreSQL.

369
00:30:40,558 --> 00:30:44,000
And so here's where I can talk a little bit about PostgreSQL.

370
00:30:44,000 --> 00:30:52,467
For convenience, we ended up building a management interface for retrying PostgreSQL jobs
that failed or seeing what the failures were.

371
00:30:52,467 --> 00:30:59,433
Problem is that we got to a point on Sundays where PostgreSQL was taking up 80 % of our
database resources, right?

372
00:30:59,433 --> 00:31:04,817
And that would cause login on our site to fail because it took too long to log in.

373
00:31:04,817 --> 00:31:06,088
I know where this is going.

374
00:31:06,208 --> 00:31:11,022
So you can't manage your system if you can't log into the system.

375
00:31:11,022 --> 00:31:15,066
So really, what we should have been doing, I mean, there's a number of lessons to take
from this.

376
00:31:15,066 --> 00:31:18,068
I'm not going to try to enumerate all of them.

377
00:31:18,389 --> 00:31:20,191
Every system has its growing pains.

378
00:31:20,191 --> 00:31:31,260
But what we should have done is had behind a VPN a control plane for PostgreSQL that its
sole job was make sure that you're an employee, make sure that you pass, you can get

379
00:31:31,260 --> 00:31:33,154
through Okta or whatever, and then

380
00:31:33,154 --> 00:31:40,337
give you the things that you need to do and have it not be tied to all of the other domain
level concerns of the primary system.

381
00:31:40,337 --> 00:31:50,342
And so that's my pitch for control planes in a nutshell is the things that you need to do
to operate a system cannot live with the system itself or you are going to hate yourself

382
00:31:50,342 --> 00:31:51,146
at some point.

383
00:31:51,146 --> 00:31:55,459
See, I feel like you're a secret advocate for microservices, just like me.

384
00:31:55,459 --> 00:32:07,018
You want your login and access control to be a separate system so that issues with another
part of the system can't impact it negatively, have separate databases, separate compute

385
00:32:07,018 --> 00:32:16,185
and running there so that containers being brought down or virtual machines having an
issue in a particular region don't cause a cascading failure for your entire system.

386
00:32:16,185 --> 00:32:17,568
Now, I think you're going to...

387
00:32:17,568 --> 00:32:20,755
say you're not and it's still going to be the model, that's the strategy.

388
00:32:20,755 --> 00:32:25,026
But honestly, think the access control is a good example there.

389
00:32:25,026 --> 00:32:35,629
Well, what I'll say is I'm an advocate for the concept of bulkheading, which is to say
that if a particular portion of your system fails, it shouldn't be cascading into the rest

390
00:32:35,629 --> 00:32:36,609
of your system.

391
00:32:36,609 --> 00:32:39,086
And microservices are...

392
00:32:39,086 --> 00:32:40,366
Okay, here's what I'll say.

393
00:32:40,366 --> 00:32:42,811
I say this at my company all the time.

394
00:32:42,811 --> 00:32:44,311
I'm not opposed to services.

395
00:32:44,311 --> 00:32:45,782
I'm opposed to microservices.

396
00:32:45,782 --> 00:32:48,472
What I think we should have is appropriately sized services.

397
00:32:48,472 --> 00:32:54,946
uh Maybe they're not broken down to a hypergranular level, but...

398
00:32:54,946 --> 00:33:00,959
But one thing that I'll say is that you can achieve a service-oriented architecture with a
monolith at the same time.

399
00:33:00,959 --> 00:33:13,435
What that means, for example, in a really easy world is if you're behind AWS, like uh
application load balancer or something, allocate some EC2 instances or some containers to

400
00:33:13,435 --> 00:33:16,036
individual teams or individual product domains.

401
00:33:16,036 --> 00:33:18,778
And it can still be running the whole monolithic build.

402
00:33:18,778 --> 00:33:19,862
It can still be...

403
00:33:19,862 --> 00:33:29,138
like a single coherent view of the world, but the resources that it's using like database
connections or what have you are scoped to a particular product domain.

404
00:33:29,138 --> 00:33:37,033
So that way if some team comes along and accidentally uses up your entire connection pool,
you know, on a per instance basis, that's their problem.

405
00:33:37,033 --> 00:33:38,982
It's not the whole compliance problem.

406
00:33:39,766 --> 00:33:43,751
We should have definitely had this be a fight about microservices versus Model S.

407
00:33:43,751 --> 00:33:46,414
think we could have gotten really far on that one.

408
00:33:46,414 --> 00:33:47,414
Well, you can have me back.

409
00:33:47,414 --> 00:33:48,554
That's fine.

410
00:33:50,134 --> 00:33:54,734
I want to mention, I mentioned briefly that we were kind moving off of Postgres to you.

411
00:33:54,734 --> 00:34:02,473
And most of what we're moving onto from that is a system called Temporal, which is a
durable execution framework.

412
00:34:02,473 --> 00:34:04,254
And I talked about that in some of my blog posts.

413
00:34:04,254 --> 00:34:11,414
But the concept of durable execution frameworks is a thing that I think is starting to
gain a little bit of steam.

414
00:34:11,414 --> 00:34:14,906
we're starting to see a few competing products in the space.

415
00:34:14,906 --> 00:34:27,053
And it's really nice because it's uh basically what it does is you have workflows or you
have particular pieces of business logic that need to definitely execute from start to

416
00:34:27,053 --> 00:34:27,913
finish.

417
00:34:27,913 --> 00:34:31,345
And you want to be resilient to uh failures.

418
00:34:31,345 --> 00:34:35,737
You want to deal with like what happens if your process crashes halfway through.

419
00:34:35,977 --> 00:34:44,814
typically you have to call out to like APIs or do some database, multiple database
transactions, and you don't want to leave the system in an inconsistent state.

420
00:34:44,814 --> 00:34:50,478
And basically, this concept of durable execution frameworks is they record sort of the
steps that you're doing along the way.

421
00:34:50,478 --> 00:35:01,094
And if the process crashes or there's a logic bug and you ship a fix for it later, then
it's able to use the serialized results of what it's done so far to kind time travel back

422
00:35:01,094 --> 00:35:05,266
to where you were and pick up where you left off and execute to completion.

423
00:35:05,266 --> 00:35:09,749
And um I don't know, it just really uh scratches and itch for me.

424
00:35:10,014 --> 00:35:11,392
And so.

425
00:35:11,392 --> 00:35:16,258
Obviously we can't talk about it for very long, I want to encourage listeners to go read
about it.

426
00:35:16,258 --> 00:35:18,500
Well, actually, it's really interesting that you brought this up for two points.

427
00:35:18,500 --> 00:35:25,215
uh I think AWS has been full on in the last few months since re-invent introducing durable
functions into Lambda.

428
00:35:25,215 --> 00:35:31,870
It's the sort of thing that we have state functions and workflows existing in all the
cloud providers, but I think...

429
00:35:31,882 --> 00:35:42,088
Azure was first and I don't think there's an option in GCP, but we do have sort of native
implementations now that allow you to execute some code and give hints to the

430
00:35:42,088 --> 00:35:50,113
orchestration layer on how to retry that function when it fails and not just retry from
the beginning and have to implement item potency everywhere, et cetera, and how to store

431
00:35:50,113 --> 00:35:51,694
state and check for that.

432
00:35:51,694 --> 00:35:59,104
I think the poor man's version of durable functions is like throw a message in SQS and
have it execute your code, like.

433
00:35:59,104 --> 00:36:05,700
every single step of there is item potent and it just executes it and then goes on to the
next one and maybe there's some logic to skip there.

434
00:36:05,700 --> 00:36:09,843
But it'd be so nice if there was a framework and I think as you said, we're starting to
see companies spin up.

435
00:36:09,843 --> 00:36:18,390
It's interesting you mentioned Temporal though because in a couple of weeks we will have
VP of Engineering from Temporal on the podcast actually to talk about this.

436
00:36:18,390 --> 00:36:27,757
So you can spoil yourself if you want and read about it ah or you can wait for that
episode and see what shenanigans we get up to on there.

437
00:36:27,970 --> 00:36:34,018
the privilege of talking at their conference last year a little bit about our learnings,
adopting it at Mercury.

438
00:36:34,018 --> 00:36:37,041
So I'm a fan boy to a certain extent.

439
00:36:38,248 --> 00:36:43,321
Okay, well then with that, we'll skip the tangents and move on to picks for the episode.

440
00:36:43,321 --> 00:36:45,526
So, Anne, what did you bring for us today?

441
00:36:45,526 --> 00:36:53,872
Yes, my pick for the episode is I want to recommend a band called Glory Hammer, which is a
power metal band.

442
00:36:53,872 --> 00:36:57,345
I'm a relatively recent power metal enjoyer.

443
00:36:57,345 --> 00:37:06,041
I'll say a lot of it I don't like, I think they're really fun as a band because they lean
into absurdity.

444
00:37:06,041 --> 00:37:08,884
Like they know that they're in a goofy genre.

445
00:37:08,884 --> 00:37:10,525
And how do I say it?

446
00:37:10,525 --> 00:37:11,685
I find their...

447
00:37:12,398 --> 00:37:14,398
lyrical choices to be really interesting.

448
00:37:14,398 --> 00:37:19,158
They do a lot of things that are incoherent with their lyrics.

449
00:37:19,158 --> 00:37:25,218
So for example, they'll say they cast the evil wizard into a tomb of liquid ice.

450
00:37:25,218 --> 00:37:27,698
you sit there and you're like, all right, awesome lyric.

451
00:37:27,698 --> 00:37:29,558
And then you're like, wait, liquid ice?

452
00:37:29,558 --> 00:37:30,538
Like water?

453
00:37:31,118 --> 00:37:32,238
Is it really?

454
00:37:32,318 --> 00:37:33,278
You know what I mean?

455
00:37:33,278 --> 00:37:35,638
Is the evil wizard actually trapped in there?

456
00:37:35,638 --> 00:37:36,498
So I don't know.

457
00:37:36,498 --> 00:37:38,858
It's just kind of like, they have a lot of fun with it.

458
00:37:38,858 --> 00:37:42,262
And I think it kind of captures that feeling of

459
00:37:42,262 --> 00:37:52,062
a really good fantasy show or like a really good video game in terms of like, but they're
just turning like the whole epic, epicness as a concept up to like 11.

460
00:37:52,062 --> 00:37:53,765
And it's just a fun listen.

461
00:37:53,765 --> 00:37:54,726
uh

462
00:37:54,848 --> 00:38:07,442
I am not the biggest music buff, back when it was still a thing, honestly, was it like uh
Nightwish or uh Metalocalypse, the Death Clock?

463
00:38:07,442 --> 00:38:11,564
know, they're the weird songs that definitely hit the fantasy angle.

464
00:38:11,564 --> 00:38:16,936
I don't know what this metal, power metal means though.

465
00:38:16,936 --> 00:38:18,400
what is that as a genre?

466
00:38:18,400 --> 00:38:19,803
Uh, that's...

467
00:38:20,147 --> 00:38:23,012
I'm sure somebody else could say it better than me, but...

468
00:38:23,012 --> 00:38:26,922
We'll get a link to it then in the episode description.

469
00:38:27,042 --> 00:38:33,187
I guess what I would say is it tends to introduce a lot of symphonic elements into it.

470
00:38:33,187 --> 00:38:35,138
They'll have organs or choirs.

471
00:38:35,138 --> 00:38:39,231
They'll have, obviously, the traditional guitars and all that jazz.

472
00:38:39,231 --> 00:38:48,338
But it kind of, I think, goes into eh a lot of the artists really lean very heavily on to
fantasy or sci-fi.

473
00:38:48,338 --> 00:38:54,954
And I think as compared to normal metal, where there's a bit of a focus on just how it
sounds, there's normal.

474
00:38:54,954 --> 00:38:55,270
of

475
00:38:55,270 --> 00:39:00,410
than power metal artists that I've really gotten into, I kind of have themes to them, I
guess you could say.

476
00:39:00,410 --> 00:39:04,430
So for example, there's another band that I've come to enjoy called Windrose.

477
00:39:04,430 --> 00:39:14,230
You may have heard them or heard some of their songs, but basically imagine Lord of the
Rings dwarves playing metal music about being dwarves.

478
00:39:15,150 --> 00:39:17,410
And there's another band called Power Wolf.

479
00:39:17,410 --> 00:39:21,630
They kind of do the same thing for vampires, or not vampires, but they do it for
werewolves.

480
00:39:22,326 --> 00:39:28,476
And so they kind of just lean really hard into these kind thematic elements.

481
00:39:28,476 --> 00:39:29,477
It's fun.

482
00:39:29,798 --> 00:39:37,614
I don't really go in for the death metal or anything, but just something that kind of
channels that fantasy vibe when I'm working is really nice.

483
00:39:37,614 --> 00:39:41,876
Okay, so Ann's recommendation will be in the podcast description for the episode.

484
00:39:41,944 --> 00:39:43,062
out description.

485
00:39:43,062 --> 00:39:44,483
Okay, I think this is a great pick.

486
00:39:44,483 --> 00:39:46,243
ah So thanks for that.

487
00:39:46,243 --> 00:39:48,664
uh I guess mine's lame in comparison.

488
00:39:48,664 --> 00:39:55,127
A couple of weeks ago, I uh had a pick about why archers don't fire volleys.

489
00:39:55,127 --> 00:39:58,088
It's not a real thing, never happened, never will happen.

490
00:39:58,088 --> 00:40:03,470
Every reenactment in any movie or television show is absolutely wrong.

491
00:40:03,610 --> 00:40:11,573
But on the same collection website, there is an article on battle logistics and how they
actually do it.

492
00:40:12,970 --> 00:40:16,813
I love this collection of articles from the Roman age, specifically warfare.

493
00:40:16,813 --> 00:40:20,877
Going to war meant literally moving your city into combat.

494
00:40:20,877 --> 00:40:23,078
That is the way that I would describe it.

495
00:40:23,139 --> 00:40:30,915
If you go to combat, you pretty much have to take all the people, whatever they're doing,
not just the combatants, into combat and move them.

496
00:40:30,915 --> 00:40:34,197
And how would you manage traveling as a city together?

497
00:40:34,197 --> 00:40:37,730
There's lots of non-combatants, everything people need to survive.

498
00:40:37,730 --> 00:40:40,493
It's still true in any sort of war zone.

499
00:40:40,493 --> 00:40:41,674
Nothing really changes.

500
00:40:41,674 --> 00:40:42,432
And it's just...

501
00:40:42,432 --> 00:40:48,497
It's really interesting to see how the author talks about it and realistically what
actually happens in these scenarios.

502
00:40:48,497 --> 00:40:57,548
It's like this area in a lot of popular culture that you just don't see movies or
television shows really ever get into at all.

503
00:40:57,548 --> 00:41:07,370
Yeah, I suppose it's kind of the, what makes for good cinema, Nobody wants to see a bunch
of people like, cheffing and repairing things.

504
00:41:07,638 --> 00:41:11,300
I mean, maybe I think they do sort of get about this a little bit.

505
00:41:11,300 --> 00:41:21,146
Like I'm not a huge Game of Thrones fan, but there are a lot of like the army's moving and
you do see like what happens at night within the guard camps or whatever and stuff is

506
00:41:21,146 --> 00:41:22,426
going on there.

507
00:41:22,426 --> 00:41:26,969
And you can sort of extrapolate like, well, if this is happening, these other things are
probably happening.

508
00:41:26,969 --> 00:41:28,150
But it makes a lot of good points.

509
00:41:28,150 --> 00:41:29,680
Like, well, where do you get food from?

510
00:41:29,680 --> 00:41:33,693
And like, how do you transport it from where they are to where they're going?

511
00:41:33,693 --> 00:41:36,034
If you're going on like a three month journey.

512
00:41:36,034 --> 00:41:37,865
How much food can you bring on your person?

513
00:41:37,865 --> 00:41:49,083
And then you have this issue of, well, I guess you can have some animals carry a cart with
food on it, but how much food do the animals need to consume in order to move the cart?

514
00:41:49,083 --> 00:41:52,665
And so there's like a maximum limit that you can actually have.

515
00:41:52,665 --> 00:41:56,668
Same thing goes for, you know, what happens when your armor breaks?

516
00:41:56,668 --> 00:41:58,869
If you're wearing some sort of armor, who fixes that?

517
00:41:58,869 --> 00:42:00,682
Do you bring blacksmiths with you?

518
00:42:00,682 --> 00:42:02,363
But what a blacksmith need to survive?

519
00:42:02,363 --> 00:42:05,864
Well, they also need like giant forges for their profession.

520
00:42:05,864 --> 00:42:09,946
And obviously they need food, too, and welding materials.

521
00:42:09,946 --> 00:42:11,667
And where do they get that from?

522
00:42:11,667 --> 00:42:13,707
And what do they do when everyone's in combat?

523
00:42:13,707 --> 00:42:15,018
Do they just sit there waiting?

524
00:42:15,018 --> 00:42:16,589
Do they just work all the time?

525
00:42:16,589 --> 00:42:19,130
They probably need rest and relaxation.

526
00:42:19,130 --> 00:42:25,812
So I know this feels like a bit pedantic, but the collection is actually called
Unmitigated Pedantry.

527
00:42:25,812 --> 00:42:30,198
And the thing that I really like about it, if anyone knows me, is that

528
00:42:30,198 --> 00:42:35,627
I find pedantry like exceptionally interesting, especially when gone into extreme detail.

529
00:42:35,627 --> 00:42:37,206
It's where the nuance lives.

530
00:42:37,206 --> 00:42:48,652
It makes me curious now that you mention it, because whenever you see sieges portrayed in
uh fiction, it's kind of taken for granted that the people that are trying to trap people

531
00:42:48,652 --> 00:42:52,595
inside of a city or inside of a castle or what have you kind of have all the time in the
world.

532
00:42:52,595 --> 00:42:58,778
But I suppose what you're mentioning here does sort of raise the question of how do they
sustain that, right?

533
00:42:58,914 --> 00:43:09,002
So this is where you can imagine realistically, and I don't know if he talks about this
one, but like a siege, you need to have the people performing the siege get supplies.

534
00:43:09,002 --> 00:43:12,665
And so you can imagine a supply line is a real thing that has to be managed.

535
00:43:12,665 --> 00:43:18,990
But how far away are they from their city, you know, when they came there and how long can
they actually lay siege?

536
00:43:18,990 --> 00:43:24,605
You can imagine if you want to take a towered or defended city, you don't need to actually
bring the walls down.

537
00:43:24,605 --> 00:43:28,610
You can just destroy all of the supply lines that go into the city and eventually.

538
00:43:28,610 --> 00:43:30,171
This is like war of attrition.

539
00:43:30,171 --> 00:43:31,152
You'll win.

540
00:43:31,152 --> 00:43:33,614
And so you sort of have this strategy.

541
00:43:33,614 --> 00:43:38,158
And this is where like alliances are born because like, if we're being attacked, where is
our army?

542
00:43:38,158 --> 00:43:42,222
uh Where are we getting goods from to sustain the population of that city?

543
00:43:42,222 --> 00:43:44,243
But yeah, it goes the other way too, right?

544
00:43:44,243 --> 00:43:53,011
Like where do you get the arrows and whatever, sledgehammers or battle rams, et cetera, in
order to even do the attack in the first place.

545
00:43:53,011 --> 00:43:56,774
So you need to sustain that part of the army as well.

546
00:43:56,854 --> 00:44:01,369
If you're very far away from your source location, well, you know, there's a good luck
there.

547
00:44:01,369 --> 00:44:04,174
And I think that's where the term living off the land comes from.

548
00:44:04,174 --> 00:44:09,114
Well, it sounds like you've got a really good candidate for a Steam game, right?

549
00:44:09,554 --> 00:44:13,714
The kind of people that like farming simulator or that sort of thing.

550
00:44:13,714 --> 00:44:18,994
think Supply Line Manager 1800 might be a big hit.

551
00:44:19,755 --> 00:44:20,695
You know what?

552
00:44:20,695 --> 00:44:23,117
At this point, I'm sure it is already a game.

553
00:44:23,117 --> 00:44:28,300
uh Maybe there is something out there and I can just go and find it.

554
00:44:28,300 --> 00:44:33,424
But I think it's interesting to play it and more interesting to sort of understand the
challenges.

555
00:44:33,424 --> 00:44:35,786
Now, maybe someone else is more into games than me.

556
00:44:35,786 --> 00:44:41,550
I know I'm a self-proclaimed gamer, but I think in reality, I play only the very few
puzzle games.

557
00:44:41,550 --> 00:44:45,232
Well, thank you, Anne, for coming on for this episode of Adventures in DevOps.

558
00:44:45,232 --> 00:44:48,298
I absolutely have loved the discussion and everything.

559
00:44:48,298 --> 00:44:55,712
CICD and thanks to the listeners for tuning in for this episode and hopefully we'll see
everyone back next week.