1
00:00:07,810 --> 00:00:11,473
Welcome back to another episode of Adventures in DevOps.

2
00:00:11,473 --> 00:00:16,056
before we get to the show, I just want to introduce today's sponsor, who is Rootly AI.

3
00:00:16,056 --> 00:00:23,740
I really appreciate the team for both being a great guest on the show in a previous
episode, as well as sponsoring this week's episode.

4
00:00:23,740 --> 00:00:30,572
Just one of the interesting things going on with their AI Labs org is their upcoming
release of the On-Call Burnout Detector.

5
00:00:30,572 --> 00:00:35,706
It's a free open source tool for detecting early signs of overwork in incident responders
and engineers.

6
00:00:35,706 --> 00:00:41,431
Besides that though, I don't want to spoil anything more about it, so check it out and as
always, there'll be a link in the description.

7
00:00:41,431 --> 00:00:43,372
And now back to the show.

8
00:00:43,372 --> 00:00:51,239
I met our guests at a recent conference and after hearing them tell story after story, I
just couldn't wait to bring them onto this episode.

9
00:00:51,239 --> 00:00:55,630
So I want to welcome today's guest, Paul Conroy, CTO at Square One.

10
00:00:55,630 --> 00:00:56,113
Thanks, Milo.

11
00:00:56,113 --> 00:00:56,944
Thanks for having me.

12
00:00:56,944 --> 00:00:58,164
It's good to be here.

13
00:00:59,016 --> 00:01:02,266
You know, it's a web agency if I've got that right.

14
00:01:02,266 --> 00:01:05,780
I don't know if you use a different terminology for what square one actually does.

15
00:01:05,780 --> 00:01:07,182
we're a full digital agency actually.

16
00:01:07,182 --> 00:01:11,355
do web, we do mobile products, we work on, uh we do consulting as well.

17
00:01:11,355 --> 00:01:13,778
We do a lot of work around Stripe payments in particular.

18
00:01:13,778 --> 00:01:14,999
So a little bit of everything.

19
00:01:14,999 --> 00:01:18,772
Yeah, so a little bit more than just the websites we started out with now at this stage.

20
00:01:18,880 --> 00:01:22,593
You know, usually I'm hesitant to bring on uh someone from a development agency.

21
00:01:22,593 --> 00:01:27,826
They usually just want to push their services or try to get their stuff out there looking
for additional customers.

22
00:01:27,826 --> 00:01:37,453
But uh when I was at the Build Stuff Conference in Vilnius, which was honestly a great
conference, just hearing you speak, just like, great guests.

23
00:01:37,453 --> 00:01:38,644
Can't wait to have them on the show.

24
00:01:38,644 --> 00:01:40,276
These are great stories that you had to share.

25
00:01:40,276 --> 00:01:46,680
And I thought most specifically it was uh the practical solutions that you had to

26
00:01:46,680 --> 00:01:48,111
basically malicious users.

27
00:01:48,111 --> 00:02:00,717
I think in today's world, there's just like a lot of this, especially ramping up with, uh
I hate to say AI, but bots galore ah everywhere and having to deal with uh extra ancillary

28
00:02:00,717 --> 00:02:03,988
users who aren't adding any value to your platform.

29
00:02:04,384 --> 00:02:06,765
Yeah, and there's a real cost to these kind of users.

30
00:02:06,765 --> 00:02:09,356
guess the AI issue is a big one at the moment.

31
00:02:09,356 --> 00:02:16,600
In my experience, I've been lucky enough that in Square One, we work with a lot of large
online publishers, news and sports sites.

32
00:02:16,600 --> 00:02:25,615
But in the past, I've also worked at large classified websites as well, which would have
uh large scale attempts to kind of be malicious actors, as you say, for one reason or

33
00:02:25,615 --> 00:02:26,105
another.

34
00:02:26,105 --> 00:02:28,646
And we had a story a few years ago where

35
00:02:28,750 --> 00:02:36,990
Working in a property portal in Ireland and Ireland's a bit of a funny market for real
estate in that it's a national obsession.

36
00:02:36,990 --> 00:02:40,450
Property prices have gone through the roof a few times and big crashes.

37
00:02:40,450 --> 00:02:43,890
But generally speaking, people in Ireland just love going on property websites.

38
00:02:43,890 --> 00:02:45,970
It's not just buying and renting properties.

39
00:02:45,970 --> 00:02:48,530
It's what does the kitchen of a million euro house look like?

40
00:02:48,530 --> 00:02:50,170
Or my neighbor down the road is selling up.

41
00:02:50,170 --> 00:02:52,610
I wonder what way they've decorated their bedroom.

42
00:02:52,890 --> 00:02:58,330
And I was talking to a friend about this at the event in Vilnius.

43
00:02:58,414 --> 00:03:01,576
And they were saying, so you're basically saying that the Irish are a nation of voyeurs.

44
00:03:01,576 --> 00:03:04,998
saying, say that exactly, but I wouldn't not say that.

45
00:03:04,998 --> 00:03:12,222
anyway, property in Ireland, huge business because you just have these very popular
websites from people looking to see all of these, these photos as much as the day-to-day

46
00:03:12,222 --> 00:03:13,342
business of property.

47
00:03:13,342 --> 00:03:17,204
And at the time I was working in a property portal, was sort of two of us, a duopoly.

48
00:03:17,204 --> 00:03:18,705
We were both very popular.

49
00:03:18,705 --> 00:03:20,626
A new entrance came and went all of the time.

50
00:03:20,626 --> 00:03:26,554
And the market at the time was a bit weird because if you're in a state agent, there
wasn't a central portal where you could go and it would syndicate your listings around.

51
00:03:26,554 --> 00:03:27,662
You basically have to.

52
00:03:27,662 --> 00:03:29,862
have a sunk cost effort for every site you engage with.

53
00:03:29,862 --> 00:03:32,962
So everyone knew who came along, had a of a chicken and egg problem.

54
00:03:32,962 --> 00:03:37,222
There's no agent spending time on it because there's no users and there's no users because
there's no content.

55
00:03:37,222 --> 00:03:38,722
So a lot of them would fade away.

56
00:03:39,142 --> 00:03:45,322
Then one day we noticed that we had an interesting new competitor came into the market and
they were from abroad.

57
00:03:45,322 --> 00:03:48,562
They were an existing property portal and they had pretty deep pockets.

58
00:03:48,722 --> 00:03:55,542
And the way they launched themselves in was they had this viral campaign, which was before
viral campaigns were even really a thing.

59
00:03:55,542 --> 00:03:57,132
They had this little video made up

60
00:03:57,132 --> 00:04:02,406
which was like the, you know, the Palm Islands in Dubai, the artificial sort of
residences.

61
00:04:02,406 --> 00:04:05,498
They were going to build a Shamrock Island in Dublin Bay.

62
00:04:05,498 --> 00:04:11,822
And it was going to have all this high rise living fast metros, the world's first giraffe
only zoo, all of this sort of stuff.

63
00:04:11,822 --> 00:04:15,945
And it was presented as like a legitimate proposal coverage on the national news, all of
this.

64
00:04:15,945 --> 00:04:17,266
So they got it.

65
00:04:17,266 --> 00:04:20,097
They turned all of this attention into basically a launch announcement for their website.

66
00:04:20,097 --> 00:04:21,799
So this was a serious new competitor.

67
00:04:21,799 --> 00:04:27,262
And the way that they solved the chicken and egg problem was they came and they scraped
all of our listings and injected them onto their website.

68
00:04:27,526 --> 00:04:30,247
The idea here was that you're searching for property.

69
00:04:30,247 --> 00:04:34,408
If you end up on their website, suddenly they have all of our listings in the search
results.

70
00:04:34,408 --> 00:04:37,479
And when you click through them, then it will come back to our listing.

71
00:04:37,479 --> 00:04:43,350
And then we ultimately deliver the lead, the inquiry or the phone call or whatever to the
advertiser, which is what we're being paid for.

72
00:04:43,350 --> 00:04:50,762
But their game plan is, if they're popular enough over time, they can go to the agents and
say, look, we have all of this traffic, cut these guys out of the loop, come work with us.

73
00:04:50,762 --> 00:04:52,453
So it's a big, threat for us.

74
00:04:52,933 --> 00:04:57,234
And when someone comes along and they're sort of maliciously against your system in this
way, you've

75
00:04:57,234 --> 00:05:05,717
of approaches and one of them is to set the lawyers on them, send off legal letters and
please leave us alone sort of stuff.

76
00:05:05,717 --> 00:05:10,858
Problems we had at the time was that the case law wasn't really settled around
screenscaping, it was very early on.

77
00:05:10,858 --> 00:05:20,161
And also we were a very young company, we become the biggest one in the market, but a lot
of us were there in our first job sort of learning as we go and built up good cohort of

78
00:05:20,161 --> 00:05:25,332
users, spending a bunch of money and waiting for two years for a legal case to settle.

79
00:05:25,410 --> 00:05:29,531
not a great option for us, especially because within two years, it might be moot.

80
00:05:29,531 --> 00:05:30,951
You could be out of business.

81
00:05:31,092 --> 00:05:32,372
So we took a different approach.

82
00:05:32,372 --> 00:05:35,311
the standard approach, someone is scraping your content.

83
00:05:35,311 --> 00:05:35,943
So what do you do?

84
00:05:35,943 --> 00:05:39,394
You find their IP addresses, you block them at firewall level.

85
00:05:39,494 --> 00:05:40,794
That's the end of it.

86
00:05:41,014 --> 00:05:42,525
But then they really wanted the content.

87
00:05:42,525 --> 00:05:44,465
So every day they would move around a little bit.

88
00:05:44,465 --> 00:05:45,386
We'd have to block them again.

89
00:05:45,386 --> 00:05:46,876
And you're into this constant back and forth.

90
00:05:46,876 --> 00:05:52,978
And the problem was that our blocking, we would catch them, but they would get a large
chunk of content every time before we caught them.

91
00:05:52,978 --> 00:05:53,870
So they're...

92
00:05:53,870 --> 00:05:57,652
They're getting a lot of content and it's not really doing any good for us this blocking.

93
00:05:57,652 --> 00:06:07,085
So we wanted to maybe if they're going to get our listings, can we do something that at
least makes the listings on their website look bad and obviously not the right one so

94
00:06:07,085 --> 00:06:08,596
people will come back to us.

95
00:06:08,956 --> 00:06:15,859
And you could do things like, you know, well, let's change all of the addresses to a
script tag alert, haha, not today guys or something like that.

96
00:06:15,859 --> 00:06:18,410
And hope it ends up like that, but that becomes very obvious.

97
00:06:18,410 --> 00:06:22,862
So you'll spot something like that very quickly and you can put in automated checks to
catch that sort of stuff.

98
00:06:22,862 --> 00:06:29,062
So we need to send back stuff that was semantically sound, but maybe just a little bit
fuzzier, a little bit off.

99
00:06:29,062 --> 00:06:33,662
So what we did was if we detected their IP addresses, we would return our listings to
them.

100
00:06:33,662 --> 00:06:40,842
But instead of say a three bedroom house for 400,000 euro, we'll say, well, this is a four
bedroom house for 280,000 euro.

101
00:06:40,842 --> 00:06:42,002
Oh no.

102
00:06:42,002 --> 00:06:44,042
Yeah, fuzz the details just a little bit.

103
00:06:44,042 --> 00:06:44,682
And it worked.

104
00:06:44,682 --> 00:06:46,062
It led through to their site.

105
00:06:46,062 --> 00:06:50,550
the thinking that we had here was that people will see this

106
00:06:50,550 --> 00:06:51,891
information, they'll say, this sounds great.

107
00:06:51,891 --> 00:06:53,141
They'll click through to our website.

108
00:06:53,141 --> 00:06:54,533
They'll see it's different on our website.

109
00:06:54,533 --> 00:06:57,935
And they'll say, well, we, know and trust this existing websites.

110
00:06:57,935 --> 00:07:01,117
Clearly these new guys are charlatans and we never use them again.

111
00:07:01,117 --> 00:07:03,759
And it worked a little bit, but in practice, not so much.

112
00:07:03,759 --> 00:07:07,712
A lot of people just got very annoyed as someone is wrong here in the chain.

113
00:07:07,712 --> 00:07:10,063
Don't know or care who it is, but you're all wasting our time.

114
00:07:10,063 --> 00:07:16,448
They would complain to agencies, state agents would complain to us and say, don't know
what you guys are doing, but this is annoying.

115
00:07:16,448 --> 00:07:17,178
People are ringing me.

116
00:07:17,178 --> 00:07:17,849
don't have time for this.

117
00:07:17,849 --> 00:07:19,049
So leave it alone.

118
00:07:19,110 --> 00:07:20,010
So we.

119
00:07:20,398 --> 00:07:29,378
wasn't really working great for us overall there, but we still had the idea that, okay,
well, if we can do something here to make their listings look a little bit less credible,

120
00:07:29,378 --> 00:07:31,258
then we'll be in a good spot.

121
00:07:31,358 --> 00:07:39,838
And we, but it needed to be something that they couldn't detect easily automatically
because at scale, you know, they're scraping 35, 40,000, 50,000 properties every day.

122
00:07:39,838 --> 00:07:45,158
So, you know, they don't want to put manual effort on this, but if it's something that
they could detect technically, okay, it's a problem.

123
00:07:45,158 --> 00:07:50,266
So we can't do stuff like mess with the province or the region because they're fairly set,
you you can

124
00:07:50,266 --> 00:07:53,228
check them from a list somewhere here at the valid provinces and areas.

125
00:07:53,228 --> 00:07:57,051
But the main part of the address is it's often free text.

126
00:07:57,051 --> 00:07:58,271
It's not hugely validated.

127
00:07:58,271 --> 00:08:00,863
know, people will use vanity addresses there.

128
00:08:00,863 --> 00:08:05,276
You know, they'll put a slightly nicer street name in or something like that or typos or
whatever it is.

129
00:08:05,276 --> 00:08:08,418
So we had that and we had the photograph that we could do something with.

130
00:08:08,638 --> 00:08:14,926
So then we set it up so that if we detected their scraper IPs coming through, what we
would do initially is

131
00:08:14,926 --> 00:08:17,186
we would have a set of fake properties.

132
00:08:17,186 --> 00:08:20,466
So we would return the core information, the price and all that would be fine.

133
00:08:20,466 --> 00:08:31,165
But instead of 15 Main Street, Dublin, you now get 10 Downing Street, Dublin, Ireland, and
a photograph of the British Prime Minister waving outside the office there.

134
00:08:31,165 --> 00:08:35,446
Or you get the White House, 1600 Pennsylvania Avenue, County Cork.

135
00:08:35,446 --> 00:08:38,626
And things like this that to a human eye, they're really obvious.

136
00:08:38,626 --> 00:08:41,986
Willy Wonka's Chocolate Factory or the Emerald Palace in Oz.

137
00:08:41,986 --> 00:08:43,150
Because we call this the...

138
00:08:43,150 --> 00:08:44,581
Project Yellow Brick Road internally.

139
00:08:44,581 --> 00:08:47,762
there was a lot of Wizard of Oz references thrown into it.

140
00:08:47,762 --> 00:08:51,356
But the data anyway went across to their website.

141
00:08:52,958 --> 00:08:54,980
I mentioned Ireland property is an obsession.

142
00:08:54,980 --> 00:09:00,455
People were constantly on social media sharing funny, interesting houses, that sort of
stuff.

143
00:09:00,455 --> 00:09:02,526
This is a very popular thing you would see.

144
00:09:05,449 --> 00:09:07,300
Well, this is what happened.

145
00:09:07,300 --> 00:09:11,714
They looked at their website the next day and saw that there was some legitimate

146
00:09:11,714 --> 00:09:15,895
properties in there, but the vast, vast majority of it was just obvious junk.

147
00:09:15,895 --> 00:09:18,556
was just joke properties all over the place.

148
00:09:18,556 --> 00:09:25,718
And, most property portals from time to time might have one or two of these of someone,
you know, joking, putting up the, oh, the prime minister's office is now vacant before an

149
00:09:25,718 --> 00:09:27,699
election, something like that.

150
00:09:27,699 --> 00:09:32,280
But the scale here was just the vast, vast majority was, was absolute nonsense.

151
00:09:32,280 --> 00:09:33,721
So people were laughing about it.

152
00:09:33,721 --> 00:09:38,862
went viral, but there was also questions being raised about how secure is their platform.

153
00:09:38,862 --> 00:09:40,842
You know, has it been hacked?

154
00:09:40,842 --> 00:09:41,542
Is this malicious?

155
00:09:41,542 --> 00:09:45,182
Because the scale of it is so big, someone must have done something to them.

156
00:09:45,202 --> 00:09:51,142
So they deleted all of the properties, they bounced their YPs, they tried again, and we
caught them again in the net.

157
00:09:51,142 --> 00:09:55,682
So for a few days, this sort of pattern repeated itself where we just had this constant
stream of junk.

158
00:09:55,722 --> 00:09:59,982
And it got to a point where it just wasn't practical for them to keep going.

159
00:09:59,982 --> 00:10:05,562
They gave up on us and they moved and started targeting our other existing competitor in
the market.

160
00:10:05,562 --> 00:10:08,578
Now, our other existing competitor, they had fewer listings than us.

161
00:10:08,578 --> 00:10:14,220
And they also had access to more expensive lawyers and were more inclined to have a
conversation in the courts.

162
00:10:14,220 --> 00:10:21,203
So long story short, this new competitor ended up winding down and going out of the market
relatively quickly afterwards.

163
00:10:21,203 --> 00:10:26,305
But I think what we've managed to do there was we kind of flipped the cost curve around a
little bit.

164
00:10:26,305 --> 00:10:30,467
know, they were going to come at us, they were going to come towards us and outright
blocking wasn't working well.

165
00:10:30,467 --> 00:10:38,094
But what we were able to do was effectively waste their time, waste their resources and
make it not cost effective for them to continue targeting us because at

166
00:10:38,094 --> 00:10:48,405
35, 40, 50,000 properties, if we're spiking it subtly like this, they're going to need a
human pair of eyes to kind of go over this and check it.

167
00:10:48,466 --> 00:10:50,468
the return on investment was just not going to be there for them.

168
00:10:50,468 --> 00:10:54,933
So it was more attractive for them to target one of the other competitors there.

169
00:10:55,444 --> 00:11:04,905
I really yeah, I know I really like that perspective I think it's something that a lot of
people miss out on is that Attackers or I hesitate to call them necessarily malicious ah

170
00:11:04,905 --> 00:11:13,514
because they they're not trying to necessarily take down your website or your But yeah,
but in a way they are trying to take down your business right there.

171
00:11:13,514 --> 00:11:17,798
They are trying to so uh malicious I you know

172
00:11:18,026 --> 00:11:21,437
on the line as far as whether or not that's appropriate word.

173
00:11:21,437 --> 00:11:22,769
I don't know if I have a better one.

174
00:11:22,769 --> 00:11:28,572
But the interesting thing is that think a lot of people miss out on is that malicious
attackers don't have an infinite revenue stream.

175
00:11:28,712 --> 00:11:36,137
They can't afford to overcome every single security block or countermeasure that's in
place.

176
00:11:36,137 --> 00:11:43,901
And you came up with a pretty cheap way to uh circumvent these particular users from
coming back at all.

177
00:11:44,281 --> 00:11:45,686
And I think that's like

178
00:11:45,686 --> 00:11:55,632
a core to practical security engineering that we don't really see that much today because
a lot of companies, a lot of security organizations, they call these things like best

179
00:11:55,632 --> 00:12:05,537
practices and then just go full steam with implementing them even if they do potentially
cost a lot, whereas there's practical low-cost alternatives that actually do solve the

180
00:12:05,537 --> 00:12:07,028
core of the problem.

181
00:12:07,212 --> 00:12:07,692
Absolutely.

182
00:12:07,692 --> 00:12:09,883
There can be, and you're right there.

183
00:12:09,883 --> 00:12:12,744
can, you have the industry best practices.

184
00:12:12,744 --> 00:12:15,564
It's always tempting to reach for, here's the established playbook.

185
00:12:15,564 --> 00:12:16,705
Let's just go with it.

186
00:12:16,705 --> 00:12:19,105
Maybe it's not a hundred percent perfect for our company.

187
00:12:19,105 --> 00:12:22,236
It was a work for Google or whoever, but it's a best practice.

188
00:12:22,236 --> 00:12:22,896
So let's do it.

189
00:12:22,896 --> 00:12:29,418
yeah, there's definitely a way to kind of look at the context in your own business and
what's going to work well in your own business.

190
00:12:29,418 --> 00:12:35,854
And the cost thing, I think is a useful point as well, because to loop back to what we
talked about at the start with the AI bots, for example.

191
00:12:35,854 --> 00:12:40,006
So Cloudflare have a service now for this sort of thing.

192
00:12:40,006 --> 00:12:45,978
So AI bots are coming to your site, your bandwidth fields are going bananas because
they're sucking up all of this content.

193
00:12:45,978 --> 00:12:50,360
again, it's not necessarily malicious users, but users we don't particularly want.

194
00:12:50,360 --> 00:12:53,201
And Cloudflare came up with what they call the labyrinth.

195
00:12:53,421 --> 00:12:54,712
So they land on your website.

196
00:12:54,712 --> 00:13:02,969
They're expecting to see a detailed website where you've poured your heart and soul over
many years writing on the ins and outs of the Dutch train system, for example.

197
00:13:02,969 --> 00:13:05,076
They're to get all this amazing training data.

198
00:13:05,076 --> 00:13:14,531
and Cloudflare starts sending them back, not your application, but AI generated slop,
plausible slop, but slop nonetheless, full of links internally to other pages of AI

199
00:13:14,531 --> 00:13:15,202
generated slop.

200
00:13:15,202 --> 00:13:24,527
So the idea here is again, just to burn through the resources that whoever's trying to
come at your website, the AI bots in this case have, and I guess Cloudflare are making the

201
00:13:24,527 --> 00:13:29,719
bet there that Cloudflare have more resources to burn than these guys do when they're
throwing them together.

202
00:13:30,100 --> 00:13:34,594
But as a customer sitting behind the big shield, let the two big guys...

203
00:13:34,594 --> 00:13:39,676
find it out and we'll just carry on writing our passion project about the Dutch train
system or whatever it is.

204
00:13:39,676 --> 00:13:44,497
ah But it's a really interesting thing that they rolled out there.

205
00:13:44,597 --> 00:13:46,648
it's uh cool system.

206
00:13:46,648 --> 00:13:55,681
It's definitely worth a look if you're seeing a big spike in bandwidth builds without any
corresponding real world benefit or real user benefit coming out of it.

207
00:13:55,681 --> 00:13:59,422
think discrepancy is happening more and more.

208
00:13:59,544 --> 00:14:09,794
I think it's an extension of what was happening initially with the idea of zip bombs where
there's like a gzipped archive that sort of was recursively pointing to made up directory

209
00:14:09,794 --> 00:14:13,307
file system that was much bigger than the disk would actually show.

210
00:14:13,307 --> 00:14:19,313
If a malicious attacker tried to download these from your site, they would search it for
credentials, for instance.

211
00:14:19,313 --> 00:14:23,998
Let's say you call it secrets.zip and they go and they try to unzip that archive.

212
00:14:23,998 --> 00:14:24,770
It will just...

213
00:14:24,770 --> 00:14:31,764
you know, consume all of their resources and and I feel like these sorts of honey parts
are great, especially when they have a real user interaction.

214
00:14:31,764 --> 00:14:44,801
I want to want to ask you something which is like, did you get lucky with having the other
major player in Ireland fold basically because without that that happening wouldn't the uh

215
00:14:44,801 --> 00:14:52,386
was a Dutch company still have access to essentially the same listings because I imagine
the real estate agents they were uploading the same data to both.

216
00:14:52,386 --> 00:14:53,446
uh

217
00:14:53,580 --> 00:14:57,424
your client's website as well as your competitor.

218
00:14:57,424 --> 00:15:01,044
And so at that point, wouldn't that data still be available somewhere?

219
00:15:01,528 --> 00:15:11,467
So the other competitor that we had, the pre-existing competitor that we had in Ireland,
yeah, because they didn't have necessarily 100 % the same listings.

220
00:15:11,467 --> 00:15:19,734
It would have been very, very strong in rentals, for example, or certain areas of the
country where our sales team had just done a better job than theirs had.

221
00:15:19,734 --> 00:15:21,195
So they would have big blind spots.

222
00:15:21,195 --> 00:15:27,781
They would be very strong in some parts of the country, but huge parts of it were more uh
focused on our side of things.

223
00:15:27,781 --> 00:15:29,862
if you only had their listings,

224
00:15:29,966 --> 00:15:37,670
oh You have a very incomplete picture of the market, which was one of the reasons that we
had become the number one website and where the big target at the time was we had done

225
00:15:37,670 --> 00:15:42,942
this because it was the way the uploading work for the agents at the time was it was quite
a good most.

226
00:15:42,942 --> 00:15:48,155
Once you manage to hook someone in to do it and put the listings onto your website, it's a
lot of effort to go and do it somewhere else.

227
00:15:48,155 --> 00:15:55,078
And it's not unusual if you have the one website that delivers most of your traffic,
you'll make sure every listing goes up there.

228
00:15:55,374 --> 00:16:02,214
you get distracted by a phone call or you're going out to lunch and you forget to do it on
the other website, you know, wasn't unheard of, even in the areas where we both had kind

229
00:16:02,214 --> 00:16:03,954
of similar numbers of estate agents.

230
00:16:03,954 --> 00:16:08,198
So that was why we ended up being the main target in the first place.

231
00:16:08,919 --> 00:16:20,168
Did you ever consider sharing with your local competitor the way to prevent the external
competitor from scraping their website because there's sort of this like, that data is

232
00:16:20,168 --> 00:16:28,224
still available and preventing the attacker or the malicious unwanted traffic to your
competitors was still somehow beneficial for you?

233
00:16:29,479 --> 00:16:30,979
no short answer.

234
00:16:30,979 --> 00:16:38,612
I think, know, I'd be honest, I think, you know, over the years and over, uh, over a few
beers at various times, it's, it's certainly been shared informally with them once or

235
00:16:38,612 --> 00:16:38,912
twice.

236
00:16:38,912 --> 00:16:47,646
But, no, think, from, from our perspective at the time, you know, we, we, just wanted
these guys to leave us alone.

237
00:16:47,646 --> 00:16:50,737
And we knew that if they left us alone, that was our main priority.

238
00:16:50,737 --> 00:16:54,409
They left us happy enough that we could fight them commercially and do whatever we needed
to do.

239
00:16:54,409 --> 00:16:57,954
Um, the fact that they were then attacking the other guys.

240
00:16:57,954 --> 00:17:01,615
That was, to be frank, it wasn't the worst outcome in the world.

241
00:17:01,615 --> 00:17:09,417
There's now these other guys, instead of rolling out new products and doing whatever they
need to do, have to spend time and effort dealing with whatever is going on here.

242
00:17:09,417 --> 00:17:13,038
So, it's bit like the old story about the guys who were on safari.

243
00:17:13,198 --> 00:17:22,131
And you're driving in a Jeep and the Jeep breaks down and everyone has to make a run for
it back to the base when the lines are coming over and one guy stops to tie his laces and

244
00:17:22,131 --> 00:17:23,547
the guy beside him says, what are you doing?

245
00:17:23,547 --> 00:17:23,971
You're crazy.

246
00:17:23,971 --> 00:17:26,422
You're never going to outrun the line if you're stopping to...

247
00:17:26,422 --> 00:17:29,134
fiddle with your laces and the guy says, well, I don't need to outrun the lion.

248
00:17:29,134 --> 00:17:30,195
I just need to outrun you.

249
00:17:30,195 --> 00:17:35,690
You don't necessarily need total victory if you can push your adversary off to someone
else's.

250
00:17:35,690 --> 00:17:41,955
Not the most, em let's say, charitable sounding way of approaching it.

251
00:17:41,955 --> 00:17:49,281
But if you're being attacked in some way, being able to just sort of deflect the attacker
is as good as a victory in many cases, I think.

252
00:17:49,281 --> 00:17:51,282
need total domination.

253
00:17:51,384 --> 00:17:59,968
But I think that's one of the aspects that's completely missing from like the whole area
of detection engineering or building up security boundaries and understanding your threat

254
00:17:59,968 --> 00:18:08,681
model is that you're not taking into account or many companies aren't taking into account
like the actual practical nature of who their attackers are, what those look like.

255
00:18:08,681 --> 00:18:10,636
ah

256
00:18:10,636 --> 00:18:14,208
When should things be blocked and when should they be let through?

257
00:18:14,328 --> 00:18:17,150
Where is the benefit there or the disadvantage?

258
00:18:17,150 --> 00:18:20,942
And the cost comes into the equation for both you and the attacker.

259
00:18:20,942 --> 00:18:25,455
There's a lot of solutions out there that are just incredibly expensive.

260
00:18:25,455 --> 00:18:27,256
And I see people putting these up all day long.

261
00:18:27,256 --> 00:18:35,760
And my question is, why do you care if you get an extra, I don't know, 2,000 RPS for an
hour or so one time?

262
00:18:35,760 --> 00:18:38,784
How much is that just going to cost you in sheer

263
00:18:38,784 --> 00:18:48,087
Monetary value and if it's like a DDS attack, you're like we can handle it because we
scale then just ignore it You know because you're that's free engineering that you don't

264
00:18:48,087 --> 00:18:55,239
have to spend you don't have to pay extra costs on external solution You don't have to
maintain it uh If there is an impact to your business like it was in this case, right?

265
00:18:55,239 --> 00:18:58,169
Like there was this wasn't like a technological threat, right?

266
00:18:58,169 --> 00:19:01,260
This was like a existential threat for the business, right?

267
00:19:01,454 --> 00:19:08,022
You you can evaluate a little bit differently and in a way I seek a good justification for
spending more money

268
00:19:08,024 --> 00:19:09,315
to actually solve the problem.

269
00:19:09,315 --> 00:19:15,301
when you look at the threat model and you look at the countermeasures, you're like, none
of these will actually fix the problem.

270
00:19:15,301 --> 00:19:18,814
Like, we'll just block some IP addresses is usually the standard one.

271
00:19:18,814 --> 00:19:30,535
It's like, well, IPv4 or IPv6 and then like all of them, what if they go on AWS and are
proxying the stuff through Lambda or CloudFront or CloudFlare?

272
00:19:30,535 --> 00:19:31,324
What are you going to do then?

273
00:19:31,324 --> 00:19:34,648
We're going to block all the IP addresses for non-residential like...

274
00:19:34,648 --> 00:19:35,439
That's ridiculous.

275
00:19:35,439 --> 00:19:37,924
Like you're not going to, you're not going to do that and keep those lists up to date.

276
00:19:37,924 --> 00:19:40,288
It's just a lot of extra work to be done.

277
00:19:40,288 --> 00:19:42,973
So I really like, I really liked the solution.

278
00:19:42,973 --> 00:19:44,142
Very practical.

279
00:19:44,142 --> 00:19:49,084
There's a lot of extra work and then there's also a lot of potential negative side
effects.

280
00:19:49,084 --> 00:19:59,887
We had an example recently where a number of our publishers were being hammered by bots
where someone had fired them up inside one of the Google Cloud IP ranges.

281
00:20:00,928 --> 00:20:07,730
They were passing themselves off as various AI research bots or whatever they were, but
they obscene amounts of traffic just out of absolutely nowhere.

282
00:20:08,396 --> 00:20:12,258
The issue there is you come and say, well, let's find where the traffic's coming from.

283
00:20:12,258 --> 00:20:12,838
It's very clear.

284
00:20:12,838 --> 00:20:13,758
Here's the block of IPs.

285
00:20:13,758 --> 00:20:18,450
Let's just smash that whole range and go back to the day job or whatever it is.

286
00:20:18,450 --> 00:20:26,173
But the problem was that whatever block they've managed to get in GCP had a partial
overlap with the Google bots own IP range.

287
00:20:26,214 --> 00:20:31,596
So a very naive block to say, let's just kill all of these things off, stop this traffic.

288
00:20:31,636 --> 00:20:32,106
Fine.

289
00:20:32,106 --> 00:20:36,600
Four or five days later, you open Google search console and you see a whole flurry of
errors saying Google blocks.

290
00:20:36,600 --> 00:20:44,200
Google box was blocked from crawling all of these pages and now you're dropping out of the
index and that's a whole uh other set of problems you need to deal with and deal with very

291
00:20:44,200 --> 00:20:44,981
quickly.

292
00:20:44,981 --> 00:20:54,784
yeah, you're dead right in that the smash the big red button and just block them off at
source isn't always the most effective solution to kill you.

293
00:20:54,946 --> 00:20:58,608
You're letting fear overcome the actual size of the impact, right?

294
00:20:58,608 --> 00:21:10,313
Like, no, they're causing, they're costing us like pennies per minute ah is a lot
different than like, no, like our business is no longer listed anywhere.

295
00:21:10,666 --> 00:21:17,987
Which you obviously aren't thinking about when you're like, we could, we could probably
just block all the range.

296
00:21:17,987 --> 00:21:21,132
I get the sense that ah you aren't just.

297
00:21:21,132 --> 00:21:30,005
You haven't just worked in the security domain though with threats that you have a lot of
experience with the different cloud providers and ah what they're doing.

298
00:21:30,082 --> 00:21:37,381
I think maybe a weird stepping stone here is that practical solutions to complex
technology problems.

299
00:21:37,381 --> 00:21:42,070
And I just get this feeling that you've got like a bunch of other stories in the same
domain.

300
00:21:42,474 --> 00:21:43,845
Yeah, that's true.

301
00:21:43,845 --> 00:21:50,959
Plugging together sort of little bits of technology from different areas to solve problems
is something we've done many, many times.

302
00:21:51,259 --> 00:21:56,402
One example is with the elections in Ireland not so long ago.

303
00:21:56,402 --> 00:22:02,946
So elections are interesting because they tend to come around very semi-scheduled.

304
00:22:02,946 --> 00:22:09,838
Sometimes they're sort of off schedule for whatever political reasons, but generally you
have a relatively short window when you know they're definitely going to happen.

305
00:22:09,838 --> 00:22:13,910
You might hear, it'll happen probably in November, but you won't get confirmation until a
certain date.

306
00:22:13,910 --> 00:22:15,542
And then you have a very short window.

307
00:22:15,603 --> 00:22:23,249
And the nice thing with elections is they really get the creative juices flowing of
journalists and editors and people who kind of live in this space all of the time.

308
00:22:23,750 --> 00:22:31,836
So we were working with a large publisher who had a CMS, uh industry-leading CMS for their
news publisher.

309
00:22:31,877 --> 00:22:36,599
And it does the job perfectly, but it's very optimized for, need to serve an article.

310
00:22:36,599 --> 00:22:38,542
I need to serve a river of articles.

311
00:22:38,678 --> 00:22:41,170
Anything else you want to do, that's someone else's problem.

312
00:22:41,170 --> 00:22:43,162
I would do this and do this job really, really well.

313
00:22:43,162 --> 00:22:50,217
The problem is that when you have something like an election coming along, you've
journalists and editors who now want to do, here's a cool data visualization we can do, or

314
00:22:50,217 --> 00:22:57,773
here's this really cool interactive thing, oh putting your ideas and see which party
matches your policy, all these really cool, interesting things.

315
00:22:57,773 --> 00:23:00,085
And you're sitting there going, okay, guys, well, the election was announced.

316
00:23:00,085 --> 00:23:02,577
We've three and a half weeks to get everything built and live.

317
00:23:02,577 --> 00:23:04,258
And you have a system here which

318
00:23:04,590 --> 00:23:13,070
The flexibility it gives us is we can basically embed stuff into an article like a tweet
or a YouTube video or something, but you're not going to be able to go hog wild and build

319
00:23:13,070 --> 00:23:15,190
like centers or anything like that.

320
00:23:15,250 --> 00:23:19,510
So one of the tools we were looking at was a coalition builder.

321
00:23:19,930 --> 00:23:27,050
So in Ireland, the way our electoral system works is we use the proportional
representation and the transferable vote.

322
00:23:27,050 --> 00:23:32,850
not sure if it's the same with you guys, but it turns elections here into almost like a
national sport.

323
00:23:32,850 --> 00:23:33,890
you have

324
00:23:33,906 --> 00:23:35,727
seats available, you've 10 candidates.

325
00:23:35,727 --> 00:23:38,148
Instead of just picking your favorite one, you rank them all.

326
00:23:38,148 --> 00:23:40,340
And then at the end of each round, candidate drops out.

327
00:23:40,340 --> 00:23:45,983
If my candidate drops out, my number one vote is dead, but my number two vote now becomes
a number one vote for someone else.

328
00:23:45,983 --> 00:23:47,133
this goes on and on.

329
00:23:47,133 --> 00:23:47,864
It sounds complex.

330
00:23:47,864 --> 00:23:52,896
It is kind of complex, but it means that after the election, there's three or four days of
manic media coverage.

331
00:23:52,896 --> 00:24:00,160
The most popular website in the country is a Google Sheets that's shared by one of the
journalists here who keeps track of all these counts and all the different centers.

332
00:24:00,160 --> 00:24:03,462
It's an odd time where there's a huge amount of interest and traffic.

333
00:24:03,630 --> 00:24:14,470
With this electoral system, what it means is that you can't get elected by being really,
really popular with 30, 35 % of people and despised by everyone else.

334
00:24:14,470 --> 00:24:19,230
You have to be, if not likable, at least less dislikable than someone else.

335
00:24:19,410 --> 00:24:22,830
But what it means is that practically we end up with coalition governments all of the
time.

336
00:24:22,910 --> 00:24:30,870
So once the votes are counted, the seats are allocated, you need to figure out, well, 27
guys from this party and 15 people from this party, how are we going to combine them and

337
00:24:30,870 --> 00:24:33,132
blah, So the idea was a game where you could

338
00:24:33,132 --> 00:24:39,126
see what the votes were and you can allocate it and see what type of government could be
made and cross the magic number, all this sort of stuff.

339
00:24:39,142 --> 00:24:47,684
So we come up with a way to do this where you're going to put a web component together,
which will do the front end rendering, nice little JavaScript animations, that sort of

340
00:24:47,684 --> 00:24:48,174
stuff.

341
00:24:48,174 --> 00:24:51,607
And it'll work as an embed, the same as a YouTuber, Twitter, or whatever.

342
00:24:51,607 --> 00:24:56,160
So it'll work within the constraints of our CMS system, but it needs to pull the data from
somewhere.

343
00:24:56,761 --> 00:25:01,845
And we were looking for a system where you're going to pull in structured data.

344
00:25:01,885 --> 00:25:02,990
It needs to be oh

345
00:25:02,990 --> 00:25:08,310
very easy to use, needs to have tight permissions on it so that only the editors and
journalists are using it.

346
00:25:08,310 --> 00:25:13,570
And ideally it'll have some kind of version history if someone makes a mess of this or a
fat finger's a number, we can roll it back easily enough.

347
00:25:14,010 --> 00:25:23,070
And I some people will be listening to this and thinking, yeah, this sounds like a great
case for a limited scope, basic off microservice that we can spin up and blah, blah.

348
00:25:23,430 --> 00:25:27,250
And could do, but again, we've three weeks to get the thing built in live.

349
00:25:28,370 --> 00:25:31,896
we've a team who's very used to working with

350
00:25:31,896 --> 00:25:36,747
this one big CMS, there's not a huge amount of additional infrastructure available to kind
of provision this sort of stuff.

351
00:25:36,747 --> 00:25:39,228
We needed to work within the constraints of what we had.

352
00:25:39,428 --> 00:25:43,129
And we turned out Google Sheets was a solution for this.

353
00:25:43,910 --> 00:25:44,910
Yes.

354
00:25:45,690 --> 00:25:49,461
the permissions sorted by your Google manages the permissions.

355
00:25:49,461 --> 00:25:50,402
That's, that's all fine.

356
00:25:50,402 --> 00:25:52,412
It has version tracking built into it.

357
00:25:52,412 --> 00:25:53,493
Usability.

358
00:25:53,493 --> 00:25:55,353
Everyone knows how to use Google Sheets.

359
00:25:55,353 --> 00:25:58,808
And what I didn't know before we started looking into this was that Google Sheets

360
00:25:58,808 --> 00:26:00,919
has this sort of scripting language internally.

361
00:26:00,919 --> 00:26:08,282
But what you can do is you can publish the sheet in such a way that that script can
transform your data into like an API response.

362
00:26:08,302 --> 00:26:11,503
So as far as the outside world is concerned, you'll hit a certain URL.

363
00:26:11,503 --> 00:26:17,636
This script runs and it maps your data from your Google Sheet into a lovely JSON response
that you can take out.

364
00:26:17,636 --> 00:26:20,707
So here now we're into a mode where, OK, we have a Google Sheet.

365
00:26:20,707 --> 00:26:25,689
We can give this to our editors, put in whatever they need, use this little script thing
that sits in the middle.

366
00:26:25,689 --> 00:26:26,904
It's like 20 lines of code.

367
00:26:26,904 --> 00:26:28,046
It sits there brilliantly.

368
00:26:28,046 --> 00:26:31,206
and our front end can call this thing, happy days, this is great.

369
00:26:31,206 --> 00:26:33,066
Google, we're worried about scale.

370
00:26:33,066 --> 00:26:34,926
know, the election, this is gonna be huge.

371
00:26:34,926 --> 00:26:37,586
People are gonna be constantly refreshing this and sharing it.

372
00:26:37,586 --> 00:26:39,366
It'll be a big viral tool.

373
00:26:39,466 --> 00:26:42,446
Google, yeah, scale's not gonna be a problem for Google, is it?

374
00:26:42,446 --> 00:26:45,066
So we tested it, looked at it, it looked okay.

375
00:26:45,066 --> 00:26:48,086
And then we started to hit a few kind of problems fairly quickly.

376
00:26:48,426 --> 00:26:50,886
So cores, first of all, was a problem.

377
00:26:51,026 --> 00:26:52,026
The cores errors.

378
00:26:52,046 --> 00:26:55,246
But then we also had an issue with the speed.

379
00:26:55,564 --> 00:27:01,137
because the script would run, but the first time it ran, it would take maybe five or six
seconds to respond.

380
00:27:01,137 --> 00:27:04,699
And on follow-up loads, it would be faster, but not by much.

381
00:27:04,699 --> 00:27:06,440
So it's not great.

382
00:27:06,440 --> 00:27:16,976
um And the bigger problem then was it's also rate limited because these URLs, they're not
really intended for public API in the night of an election, I guess, ah but they're

383
00:27:16,976 --> 00:27:17,466
limited.

384
00:27:17,466 --> 00:27:20,808
think it was something like 50 requests a minute or something like that.

385
00:27:20,808 --> 00:27:25,496
Under normal circumstances, should be fine, but the scale we were looking at just

386
00:27:25,496 --> 00:27:27,007
That was going to be problematic.

387
00:27:27,247 --> 00:27:30,689
We had a look and we came around to Cloudflare Workers.

388
00:27:30,790 --> 00:27:35,253
So Cloudflare Workers, we could put a very small script together that would run at the
edge somewhere.

389
00:27:35,253 --> 00:27:38,635
And what it would do is, so first of all, it gets us around the cores error.

390
00:27:38,675 --> 00:27:40,066
It's making a server-server communication.

391
00:27:40,066 --> 00:27:40,447
That's fine.

392
00:27:40,447 --> 00:27:42,858
And we can put whatever cores headers we want on the response.

393
00:27:42,858 --> 00:27:45,800
So great, we can now consume it from the front end.

394
00:27:46,201 --> 00:27:50,984
We could then also push Cloudflare's caching headers on it.

395
00:27:50,984 --> 00:27:53,806
So what we could do is we could call the sheet.

396
00:27:54,230 --> 00:27:59,682
And then when we're sending the response from the worker, we can put a cache to say, okay,
for the next minute, this is publicly cacheable.

397
00:27:59,682 --> 00:28:00,832
It's not going to change.

398
00:28:00,832 --> 00:28:05,193
And then we use Cloudflare's own caching rules to say, cache this in the CDN.

399
00:28:05,193 --> 00:28:06,384
Don't even go back to the worker.

400
00:28:06,384 --> 00:28:07,554
That's all fine.

401
00:28:07,554 --> 00:28:11,345
And what that solved for us was two things fairly quickly.

402
00:28:11,345 --> 00:28:18,468
It solved the rate limiting issue, because now we're going to hit it at most once a
minute, two or three times, maybe depending on the, dog button, but not going to hit

403
00:28:18,468 --> 00:28:19,992
enough to challenge the rate limit.

404
00:28:19,992 --> 00:28:22,314
But it also means that the speed should be quite quick.

405
00:28:22,314 --> 00:28:26,127
Cloudflare CDN is going to be order of magnitude faster than Google.

406
00:28:26,247 --> 00:28:29,810
Now, it doesn't solve the problem of the cold cache every minute or so.

407
00:28:29,810 --> 00:28:35,945
So we had a cron set up on a machine somewhere that every 30 seconds just curled this URL.

408
00:28:35,945 --> 00:28:42,620
So it took the hit on whenever the cache was cold, but it meant the front end would always
be sending back a warm cache.

409
00:28:42,641 --> 00:28:43,402
So we looked at this.

410
00:28:43,402 --> 00:28:44,192
So this is now working.

411
00:28:44,192 --> 00:28:44,593
It's fine.

412
00:28:44,593 --> 00:28:46,044
It's really responsive.

413
00:28:46,044 --> 00:28:47,195
We had a chat with the editors.

414
00:28:47,195 --> 00:28:49,096
We were talking about cache and validation.

415
00:28:49,218 --> 00:28:52,089
So when they update us, they want to change to be live immediately.

416
00:28:52,089 --> 00:28:56,241
we were looking at ways, oh CloudFair has great APIs for purging the cache.

417
00:28:56,241 --> 00:29:00,323
We were looking at integrating this with some kind of, I don't know, a button into Google
Sheets that would do it.

418
00:29:00,323 --> 00:29:10,587
And we started looking at this and then one of the editors said to us, it's a lot of work
guys for the sake of data being a minute stale, I think we're okay for this game.

419
00:29:10,587 --> 00:29:11,947
So that's just not that on the head.

420
00:29:11,947 --> 00:29:15,859
We live with a minute cache and great music to our ears.

421
00:29:15,859 --> 00:29:18,630
Cause what's the old joke about the...

422
00:29:18,705 --> 00:29:24,846
The two hardest things in computer science are cache and validation, naming things, and
off by one errors.

423
00:29:25,446 --> 00:29:31,766
We were kind of thinking, okay, this could get bad, but no, it worked and it went live and
it did exactly what we needed it to.

424
00:29:31,886 --> 00:29:33,326
It stood up to huge traffic.

425
00:29:33,326 --> 00:29:35,406
It was very popular, it worked really well.

426
00:29:35,406 --> 00:29:38,626
And the whole editorial team didn't have to learn a new tool.

427
00:29:38,626 --> 00:29:46,046
They could live in this Google Sheet, which then once the election is over, we can
snapshot the API response, stick it in an S3 bucket, and it's effectively static now until

428
00:29:46,046 --> 00:29:47,158
the next election.

429
00:29:47,158 --> 00:29:49,570
wind down the worker and job is done.

430
00:29:49,570 --> 00:29:58,998
So that was a nice example, I think, where we took a couple of different smaller
technologies and kind of plugged them together in ways that were probably not on the

431
00:29:58,998 --> 00:30:01,130
manual when you go through Google Sheets.

432
00:30:01,130 --> 00:30:02,491
Here's what you should do with this.

433
00:30:02,491 --> 00:30:05,984
it worked and it worked really, really well in this case for us.

434
00:30:06,370 --> 00:30:16,136
was actually going to ask you about whether or not you were going to be comfortable with
the SLA's being provided by Google Sheets to be uh run for the, you know.

435
00:30:16,136 --> 00:30:24,468
I'd say success of the news company because you know if you're using one news site and
you're only going there to see the you know up to date or one minute behind election

436
00:30:24,468 --> 00:30:34,131
results and you know it's down you're immediately going to you know switch engines you
know maybe permanently to somewhere else but I think the switching over to cloud flare

437
00:30:34,131 --> 00:30:43,484
pretty much put you in a full serverless land right like your database is Google Sheets
and everything else is being cached and I think every cloud provider offers some sort of

438
00:30:43,484 --> 00:30:44,704
strategy like that.

439
00:30:45,066 --> 00:30:46,057
Yeah, yeah.

440
00:30:46,057 --> 00:30:47,967
it worked like a charm with Cloudfair.

441
00:30:47,967 --> 00:30:56,891
Once you managed to get the different layers tuned between the worker and then the caching
rules at the account level, it was great and it went really, really smoothly.

442
00:30:56,891 --> 00:31:03,414
Now, it went really, really smoothly because I think we've had one experience of not
tuning those things correctly in the past.

443
00:31:03,414 --> 00:31:12,042
ah The caching layers, I'd say there's very few problems I've come across over the years
that have caused me as much pain and sleepless nights as...

444
00:31:12,042 --> 00:31:14,123
misconfigured cache or multiple cache layers.

445
00:31:14,123 --> 00:31:17,225
Because you know yourself, people are working on an application.

446
00:31:17,285 --> 00:31:22,698
On day one, there's a very well-considered and thought out caching strategy, but then over
time, things can drift.

447
00:31:22,698 --> 00:31:26,340
Suddenly there's a uh slow part of the application over here.

448
00:31:26,340 --> 00:31:28,691
well, let's just wrap a cache around this particular function.

449
00:31:28,691 --> 00:31:30,922
just, now this part is slow.

450
00:31:30,922 --> 00:31:32,553
Well, let's put a cache over here as well.

451
00:31:32,553 --> 00:31:37,506
And it's like you have a giant rug and you're just shoving all these lumpy things under
it.

452
00:31:37,506 --> 00:31:41,518
know that over time it builds up and you can end up then at one point where

453
00:31:41,518 --> 00:31:48,078
Okay, we have a cache on this particular resource retrieval and we have a cache on the
formatting around it and then we have a cache on this thing getting injected and then we

454
00:31:48,078 --> 00:31:52,058
have a response cache and maybe then there's a browser cache going out to the users.

455
00:31:52,398 --> 00:32:01,018
And when you're trying to purge these, you're working with news websites and when you work
with news websites, it is inevitable that someone is going to make a mistake at some

456
00:32:01,018 --> 00:32:01,558
point.

457
00:32:01,558 --> 00:32:10,822
And you'll write a big article about someone being convicted of an absolutely horrible
crime, this disgusting, despicable stuff and the photograph will be

458
00:32:10,830 --> 00:32:14,790
some guy who ran a marathon for a local charity, he's been accidentally put on the wrong
thing.

459
00:32:14,790 --> 00:32:16,650
Something silly like that, it happens.

460
00:32:16,650 --> 00:32:16,950
It's human error.

461
00:32:16,950 --> 00:32:20,430
It happens more than you'd probably expect.

462
00:32:21,150 --> 00:32:25,890
There's other things that leads to people's hair getting on fire and saying, is going to
and vanish really quickly.

463
00:32:25,930 --> 00:32:34,330
And it can be incredibly frustrating when you have your nice caching strategy that was
there on day one, you've your nice cash invalidation strategy that was there on day one.

464
00:32:34,410 --> 00:32:37,900
The caching structure and the invalidation structure may not have kept up with each other.

465
00:32:37,900 --> 00:32:46,223
So you can end up with these really uh hairy problems where, at uh a simple level, let's
say you have varnish cache around an application, you have a CloudFair cache at the edge,

466
00:32:46,223 --> 00:32:48,134
and it's going off to the user.

467
00:32:48,134 --> 00:32:55,427
But when you're trying to purge these, if you purge them in slightly the wrong order, or
you're doing them async and they execute in slightly the wrong order, you clear your

468
00:32:55,427 --> 00:32:56,278
CloudFair cache.

469
00:32:56,278 --> 00:32:57,638
Great, that's nice and fresh.

470
00:32:57,638 --> 00:32:59,569
Your internal varnish cache hasn't cleared.

471
00:32:59,569 --> 00:33:00,619
A new request comes in.

472
00:33:00,619 --> 00:33:01,840
CloudFair gets the old data.

473
00:33:01,840 --> 00:33:02,840
It recaches it.

474
00:33:02,840 --> 00:33:05,281
Now your varnish thing is cleared, but CloudFair is gone.

475
00:33:05,281 --> 00:33:06,936
So you sometimes end up with, well,

476
00:33:06,936 --> 00:33:08,687
What's the policy of clearing the cache guys?

477
00:33:08,687 --> 00:33:13,071
Or you just keep clicking this kill all caches button until eventually it works.

478
00:33:13,071 --> 00:33:22,769
It can be very stressful if that kind of strategy isn't well taught out, isn't well tested
and isn't structured because it can be a big source of pain, I think.

479
00:33:22,769 --> 00:33:31,425
And not just in the reputational sense, in news organizations, pretty much every app
that's using cache, it's using it for some sort of sensitive performance data.

480
00:33:32,878 --> 00:33:42,640
conscious of the cash-in validation strategy and making sure it's up to date is a key way
to avoiding these kind of hyper-stress moments when something big comes out of the blue

481
00:33:42,640 --> 00:33:46,053
and you're suddenly fighting against the screaming noise.

482
00:33:46,680 --> 00:33:55,744
Well, you and I think that's, know, you had another practical success here as well,
because the newsagent is like, okay, you know, we'll just let the data be stale, you know,

483
00:33:55,744 --> 00:33:56,415
for one minute.

484
00:33:56,415 --> 00:34:05,799
It's like celebration time, honestly, because trying to figure out how to get in the
invalidations and then have them actually work correctly is just like, well, you know,

485
00:34:05,799 --> 00:34:06,719
good luck.

486
00:34:06,920 --> 00:34:10,461
in a way you sort of lucked out because obviously for the,

487
00:34:11,554 --> 00:34:18,900
like Irish elections, only the edges in Ireland were particularly of uh note here at the
moment.

488
00:34:18,900 --> 00:34:27,036
And who cares if there's even a 10 minute delay on invalidation if someone's loading the
data from another country or whatnot.

489
00:34:27,036 --> 00:34:41,277
And assuming the news provider was even willing to share it publicly, because I know a lot
of uh newspapers or online content just is region locked for whatever reason.

490
00:34:41,449 --> 00:34:49,513
A long time ago, I was advising actually uh the corollary in Switzerland early before an
election.

491
00:34:49,513 --> 00:34:56,256
They had run all of their technology on-prem and had a smart idea of finally moving to the
cloud.

492
00:34:56,256 --> 00:34:58,327
know, caching was a huge problem for them as well.

493
00:34:58,327 --> 00:35:06,601
And I think this is one thing that a lot of people who aren't in the content domain don't
realize is that it's not like you just have a lot of public data that just has to be

494
00:35:06,601 --> 00:35:07,415
exactly cached.

495
00:35:07,415 --> 00:35:08,012
There's a like

496
00:35:08,012 --> 00:35:14,304
a lot of small little tweaks, A-B testing that ends up getting in the cache data and for
them it was like paywall stuff.

497
00:35:14,304 --> 00:35:23,846
It's like you don't want to cache per user and cache on like authorization tokens or
bearer tokens or whatever because it's the same data for everyone but you don't want it to

498
00:35:23,846 --> 00:35:27,787
be public because then otherwise you wouldn't need to log in to get it.

499
00:35:27,848 --> 00:35:32,669
getting that right is really uh quite a tricky thing and same goes with invalidation.

500
00:35:32,669 --> 00:35:36,780
Like you remove stuff from there and search bots or search engines,

501
00:35:36,780 --> 00:35:43,593
that have their bot that are going and scraping websites, they want to see like a 410 gone
when content gets removed and they don't want to see a 404.

502
00:35:43,593 --> 00:35:52,597
And it's like, okay, well, that means we need to keep track of when things were deleted
and have a note there and then expose that, but only if it's for the Google bot and not

503
00:35:52,597 --> 00:35:55,708
for like actual users, because that would be a confusing experience.

504
00:35:55,708 --> 00:36:00,680
so, but then that data could get cached and we're not paying attention to the user agent
who's actually calling us.

505
00:36:00,680 --> 00:36:04,482
And, you know, I think that's where you start to realize, actually.

506
00:36:04,482 --> 00:36:07,636
This is a little bit more complicated and it has actually nothing to do with caching.

507
00:36:07,636 --> 00:36:09,526
But then there's caching on top of that.

508
00:36:09,526 --> 00:36:10,366
Yeah.

509
00:36:10,386 --> 00:36:12,548
So, so many layers, it turtles all the way down.

510
00:36:12,548 --> 00:36:19,874
And then you get into problems, you know, you're supposed to send one code back to the
Google bot, but Google's guidelines are also very strong on not treating the Google bot

511
00:36:19,874 --> 00:36:21,175
differently than regular users.

512
00:36:21,175 --> 00:36:30,812
Well, so, know, you're damned if you do and you're damned if you don't, but it's a, yeah,
the caching in particular can be, it's one of those things that the day you first put it

513
00:36:30,812 --> 00:36:32,883
in, this is the greatest thing in the world.

514
00:36:32,883 --> 00:36:36,790
Now my app is hyper-performant and it's brilliant and I'm going to cache everything, you
know, suddenly.

515
00:36:36,790 --> 00:36:43,668
have the cash hammer and everything looks like a nail and then eh it's when you start need
to flush the cash as that thing start to get a little bit a little bit hairier.

516
00:36:43,758 --> 00:36:52,158
Yeah, and I think I think that isn't really evaluated enough and it usually is a result of
major production downtime because when you put the caches in place, what people don't

517
00:36:52,158 --> 00:36:56,458
consider is that they're not planning for the failure mode when the cache gets flushed.

518
00:36:56,458 --> 00:36:59,298
They're looking at the new steady state resource usage.

519
00:36:59,298 --> 00:37:06,178
So if you have a cache in front of your in front of your service that you put there
because the underlying database was slow and you didn't want to scale it up or you

520
00:37:06,178 --> 00:37:13,712
couldn't because you had some sort of index that just wasn't well optimized or couldn't be
given the data that it was querying for or it's a third party server.

521
00:37:13,712 --> 00:37:20,112
right like Google Sheets and you don't have any control over it in the first place and you
put the cash there and then all of a sudden you're like oh well some of the data and the

522
00:37:20,112 --> 00:37:27,532
cash is wrong let's flush it well you flush it and now all those new requests are coming
directly to your database and will overwhelm it and now not only do you not have the data

523
00:37:27,532 --> 00:37:37,872
in your cash no one's getting any results up and now it's like a real production incident
and you just don't really think about the long-term impact of setting up that that system

524
00:37:37,872 --> 00:37:42,534
and so one of the things that I want to ask about and I think does come up

525
00:37:42,552 --> 00:37:49,918
frequently in the conversations I had, especially around serverless stuff is like, how do
you evaluate the short, medium, long term for some of these options?

526
00:37:49,918 --> 00:37:54,212
Like I see especially like very staunch serverless component uh proponents.

527
00:37:54,212 --> 00:38:02,719
Like I'll say I'm one of them, but I try to take this approach of like, well, are we going
to end up in a situation where we have like tens of thousands of these cloud flare workers

528
00:38:02,719 --> 00:38:04,340
all doing independent actions?

529
00:38:04,340 --> 00:38:08,424
How are we going to keep track of them and know what they're supposed to be doing and even
manage that?

530
00:38:08,424 --> 00:38:10,746
Are those like separate repositories or whatnot?

531
00:38:10,746 --> 00:38:11,776
I assume.

532
00:38:12,237 --> 00:38:14,283
You've seen this in some regard.

533
00:38:14,444 --> 00:38:15,154
Yeah, yeah.

534
00:38:15,154 --> 00:38:22,076
think like your point about all the different work is it very similar to a conversation
that you'll have about say microservices versus the monolith.

535
00:38:22,076 --> 00:38:24,437
You're breaking these things down into a million small pieces.

536
00:38:24,437 --> 00:38:34,229
And for those who are say microservice skeptic, you'd be pointing to the cases where now
to debug one request, I need to spin up 15 microservices locally and try chasing them all

537
00:38:34,229 --> 00:38:34,589
through.

538
00:38:34,589 --> 00:38:42,702
And that kind of holistic view you're talking about there of having an idea of what lives
where and what is doing what, it can really get

539
00:38:43,854 --> 00:38:46,574
complex very quickly as you're breaking down these components.

540
00:38:46,574 --> 00:38:55,074
So I think that that's always the trade off with microservices, small workers, whatever it
is that you're going to do is the extra velocity you get from breaking these things out

541
00:38:55,074 --> 00:39:03,394
into individual components that have just one thing to do and one thing to focus on and
have a clearly defined boundary versus the complexity of, okay, now I have four of these

542
00:39:03,394 --> 00:39:03,614
things.

543
00:39:03,614 --> 00:39:05,954
How many nodes do I have of how they're interacting with each other?

544
00:39:05,954 --> 00:39:12,738
And how many things do I need to load into my own mental model to do this to kind of get a
handle on it as those boundaries are

545
00:39:12,738 --> 00:39:20,292
I think more often than not where things start to fail and things start to get, let's
diplomatically say interesting, gets very colorful.

546
00:39:20,292 --> 00:39:28,757
Whether it's the microservices or you're working with a very complex Kubernetes stack, for
example, and suddenly you're trying to find there's one pod randomly in the middle

547
00:39:28,757 --> 00:39:34,470
somewhere that is misfiring for whatever reason, ah pinning it down and diagnosing it.

548
00:39:34,470 --> 00:39:41,494
It's a lot more challenging than when everything used to run on just one massive server
sitting in the corner somewhere and you can shell in and dig your way through it.

549
00:39:41,494 --> 00:39:42,264
ah

550
00:39:42,264 --> 00:39:44,146
There's definitely trade-offs to be had there.

551
00:39:44,146 --> 00:39:45,407
And it is difficult.

552
00:39:45,407 --> 00:39:52,172
It is difficult to keep that high level view of what's happening, where, what we need to
be aware of going on throughout our whole system.

553
00:39:52,172 --> 00:39:56,966
When we ourselves are straddling more than one of these services, you know, it's not all
isolated teams.

554
00:39:56,966 --> 00:39:58,547
That can be a very hard view to get.

555
00:39:58,547 --> 00:40:08,545
I think that's one of the areas where some of the AI tooling is certainly helping with,
because what we're talking about here is it's effectively documentation, system

556
00:40:08,545 --> 00:40:10,637
documentation.

557
00:40:10,637 --> 00:40:11,904
you know, developers...

558
00:40:11,904 --> 00:40:20,474
Most developers are not developers because they love sitting down and writing 15 page docs
that are out of date 10 minutes after you've hit publish on it.

559
00:40:20,718 --> 00:40:29,973
But I think with the AEI tools now, particularly the last few iterations of them, they're
getting stronger and stronger and better and better to a degree where there's really less

560
00:40:29,973 --> 00:40:36,718
of an excuse to not have this type of documentation available or this type of
documentation not only available, but current as well.

561
00:40:36,718 --> 00:40:48,278
I don't know if you've seen many of the more recent models, but I know say over Christmas,
for example, there's been a huge boon in people talking about the Opus 4.5, Pink Law Code

562
00:40:48,278 --> 00:40:58,378
and new versions of Codex that they've kind of hit a certain inflection point where there
suddenly now a lot more stuff just works where it wasn't six or nine months ago.

563
00:40:58,378 --> 00:41:02,158
things like generating better documentation, you know, getting

564
00:41:02,158 --> 00:41:09,718
getting graphs and flow charts that actually look like graphs and flow charts and not some
weird ASCII art thing going on is easier.

565
00:41:09,718 --> 00:41:12,598
It's happening and it's quick to do and it's quick to have.

566
00:41:12,678 --> 00:41:19,758
And then it's quick to then feed back to these tools when something goes wrong and says,
listen, I have 15 different services documented here.

567
00:41:19,758 --> 00:41:20,878
Something is breaking somewhere.

568
00:41:20,878 --> 00:41:21,718
Help.

569
00:41:21,998 --> 00:41:22,858
I'll look over here.

570
00:41:22,858 --> 00:41:26,358
You look at all of this stuff and together we'll figure it out between us.

571
00:41:26,478 --> 00:41:31,064
So I think that's an area that is going to be more helpful over time.

572
00:41:31,064 --> 00:41:42,099
The usual caveats about AI hallucinations and stuff needing to be sanity checked, but
degenerating of that documentation and updating it is now a much, much smaller task than

573
00:41:42,099 --> 00:41:46,634
it was 12 or 18 months ago, I think, to do this.

574
00:41:46,902 --> 00:41:49,559
Also color me naive here.

575
00:41:49,559 --> 00:41:52,067
Why do we need documentation with LLMs?

576
00:41:52,067 --> 00:41:55,745
Can't we just have it be a text prompt that gives us the answer?

577
00:41:56,878 --> 00:42:00,950
Well, I suppose what I'm talking about here is, well, we could, a text prompt.

578
00:42:00,950 --> 00:42:03,582
So let's go to the LLM and say, my system's broken, please fix this.

579
00:42:03,582 --> 00:42:05,473
I'll tell you no more, figure it out for yourself.

580
00:42:05,473 --> 00:42:07,224
mean, that's the dream stage.

581
00:42:07,224 --> 00:42:14,548
ah Well, I'm talking more about how, know, historically when we're putting together these
microservices or we have workers or whatever, we might have a repo.

582
00:42:14,548 --> 00:42:19,671
We want to have a readme in the repo that explains what it's doing, what it's not doing,
what it's taking responsibility for.

583
00:42:19,671 --> 00:42:25,102
So when we're trying to chase down, uh this cache thing is misfired and it's gone through
these 16 different channels.

584
00:42:25,102 --> 00:42:32,322
what are each of these channels actually supposed to be doing, rather than having the LLM
go and say, I'm going to analyze this monster code base.

585
00:42:32,322 --> 00:42:37,082
We've effectively distilled the monster code base into, here's the documentation
explaining at a high level.

586
00:42:37,082 --> 00:42:45,902
is what, with any documentation, it's never going to be, we're 100 % sure this is what
it's doing, but we're about maybe 90, 95 % sure this is what we should be doing.

587
00:42:45,902 --> 00:42:48,262
So you could have a double pass.

588
00:42:48,262 --> 00:42:53,462
From your perspective as the incident investigator, yes, you get a text box saying, my
system's broken, go figure it out.

589
00:42:53,462 --> 00:42:55,302
The LLM can do a first pass.

590
00:42:55,470 --> 00:43:03,550
read in the 15 documentations and then figure out, it's in one of these two repos and then
go deep, spin off sub agents that goes deep and says, okay, we really know the ins and

591
00:43:03,550 --> 00:43:09,190
outs of what's going on in the internals of this application over here and go deeper and
deeper.

592
00:43:09,190 --> 00:43:12,730
yeah, ultimately a text box where I can fire in a question and everything gets sorted out
for me.

593
00:43:12,730 --> 00:43:13,850
That's the dream.

594
00:43:14,586 --> 00:43:24,949
I was seeing a pattern here because when you put it like that, the documentation for me is
like a LLM knowledge-based cache where rather than reading through the whole source code

595
00:43:24,949 --> 00:43:34,092
and then having to do that on the fly, cache the results in what you're calling a document
in the readmate, which it can then read back later to generate an answer.

596
00:43:34,092 --> 00:43:35,322
And that seems obvious.

597
00:43:35,322 --> 00:43:40,394
And then you just get into the standards like, well, you're going to have to invalidate
that cache at some point.

598
00:43:40,699 --> 00:43:41,288
I like it.

599
00:43:41,288 --> 00:43:42,167
Yeah, that's it.

600
00:43:42,167 --> 00:43:44,075
That's a great way of putting it.

601
00:43:44,450 --> 00:43:49,094
Maybe this is a good point to switch over to PICS.

602
00:43:49,795 --> 00:43:52,886
I'll ask Paul, what did you bring for the audience today?

603
00:43:52,886 --> 00:43:53,156
Yes.

604
00:43:53,156 --> 00:43:57,409
So my pick is it's the book, the code book by Simon Singh.

605
00:43:57,409 --> 00:44:02,533
So it's all about a cryptography and secrecy from uh ancient Egypt up to about Enigma.

606
00:44:02,533 --> 00:44:08,166
So I love history, but I would say probably more of a pop history fan.

607
00:44:08,166 --> 00:44:13,880
You know, I'm not going to be sitting down with a 1200 page deep college textbook on
history.

608
00:44:13,880 --> 00:44:20,225
I love historical stories that are basically illustrated by anecdotes or whatever personal
information.

609
00:44:20,225 --> 00:44:22,446
And this book is great for that because there's a base.

610
00:44:22,478 --> 00:44:30,878
five or six different eras of cryptography where they're talking about, you know, from
Caesar ciphers back in the day, all the way up to the enigma and what goes on in between.

611
00:44:30,878 --> 00:44:32,918
You know, there's a story in it that I love.

612
00:44:32,918 --> 00:44:38,838
I think I've probably told it to so many people at this stage that I owe Simon Singh
royalties for finding it in the first place.

613
00:44:38,838 --> 00:44:40,658
But it's about the 30 years war.

614
00:44:40,658 --> 00:44:46,098
when the Spanish were going to war with the French, the Spanish also controlled the
Netherlands.

615
00:44:46,098 --> 00:44:49,762
So they needed to get kind of back and forth quite a bit over

616
00:44:49,762 --> 00:44:51,153
dodgy territory.

617
00:44:51,153 --> 00:44:57,185
the problem the Spanish kept having was that every time their armies and their diplomats
were moving around, the French were there waiting.

618
00:44:57,185 --> 00:44:58,816
They were absolutely hammering them.

619
00:44:58,816 --> 00:45:00,386
They were destroying them militarily.

620
00:45:00,386 --> 00:45:01,297
They knew all of their plans.

621
00:45:01,297 --> 00:45:02,257
They knew all of their strategy.

622
00:45:02,257 --> 00:45:08,170
So the Spanish went through the normal stuff of, there's double agents and let's purge a
whole bunch of animals and whatever.

623
00:45:08,170 --> 00:45:09,990
But it turned out that this wasn't happening at all.

624
00:45:09,990 --> 00:45:14,092
Spanish, we're using really strong eh cryptography.

625
00:45:14,092 --> 00:45:19,320
It wasn't a Caesar cipher, was like a slightly modified version of that where you're
swapping out.

626
00:45:19,320 --> 00:45:21,512
Instead of an A, you're using 127.

627
00:45:21,512 --> 00:45:23,261
Instead of an E, it's 329.

628
00:45:23,261 --> 00:45:27,856
So it's not subject to the same sort of frequency analysis you might get from just the
plain text.

629
00:45:27,856 --> 00:45:31,619
So they were confident that they had this amazing system in place.

630
00:45:31,619 --> 00:45:33,290
So how on are the French breaking it?

631
00:45:33,290 --> 00:45:37,643
They've gotten rid of anyone who could have been a double agent, but the French were just
routinely breaking it.

632
00:45:37,643 --> 00:45:41,626
So they went to the Pope and the Pope at the time, you know, was kind of the boss of
Europe, boss of all the Kings.

633
00:45:41,626 --> 00:45:44,008
ah But he also had his own army there.

634
00:45:44,008 --> 00:45:46,850
And the Spanish went to him and says, you know, your holiness.

635
00:45:47,150 --> 00:45:48,610
This makes no sense.

636
00:45:48,810 --> 00:45:51,610
We're using the best cryptography in the world, but they're reading it.

637
00:45:51,610 --> 00:45:54,910
The only way that this could be happening is if there's something else going on here.

638
00:45:54,910 --> 00:46:00,290
The King of France has clearly made a deal with the devil and sold his soul to get our
cryptography.

639
00:46:00,290 --> 00:46:04,810
It's the only possible, this may be the 17th century equivalent of there's a bug in the
compiler.

640
00:46:04,810 --> 00:46:08,010
You're going to devil did it.

641
00:46:08,010 --> 00:46:14,430
then Pope heard them very patiently and says, yeah, I'm not going to excommunicate him
because the Pope had also broken their cryptography.

642
00:46:14,430 --> 00:46:15,598
It turned out that

643
00:46:15,598 --> 00:46:22,478
What they were doing was they had like a key and this sort of key was under lock and key
was secure.

644
00:46:22,478 --> 00:46:24,418
It was kind of passed around, but it wasn't changing.

645
00:46:24,418 --> 00:46:28,738
So operationally they kept it the same and it they were using the same key for their
messages all of the time.

646
00:46:28,738 --> 00:46:37,538
And while they were swapping, you know, letter A for one, two, seven, whatever it was, it
wasn't subject to basic frequency analysis, but with a bit more time, it's still

647
00:46:37,538 --> 00:46:38,258
effectively subject.

648
00:46:38,258 --> 00:46:44,678
And if you think about diplomatic cables going back and forth, a lot of them start with my
esteemed Lord or

649
00:46:45,129 --> 00:46:47,390
There's phrases in them that you can start to pick apart.

650
00:46:47,390 --> 00:46:51,150
And when you're using the same keys for so long, the French eventually just figured it
out.

651
00:46:51,150 --> 00:46:52,550
That's okay, these are the maps.

652
00:46:52,810 --> 00:46:57,170
And so did the Pope's armies and so did the British and pretty much everybody except for
the Spanish.

653
00:46:57,170 --> 00:47:05,430
this was a lesson in the importance of your keys not being static for too long.

654
00:47:06,130 --> 00:47:10,136
And assuming that different threat models do exist because after this for...

655
00:47:10,136 --> 00:47:15,978
couple of centuries, the Spanish cryptographer was used as a sort of pejorative term
around Europe in security terms.

656
00:47:15,978 --> 00:47:20,919
So that is one of many, I could probably fill a whole episode telling you the different
stories in this book.

657
00:47:20,919 --> 00:47:25,180
So it's one I would definitely recommend if you're into, it's a fairly easy read.

658
00:47:25,180 --> 00:47:29,811
It's not going in and out of the full algorithms behind quantum cryptography or anything
like that.

659
00:47:29,811 --> 00:47:30,792
It's a nice light read.

660
00:47:30,792 --> 00:47:33,122
It sort of blitzes through the different ages and very distinct chapters.

661
00:47:33,122 --> 00:47:35,063
So I enjoyed it quite a lot.

662
00:47:35,063 --> 00:47:39,774
So I would recommend that to anyone who's even tangently interested in the area.

663
00:47:40,022 --> 00:47:40,723
I like your pick.

664
00:47:40,723 --> 00:47:41,853
think it's super relevant.

665
00:47:41,853 --> 00:47:48,758
I'm sure there's at least one person who's interested in reading that book and I'm going
to put it on my list for my vacation to pick up and read.

666
00:47:48,758 --> 00:47:51,971
I Thank you Paul for bringing that.

667
00:47:51,971 --> 00:47:53,212
I guess I'll move over to mine.

668
00:47:53,212 --> 00:47:57,745
So my pick is I'm living in Switzerland so I go hiking a lot.

669
00:47:57,745 --> 00:47:59,436
So I had to I had to bring a hiking shoe.

670
00:47:59,436 --> 00:48:02,608
I just recently bought this and I'm sure I'm going to get some some crap for it.

671
00:48:02,608 --> 00:48:03,559
I'll just hold it up here.

672
00:48:03,559 --> 00:48:06,701
It's a it's a north face.

673
00:48:06,701 --> 00:48:07,942
uh

674
00:48:07,980 --> 00:48:17,756
Hedgehog hiking six I think is the actual version and yeah, I know what people are
thinking that like North Face isn't known for their uh quality hiking gear.

675
00:48:17,756 --> 00:48:21,798
ah It's a glamping uh more than anything.

676
00:48:21,798 --> 00:48:30,743
But honestly, I feel like there's like there's some that they've locked into that are
still quality that are that I like have been managing to still get for like the last

677
00:48:30,743 --> 00:48:31,544
decade or so.

678
00:48:31,544 --> 00:48:35,456
uh I don't know what it is and maybe people will say that I'm

679
00:48:35,456 --> 00:48:38,887
I'm just shilling their brand.

680
00:48:38,887 --> 00:48:48,570
Like honestly, like I think it was an accident that they started making like actual useful
hiking gear and the boots are just one of the things that I just I just really like.

681
00:48:48,570 --> 00:48:50,750
um You have to get them with the vibram souls.

682
00:48:50,750 --> 00:48:56,572
There's a lot of knockoffs like even from North Face that are just they're there to just
be worn and not actual hike in.

683
00:48:56,572 --> 00:48:58,133
So you have to be careful which ones you go for.

684
00:48:58,133 --> 00:49:01,554
But these waterproof and everything really sturdy.

685
00:49:01,554 --> 00:49:03,144
I absolutely love them.

686
00:49:03,562 --> 00:49:12,386
And I'm not a huge hiker myself, but what would you say is the of the main difference with
them and sort of regular hiking shoes is, it the comfort in the soles or more reliable

687
00:49:12,386 --> 00:49:13,177
water?

688
00:49:13,272 --> 00:49:16,303
Okay.

689
00:49:16,303 --> 00:49:18,124
I think this is where it's different for everyone.

690
00:49:18,124 --> 00:49:24,627
uh When you're when you're buying hiking stuff, it really is like each brand makes subtly
different sizes and shapes.

691
00:49:24,627 --> 00:49:32,380
And so it's hard to say that it's you know, it's definitely better in some way than than
other ones like I would have definitely preferred to go with a more traditional company

692
00:49:32,380 --> 00:49:33,540
when it comes to hiking gear.

693
00:49:33,540 --> 00:49:34,871
But a lot of them.

694
00:49:34,871 --> 00:49:38,302
uh They don't fit my foot well, because do I have special feet?

695
00:49:38,302 --> 00:49:39,292
Maybe.

696
00:49:40,013 --> 00:49:42,134
But for whatever reason, I really like these.

697
00:49:42,134 --> 00:49:42,934
think

698
00:49:43,106 --> 00:49:52,773
For me, it's like light and malleable, but you have to be careful when you buy hiking
boots or even shoes because the malleability on the bottom of the shoe causes your foot to

699
00:49:52,773 --> 00:49:57,837
do more work when you're walking on say non flat surfaces, rocks and whatnot.

700
00:49:58,238 --> 00:49:59,859
terrain is also super important.

701
00:49:59,859 --> 00:50:05,763
ah If I say like you're in Switzerland and you're doing T2s and T3s, it could be wet and
you don't want to slip.

702
00:50:05,763 --> 00:50:07,965
ah You may need ankle support or not.

703
00:50:07,965 --> 00:50:09,438
Like these are great.

704
00:50:09,438 --> 00:50:14,528
you go up to T fours or T fives, or you're walking on paved roads or whatnot.

705
00:50:14,528 --> 00:50:14,949
Yeah.

706
00:50:14,949 --> 00:50:16,852
Get a different, get a different shoe for sure.

707
00:50:16,852 --> 00:50:20,759
Well, thank you so much, Paul, for coming on and joining us in this episode.

708
00:50:20,759 --> 00:50:22,530
It's been absolutely fantastic.

709
00:50:22,530 --> 00:50:23,163
Yeah, great stuff.

710
00:50:23,163 --> 00:50:24,003
I know I had a blast.

711
00:50:24,003 --> 00:50:24,429
was really great.

712
00:50:24,429 --> 00:50:25,502
Thanks for having me on.

713
00:50:25,698 --> 00:50:28,070
Yeah, and would happy to do again.

714
00:50:28,070 --> 00:50:30,852
And thank you for Rootly for sponsoring this episode.

715
00:50:30,852 --> 00:50:35,942
And I appreciate all the listeners and hopefully you'll join us back again next week.