1
00:00:00,934 --> 00:00:01,635
okay.

2
00:00:01,635 --> 00:00:07,203
So just some pre show stuff, some administration for us to get through before we actually
we actually start.

3
00:00:07,203 --> 00:00:11,369
Um You said you've you've done this before?

4
00:00:11,450 --> 00:00:12,943
it's good to hear.

5
00:00:12,943 --> 00:00:14,724
exactly exactly once.

6
00:00:14,724 --> 00:00:21,960
I I won't say I built up any expertise other than getting over the barrier of being really
nervous about it.

7
00:00:24,507 --> 00:00:26,819
Well, there's no reason to be nervous, hopefully.

8
00:00:26,819 --> 00:00:29,862
I I'll say that we go through an editing process.

9
00:00:29,862 --> 00:00:39,080
So if you feel like you said something in a weird way and you wanna try to phrase it
differently, feel free to take a moment, collect your thoughts, and jump in and repeat it

10
00:00:39,080 --> 00:00:41,252
just differently and we'll take it out in the edit.

11
00:00:41,252 --> 00:00:42,273
That's no big deal.

12
00:00:42,273 --> 00:00:47,437
My goal is to make you seem like you're the best guest that has ever been on the show.

13
00:00:47,738 --> 00:00:51,497
So uh we're gonna try to do our job anyway to

14
00:00:51,636 --> 00:01:01,069
cut out anything that seems like it it wasn't the most riveting content uh for for the
audience so that your the best parts of you shine through.

15
00:01:01,184 --> 00:01:02,026
Okay.

16
00:01:02,026 --> 00:01:03,497
Yeah, sounds good.

17
00:01:03,729 --> 00:01:07,030
Uh I I d it is sort of a conversation though.

18
00:01:07,030 --> 00:01:11,312
I I like to have it be uh more flexible and fluid organic.

19
00:01:11,312 --> 00:01:14,854
So uh sometimes we do get guests that go on long rants.

20
00:01:14,854 --> 00:01:22,147
Rants are great, but if you get around like the five minute mark, I may put my hand up and
be like, Mark, we caught some great stuff there.

21
00:01:22,147 --> 00:01:30,790
Let's let's maybe dive into something about that or or flip the conversation or go on uh a
different direction, um, just so that we can make sure we capture as much as we can.

22
00:01:30,790 --> 00:01:32,961
Uh that's that's valuable.

23
00:01:34,208 --> 00:01:34,825
Okay.

24
00:01:34,825 --> 00:01:36,383
Yeah, sounds good.

25
00:01:36,483 --> 00:01:37,403
okay.

26
00:01:37,634 --> 00:01:39,205
the audience has varied backgrounds.

27
00:01:39,205 --> 00:01:47,260
So uh even if you are describing something that you think everyone should know, feel free
to take a moment and and dive into it more, spend a couple minutes sort of explaining what

28
00:01:47,260 --> 00:01:51,693
that is or why it's important, um, some of the challenges with it, that that's always
great.

29
00:01:51,693 --> 00:02:00,999
I will say we have had episodes on lots of things, ML um related or AI or agent related
skills, agents, MCP, RAG, etc.

30
00:02:01,179 --> 00:02:04,101
so don't feel like you need to describe those, but if it's relevant to

31
00:02:04,101 --> 00:02:08,571
the topic that w that is at hand, you know, it's always great to go into it in more
detail.

32
00:02:08,571 --> 00:02:12,238
It doesn't matter how many times we re describe a topic, it's totally fine.

33
00:02:12,238 --> 00:02:15,715
Having your perspective is always additionally valuable.

34
00:02:17,102 --> 00:02:17,809
Okay.

35
00:02:17,809 --> 00:02:19,721
Yeah, that sounds good.

36
00:02:21,041 --> 00:02:31,159
Uh most importantly, we want to capture sort of concrete stories and personal accounts
from your experiences, uh, things that you're currently working on or challenges for the

37
00:02:31,159 --> 00:02:35,873
team that have come up, uh, your technical experiences from the past, anything like that.

38
00:02:35,873 --> 00:02:39,826
I know part of your history is at Meta.

39
00:02:39,826 --> 00:02:42,118
I may ask you some questions about that depending on the conversation.

40
00:02:42,118 --> 00:02:48,583
Um feel free to go as deep and technical as as you can remember or you can think about.

41
00:02:48,583 --> 00:02:51,055
It's always great to sort of dive into that.

42
00:02:51,060 --> 00:02:53,496
that sits really well with the audience.

43
00:02:53,976 --> 00:02:54,732
All right.

44
00:02:54,732 --> 00:02:55,256
Yeah.

45
00:02:55,256 --> 00:02:56,691
All of that sounds good.

46
00:02:56,779 --> 00:02:57,279
Okay.

47
00:02:57,279 --> 00:03:01,052
Uh a couple more things and then uh we'll be good to go, I swear.

48
00:03:01,052 --> 00:03:08,729
Uh first thing is uh at the end of the email uh that I sent, uh there's something you may
have read, maybe you haven't.

49
00:03:08,729 --> 00:03:16,556
We do at the end of the episode what I call PICS, which is bringing something technical or
non-technical for the audience to give a little bit of flavor of who you are and what you

50
00:03:16,556 --> 00:03:17,376
like.

51
00:03:17,537 --> 00:03:22,481
can be a book, a television show, it can be a piece of technology, hardware or software.

52
00:03:22,513 --> 00:03:28,687
Do you already have something that you were thinking about, or should we take a couple of
minutes and sort of figure out what what's your gonna be your thing?

53
00:03:28,687 --> 00:03:30,008
some time on it.

54
00:03:30,008 --> 00:03:31,089
So what's the nature of it?

55
00:03:31,089 --> 00:03:36,734
Is it just like something that's worked on or uh or or or what?

56
00:03:37,530 --> 00:03:40,502
Honestly, it can be anything that's a personal preference.

57
00:03:40,502 --> 00:03:51,277
Like today I I'm bringing this article uh that I read that I think has some interesting
insights, but lots of guests bring uh a book that they've read, science fiction or or or

58
00:03:51,277 --> 00:03:54,259
nonfiction on leadership or particular technology.

59
00:03:54,259 --> 00:03:56,499
Some guests bring a television show.

60
00:03:56,860 --> 00:04:01,582
sometimes someone's like, Here I mean the canonical one is like, here's my keyboard that I
put together.

61
00:04:01,582 --> 00:04:06,184
I love keyboards, here's here's my current keyboard, uh, which you can of course do.

62
00:04:07,493 --> 00:04:09,784
Does this help inspire any anything?

63
00:04:09,878 --> 00:04:17,334
Okay, so it seems to be like either a project or or like piece of uh like media that
someone has consumed.

64
00:04:18,385 --> 00:04:19,395
Could could be.

65
00:04:19,395 --> 00:04:20,361
Like, you know.

66
00:04:20,361 --> 00:04:25,960
spending time on lately that is not like hopefully not too directly related to to work?

67
00:04:26,949 --> 00:04:27,379
Yeah, right.

68
00:04:27,379 --> 00:04:30,821
You know, something not something definitely outside of your your work.

69
00:04:30,821 --> 00:04:34,053
I mean, it could be something that you've been doing separately to help you get better at
work.

70
00:04:34,053 --> 00:04:41,278
You know, maybe there's a particular book that you're reading that you love, you know, a
book about operating systems, or um, I know we're talking about classification today.

71
00:04:41,278 --> 00:04:45,480
You know, that's not directly related to, yeah, I'm using this like a specific tool.

72
00:04:45,480 --> 00:04:48,582
Although it could be like someone's like, yeah, you I love Docker this week.

73
00:04:48,582 --> 00:04:49,389
here's why.

74
00:04:49,389 --> 00:04:50,683
Uh could be that.

75
00:04:50,683 --> 00:04:53,398
Um any thoughts?

76
00:04:53,398 --> 00:04:56,409
Okay, yeah, give me like thirty seconds to think about it then.

77
00:04:56,409 --> 00:04:56,864
Um

78
00:04:56,864 --> 00:04:57,685
for it.

79
00:05:13,578 --> 00:05:19,221
Um these are some like older like personal projects, but I still kinda think about them.

80
00:05:19,221 --> 00:05:34,490
So I I I made these like um like physics simulations uh in my in my my spare time that
just capture um like generating fractals, finding the optimal path for like a rolling ball

81
00:05:34,490 --> 00:05:43,166
down um down the hill, all um like the heat equation dispersion, uh all of that.

82
00:05:43,166 --> 00:05:51,521
Um that's something I spend I I s spent a lot of time on and even if I don't like c a
continue adding code I think about it uh a a lot.

83
00:05:51,521 --> 00:05:54,956
Um I think go ahead.

84
00:05:55,330 --> 00:05:59,405
Um, so one thing we try to do is always like provide some sort of link to whatever the
content is.

85
00:05:59,405 --> 00:06:08,026
So is there like a physics engine that you constantly use to model that stuff or a
particular um I don't know, interface or tool that you're utilizing or something that you

86
00:06:08,026 --> 00:06:10,207
just built up in code directly?

87
00:06:10,207 --> 00:06:22,335
So I I guess like I kind of um I kind of uh like built up this like physics engine over
time in in in in like Haskell to to play around with it and a visualization engine too.

88
00:06:22,335 --> 00:06:31,361
Um I g I guess like if it's in in the sense that's relevant to my story, like it it it
taught me a bunch about programming.

89
00:06:31,361 --> 00:06:32,962
And I can I can kind of share.

90
00:06:32,962 --> 00:06:38,866
Like I I said this in Haskell, like part of my background is like I'm a functional
programming like obsessive.

91
00:06:38,902 --> 00:06:44,136
And and like that's that was kind of like the stepping stone for everything else in in my
career.

92
00:06:44,787 --> 00:06:46,150
Honestly,

93
00:06:47,148 --> 00:06:55,805
And if if you want like if there's like an underlying principle, I I guess it's like of
course I work on other things like besides like functional programming.

94
00:06:55,805 --> 00:06:59,488
Like I have to like we I'm a data software company.

95
00:06:59,488 --> 00:07:01,430
Like we don't it's not all in Haskell.

96
00:07:01,430 --> 00:07:02,600
And like I do sales too.

97
00:07:02,600 --> 00:07:10,477
So the the I guess the underlying pr principle is is it's kind of like my it was kind of
my stepping stone to doing everything else.

98
00:07:10,477 --> 00:07:16,822
Whether that be like learning other things or or um like le learning about what I needed
to start this company

99
00:07:16,822 --> 00:07:18,157
Or yeah.

100
00:07:18,157 --> 00:07:24,989
Uh I I think this is something I even may want to include in the actual episode, more than
a a pick for the end.

101
00:07:24,989 --> 00:07:35,663
Uh I I think it's a really great little piece of anecdote of how you got to where where
you are today, more so than just uh, you know, random throwaway thing.

102
00:07:35,663 --> 00:07:44,006
So uh just to give you like maybe some exam more examples that could be helpful, some
people pick like a like a particular

103
00:07:44,006 --> 00:07:51,691
bike that they have, like they live biking and they bike or hiking shoes or uh let's see
what else have people done recently?

104
00:07:51,691 --> 00:08:03,398
A a particular protein bar for like a new company or like a uh a particular uh smoothie or
shake from a from a shake shop or uh a particular recipe that you like making if you cook.

105
00:08:03,549 --> 00:08:07,801
I don't know if these are giving you any any inspiration in like sort of the non technical
direction.

106
00:08:08,129 --> 00:08:08,915
Um

107
00:08:10,784 --> 00:08:15,692
I guess if if you want something a non technical direction, like I guess um

108
00:08:17,612 --> 00:08:21,624
I started doing a bunch of um like burpees for for exercise.

109
00:08:21,624 --> 00:08:25,867
Especially I I like travel a bunch to do sales and and stuff like that.

110
00:08:25,867 --> 00:08:37,353
I I like exercising, but the problem is um when you're up to a bunch of of of stuff, like
traveling all over the place, then you fall out of the habit.

111
00:08:37,353 --> 00:08:41,336
But like jumping up and down your hotel room is pretty good.

112
00:08:41,336 --> 00:08:45,365
Like uh I'm in my hotel room right now, if uh it wasn't

113
00:08:45,365 --> 00:08:48,087
I no, I I think I think this is great.

114
00:08:48,087 --> 00:08:54,269
Um I I I I think we're gonna have to get Mark's uh exercise routine travel exercise
routine.

115
00:08:54,269 --> 00:08:56,830
it's gonna be a whole a whole new product.

116
00:08:56,830 --> 00:08:59,172
Uh you can push this training out, it could be a learning.

117
00:08:59,172 --> 00:09:00,612
Uh this this is fantastic.

118
00:09:00,612 --> 00:09:02,657
Um for the um

119
00:09:02,657 --> 00:09:05,331
I can talk about hotel room exercise.

120
00:09:05,331 --> 00:09:06,055
Yeah.

121
00:09:06,055 --> 00:09:07,476
Hotel room exercises.

122
00:09:07,476 --> 00:09:08,605
I l I love I love this.

123
00:09:08,605 --> 00:09:11,288
This is this is this is quite a new direction.

124
00:09:11,288 --> 00:09:12,248
Okay.

125
00:09:13,649 --> 00:09:13,999
sure.

126
00:09:13,999 --> 00:09:19,651
Uh and if you think of anything during the episode, you know, feel free to switch your
pick over to to that in instead.

127
00:09:19,651 --> 00:09:24,333
Um sometimes they do do do come up uh spontaneously.

128
00:09:25,794 --> 00:09:33,147
and so second and last thing, at the end of the episode, when I say bye and we close out,
please just stay on

129
00:09:33,147 --> 00:09:35,297
the recording so the post processing can happen.

130
00:09:35,297 --> 00:09:37,013
I'll let you know when you can actually help.

131
00:09:37,537 --> 00:09:38,900
Okay, cool.

132
00:09:39,813 --> 00:09:43,176
And the last thing is y um your name and role.

133
00:09:43,176 --> 00:09:46,710
I think I have here co-founder and CTO at TextQL and it's Mark Hay.

134
00:09:46,710 --> 00:09:47,170
Is that right?

135
00:09:47,170 --> 00:09:49,402
Okay.

136
00:09:49,402 --> 00:09:56,030
Um so before we jump in and get this started, any outstanding questions for me?

137
00:09:56,030 --> 00:10:00,212
um is like the camera used here uh at at all?

138
00:10:00,212 --> 00:10:00,733
Okay.

139
00:10:00,733 --> 00:10:01,514
Cool.

140
00:10:01,737 --> 00:10:10,983
So we do release on YouTube, but it's a very small fraction of our audience that actually
listens to the um to the recording via that mechanism.

141
00:10:12,226 --> 00:10:13,426
no, that that's that's fine.

142
00:10:13,426 --> 00:10:20,433
I I just wanted to know whether I should be cognizant of like what's in front of the the
camera and and and and stuff.

143
00:10:20,433 --> 00:10:21,441
So so it sounds like yeah.

144
00:10:21,441 --> 00:10:26,886
if so if if something happens, I'm like we can't we we we have to cut that, like, don't
worry, I'll I'll I'll call that out.

145
00:10:26,886 --> 00:10:30,350
And if I don't, you know, our our editor will definitely uh take care of it.

146
00:10:30,350 --> 00:10:30,859
Okay.

147
00:10:30,859 --> 00:10:32,189
Yeah, no problem.

148
00:10:35,641 --> 00:10:36,466
Is that it?

149
00:10:36,466 --> 00:10:38,275
You're just like, let's do it.

150
00:10:38,275 --> 00:10:39,116
Okay.

151
00:10:40,665 --> 00:10:42,107
Is there anything else I should be asking?

152
00:10:42,107 --> 00:10:42,930
Like

153
00:10:47,316 --> 00:10:48,299
no, I don't think so.

154
00:10:48,299 --> 00:10:51,305
Um honestly it

155
00:10:53,445 --> 00:10:57,097
People come from different areas and have different expectations.

156
00:10:57,097 --> 00:11:02,969
So it's it's not it's hard for me to say what what will be valuable or not.

157
00:11:02,969 --> 00:11:12,774
If you feel like we're doing the recording and you get this like overwhelming urge to just
have an answer to a particular question that you're not sure about, feel free to be like,

158
00:11:12,774 --> 00:11:20,727
you know, off camera, you know, hey Warren, can we just, you know, I have this question I
just want to answer and we'll just cut that part out of the episode.

159
00:11:21,620 --> 00:11:22,374
Okay.

160
00:11:22,374 --> 00:11:23,657
Yeah, sounds good.

161
00:11:23,845 --> 00:11:24,665
Okay.

162
00:11:24,866 --> 00:11:28,168
Well then uh we're we're gonna get this show on the road then.

163
00:11:28,168 --> 00:11:31,020
Um let me just finish setting up here.

164
00:11:31,020 --> 00:11:33,651
Okay.

165
00:11:39,760 --> 00:11:40,570
Okay.

166
00:11:41,631 --> 00:11:43,903
Welcome back to Adventures in DevOps.

167
00:11:43,903 --> 00:11:52,340
This week we turn over some of the largest rocks in the ML Tech area as we dive into what
take what it takes to run ML classification.

168
00:11:52,340 --> 00:11:57,925
Our guest previously led Meta's Text Classification Infrastructure and is now the CTO and
co-founder at TechsQL.

169
00:11:57,925 --> 00:12:00,176
Welcome Mark Hay to the show.

170
00:12:01,058 --> 00:12:02,440
Thanks for having me on.

171
00:12:03,801 --> 00:12:13,386
You know, when I think classification, there are aspects like sentiment analysis, um, like
how does a text read and entity recognition, identifying relevant aspects, as well as

172
00:12:13,386 --> 00:12:14,116
automatic labeling.

173
00:12:14,116 --> 00:12:19,688
And I'm sure there's so many more that I haven't I since I haven't done this in such a
long time, even even remember at this point.

174
00:12:19,688 --> 00:12:27,833
While everyone is focusing on like LLMs, just predicting the next word, uh, we're stuck
over here actually trying to solve the hard problems.

175
00:12:28,778 --> 00:12:30,409
Um yeah, for for sure.

176
00:12:30,409 --> 00:12:34,883
I mean I I I guess like in the sense of, hey, I worked on it at at Meta.

177
00:12:34,883 --> 00:12:43,131
Um fortunately or unfortunately, classification is a lot more primitive than that, just by
the definition.

178
00:12:43,131 --> 00:12:47,444
The definition was just like is it like one or zero?

179
00:12:47,444 --> 00:12:49,256
Like just just like true or false.

180
00:12:49,256 --> 00:12:56,782
There's it it's not necessarily linked to any specific technique, like not not linked to
to like logistic regression or

181
00:12:57,100 --> 00:12:59,372
or like GBDTs or anything like that.

182
00:12:59,372 --> 00:13:07,628
The thing I worked on at Meta was um trying to take in ev basically every single event
that we could um on the platform.

183
00:13:07,628 --> 00:13:22,689
So that that includes um like uh Instagram posts, Facebook posts, uh comments on each one,
uh messages sent, even uh even stuff like likes and friend requests and follow requests.

184
00:13:22,689 --> 00:13:24,950
And just um like r

185
00:13:25,240 --> 00:13:30,655
classifying it as as like should we do something about it, like is it abusive or or not?

186
00:13:30,655 --> 00:13:41,904
Abusive meaning like is it selling drugs or is it just generally activity we don't want on
the platform or is it even uh e even like criminal or or or or just spam anything like

187
00:13:41,904 --> 00:13:42,585
that.

188
00:13:42,585 --> 00:13:52,252
And and and kind of the the the whole deal was um like how do you know whether whether
something's bad or not and what technique do you use?

189
00:13:52,273 --> 00:13:52,931
It depends.

190
00:13:52,931 --> 00:13:54,466
And and kind of like

191
00:13:54,466 --> 00:13:59,932
being on the classification infrastructure side of it meant that we had to capture all
that it depends.

192
00:13:59,932 --> 00:14:11,225
So on all of these events we have to run like literally thousands of of of different
techniques, whether that be like regex on on like the actual text.

193
00:14:11,225 --> 00:14:12,446
Um

194
00:14:14,188 --> 00:14:19,262
looking at like various like language embeddings, like media embeddings, stuff like that.

195
00:14:19,262 --> 00:14:24,366
Um to to actually um stuff people might not uh e expect.

196
00:14:24,366 --> 00:14:32,144
Like people I I think when when one says like, hey, finding like spam or bad stuff on on
on Facebook, it's all about like content.

197
00:14:32,144 --> 00:14:37,615
Like can you can you recognize the image or um can you like classify the text as abusive
or not?

198
00:14:37,615 --> 00:14:41,928
But um really for for um for for a lot of it.

199
00:14:41,986 --> 00:14:45,808
the vast majority of the value comes from behavioral features.

200
00:14:45,899 --> 00:14:47,810
so like features about the graph.

201
00:14:47,810 --> 00:14:57,237
Like um has this person just like sent out too many friend requests to to like seemingly
unfamiliar people in it l uh for for the last bit?

202
00:14:57,237 --> 00:15:11,358
Or um is there is there something wrong with like the the rate or or um the unfamiliarity
of the of the folks they're send they're they're sending it out to such that you can

203
00:15:11,358 --> 00:15:21,327
you can maybe like classify a a drug dealer like without looking at a single piece of
content that's actually posted, um, just by looking at like what is their pattern compared

204
00:15:21,327 --> 00:15:24,289
to what is their pattern compared to like a normal person.

205
00:15:24,289 --> 00:15:30,454
Like maybe a drug dealer is they get uh they get a new friend or like five new friends
every week.

206
00:15:30,454 --> 00:15:40,162
They send like two messages to them and then uh and then they get two two more messages
from the same person like three weeks later, and it's it happened over and over again.

207
00:15:40,162 --> 00:15:50,811
You can think you can kind of see like, okay, people try and like hide their activity um i
in in the content, but the the patterns of of what's going on underneath all all tend to

208
00:15:50,811 --> 00:15:54,694
be very much more easily detectable.

209
00:15:54,980 --> 00:15:59,802
It's it's the whole um woman in the red dress coding in the matrix, right?

210
00:15:59,802 --> 00:16:08,816
Like you can look at the code on the screen and have no idea what the actual message is or
what's being rendered for real inside the matrix, but see the pattern of of that, right?

211
00:16:08,816 --> 00:16:13,438
And I feel like that's really interesting what what you're getting at there, which is that
it doesn't matter what the content of the message is.

212
00:16:13,438 --> 00:16:19,451
You can just look at the the pattern of the content to identify the the type of intuit in
individual.

213
00:16:19,451 --> 00:16:25,373
So even if you're uh obscuring your content or using some sort of code in your message.

214
00:16:25,663 --> 00:16:29,844
You can't necessarily obscure how you interact with the platform.

215
00:16:29,844 --> 00:16:31,345
That's quite interesting.

216
00:16:31,345 --> 00:16:36,726
And so one aspect is the accuracy, and another one is the scale.

217
00:16:36,726 --> 00:16:40,687
As you mentioned, you're doing this potentially thousands of times per request.

218
00:16:40,687 --> 00:16:43,348
How do you improve the accuracy really?

219
00:16:43,348 --> 00:16:48,761
Like how did you how did you actually identify what the metadata around

220
00:16:48,761 --> 00:16:58,520
the connections or frequency rates that a, you know, hypothetical drug dealer would would
take versus someone that's just messaging their their friends and family.

221
00:16:58,876 --> 00:17:10,095
so the the good thing about uh the good thing about working at um at at Facebook is that
okay, even even though I I'm even though we try and do everything in an automated manner,

222
00:17:10,095 --> 00:17:22,924
there's still like a huge amount of human reviewers checking things for um or whether that
be just a small sample, um or or even like a larger amount for um okay, i is this thing

223
00:17:22,924 --> 00:17:26,166
like um is this thing legitimate?

224
00:17:26,188 --> 00:17:34,925
like for w what would a general like pattern be uh for for like good content or or bad
content or a good event or a bad event.

225
00:17:34,925 --> 00:17:44,834
And that helps a lot in the ground truth, especially for um especially for like various
statistical classifiers that need like r retraining.

226
00:17:44,834 --> 00:17:51,860
So um the interesting thing about like abuse classification is that it's like somewhat
adversarial.

227
00:17:51,860 --> 00:17:55,074
Like they can kind of catch on to like are you

228
00:17:55,074 --> 00:18:01,456
Like what what are they catching, um what are we catching in in in their behavior and then
try and change it like back and forth.

229
00:18:01,456 --> 00:18:10,478
So we need actually a lot of like readjustment, like rethinking of the approach and so on,
simply because you're playing against an opponent, kind of.

230
00:18:10,478 --> 00:18:15,680
And and your your opponent's gonna make moves to to um try and get around that.

231
00:18:15,680 --> 00:18:24,342
Um of course, like the human is a lot the human people at at Facebook meta are probably a
lot better than than than the robots.

232
00:18:24,342 --> 00:18:24,994
I mean

233
00:18:24,994 --> 00:18:28,575
Definitely like pre LLM this was this was the case.

234
00:18:29,355 --> 00:18:40,538
and so what we do is like we we just constantly take out like samples to to check, okay,
is is this classifier like still reasonable or not, or is it like f completely fall under

235
00:18:40,538 --> 00:18:47,760
like thresholds for like false false positives, like accuracy rate and so on.

236
00:18:47,760 --> 00:18:54,042
And then if we can like retrain it and try and get back um above the threshold,

237
00:18:54,496 --> 00:19:02,612
If not, then like go back to the drawing board and see, okay, how do we change our policy
mix to actually like better solve the thing?

238
00:19:03,035 --> 00:19:17,140
How do you at scale identify that uh say a malicious actor on your platform has I
correctly identified what the classification is keying off of and changing their behavior?

239
00:19:17,363 --> 00:19:19,565
like for instance, ha well, I guess it's two questions.

240
00:19:19,565 --> 00:19:22,997
Number one, how do they learn about where the limits are?

241
00:19:22,997 --> 00:19:32,124
Is it that they're somehow creating lots of accounts on the fly and then trying different
things and seeing which accounts get banned and using that to reason about your sort of

242
00:19:32,124 --> 00:19:34,465
back end uh validation system?

243
00:19:34,846 --> 00:19:44,652
Um I th I think something like that, along with like trying like trying new content and
the and then seeing like what what sticks and and what doesn't.

244
00:19:45,633 --> 00:20:00,924
That kind of means that uh that kind of means that like looking the only really the only
real way to to like stay ahead is by having like a person like look at the data, whether

245
00:20:00,924 --> 00:20:04,674
that be um inaggurate

246
00:20:04,674 --> 00:20:11,013
Like based on their intuition or even like specific pieces of content to to check like
what's what's going on because

247
00:20:12,724 --> 00:20:20,034
Oftentimes like the the trick is very like cul culturally based, uh if that makes sense.

248
00:20:20,034 --> 00:20:22,216
Without going into specifics, like

249
00:20:22,361 --> 00:20:23,875
go go go into p specifics.

250
00:20:23,875 --> 00:20:25,519
Like let let's let's let's get to it.

251
00:20:25,519 --> 00:20:27,562
I I think that's incredibly interesting.

252
00:20:27,562 --> 00:20:28,532
Okay, sure.

253
00:20:28,532 --> 00:20:39,128
Well, um I I it might not be appropriate for this podcast, but the the specific I always
go to is um people trying to sell um penis enlargement pills.

254
00:20:39,128 --> 00:20:44,331
That's probably one of the most uh that's probably the the example I can most clearly
remember.

255
00:20:44,331 --> 00:20:52,196
Um and uh so you can think of it like it it starts out as uh it it just says that.

256
00:20:52,196 --> 00:20:56,887
Like we're we're uh we're selling pills to make your junk bigger and

257
00:20:56,887 --> 00:20:57,394
Yeah.

258
00:20:57,394 --> 00:21:00,216
You can imagine that's like pretty easy to to detect.

259
00:21:00,216 --> 00:21:08,529
Like you can just detect the the yeah, you can regex or just like OCR some some like text
and and and so on.

260
00:21:08,529 --> 00:21:09,006
But then

261
00:21:09,006 --> 00:21:10,658
then the text moves into an image, right?

262
00:21:10,658 --> 00:21:11,598
Like that's the that's the upgrade.

263
00:21:11,598 --> 00:21:16,522
Like first it's first is the text, and then it's like a picture of the text that's it
that's in there.

264
00:21:16,522 --> 00:21:21,706
Uh and I mean you see this in a lot of like game forums or or community chats, right?

265
00:21:21,706 --> 00:21:29,901
Like where they're like, Well, what if we misspell some words where everyone still knows
what the what the thing is, but it's spelled in such a way that the regex doesn't catch

266
00:21:29,901 --> 00:21:29,972
it?

267
00:21:29,972 --> 00:21:32,203
Or like what happens if we put stars between characters?

268
00:21:32,203 --> 00:21:34,688
That's definitely all like first level evasion.

269
00:21:34,688 --> 00:21:34,998
Right.

270
00:21:34,998 --> 00:21:42,483
So there's there's like, yeah, write down text and then like misspell it, like move into
an image.

271
00:21:42,483 --> 00:21:56,072
Um, once it seems like we're detecting it in an image, start like putting lines and stuff
in the image to to kind of like trick a like a naive text detector and and stuff like

272
00:21:56,072 --> 00:21:56,893
that.

273
00:21:56,893 --> 00:21:58,608
And then yeah.

274
00:21:58,608 --> 00:22:00,821
a it's like a reverse capture, I feel like.

275
00:22:00,821 --> 00:22:02,976
Like you're going the opposite direction, right?

276
00:22:02,976 --> 00:22:06,208
It's uh it's not you're trying Yeah, right?

277
00:22:06,208 --> 00:22:15,742
the person the and and then like the the the abusive actor is is the one like making the
capture, trying to to go around with stuff.

278
00:22:15,796 --> 00:22:17,322
Right, right.

279
00:22:17,974 --> 00:22:24,699
And the uh and and and I guess like the the kind of like highest level is to like go into
the symbolic realm.

280
00:22:24,699 --> 00:22:28,201
Like just have a picture of of like like two cucumbers.

281
00:22:28,201 --> 00:22:31,444
Like the like the left one is smaller than than the right one.

282
00:22:31,444 --> 00:22:38,188
And then and like not nothing else other than like like go to this website or or or or
like all this number.

283
00:22:38,188 --> 00:22:41,871
And then and and that's uh that that's really hard.

284
00:22:41,871 --> 00:22:46,328
And and that's kind of like you have to you have to have some some some like people.

285
00:22:46,328 --> 00:22:50,880
trying to apply their own in like symbolic intuition to try and figure out how to best
detect it.

286
00:22:50,880 --> 00:22:57,794
Um but definitely like in generality like to totally unsolved territory without like a lot
of cultural expertise.

287
00:22:57,794 --> 00:23:02,766
'Cause we had to do this for for like not just the United States and not just

288
00:23:05,157 --> 00:23:15,520
So I am I imagine that there there is this huge I mean there's this interesting aspect
here, which is that unlike other adversaries where the the target is sort of infinitely

289
00:23:15,520 --> 00:23:25,843
far away, whatever the malicious actor on your platform is actually trying to do, they
still have a need for their content to be understandable by their their mark in some way,

290
00:23:25,843 --> 00:23:26,323
right?

291
00:23:26,323 --> 00:23:33,775
Because so I can imagine your goal is actually to not only identify like you you don't
have to identify

292
00:23:33,775 --> 00:23:42,308
the malicious actors on the platform, if you can make it that their message doesn't result
in any any negative impact to the platform.

293
00:23:42,308 --> 00:23:50,861
I mean, I feel like the the c having two cucumbers of different sizes, you know, may be
just a legitimate meme that could be there as a real picture that someone could find

294
00:23:50,861 --> 00:23:57,739
humorous and not actually in the land of uh problematic content that could lead someone
astray or convert them to whatever

295
00:23:57,739 --> 00:23:58,501
that that can be true.

296
00:23:58,501 --> 00:23:59,373
I've never thought of that.

297
00:23:59,373 --> 00:24:06,093
Maybe you can get so good that you just like totally exhaust the space of like of symbols
that that that one might be offended by.

298
00:24:06,093 --> 00:24:07,445
R right, right, exactly.

299
00:24:07,445 --> 00:24:12,707
I uh I because I mean, like at this point, like do you ban uh eggplant emoji, right?

300
00:24:12,707 --> 00:24:23,413
It it's so perversive pervasive in our in our culture, uh in in at least in the Western
world that I'm aware of, that it seems like it it doesn't do you any good to have concerns

301
00:24:23,413 --> 00:24:25,804
over you know, the printing of that.

302
00:24:26,005 --> 00:24:30,937
and then there's another aspect of the underst like is it understandable by the the other
party?

303
00:24:30,937 --> 00:24:34,319
Uh and then so here's here's another aspect.

304
00:24:34,319 --> 00:24:36,590
That I've always been sort of curious about.

305
00:24:36,951 --> 00:24:49,087
And we can we can pivot pivot off of this in a moment, but real realistically, there is
this aspect of a way to deter someone from identifying what the defense mechanisms are is

306
00:24:49,087 --> 00:24:58,492
by separating out in time the moment in which you actually perform like have a retribution
action against uh a a spammer or whatever.

307
00:24:58,492 --> 00:25:03,759
Uh, because humans are pretty bad at long-term feedback loops.

308
00:25:03,759 --> 00:25:14,842
When an action is performed at T0 and 10 minutes later something happens, uh your account
gets banned, you probably can guess somewhere in the last 10 minutes, an a violation has

309
00:25:14,842 --> 00:25:15,472
occurred.

310
00:25:15,472 --> 00:25:23,324
But if you ban them after an hour or a day or a week or a month, it makes it very
difficult for them to narrow down what the problem is.

311
00:25:23,324 --> 00:25:33,871
So there's like some sort of mean time to uh like a delay or threshold that you
automatically let spammers have so that they're unable to actually understand.

312
00:25:33,871 --> 00:25:37,206
the merits of the classification system that's in play.

313
00:25:38,026 --> 00:25:38,346
Mm.

314
00:25:38,346 --> 00:25:40,537
Yeah, no, I I I I think that makes sense.

315
00:25:40,537 --> 00:25:44,098
And and there I guess there are kind of ways you can you can do it.

316
00:25:44,098 --> 00:25:48,529
They're consistent with a rule system, like um like having strikes for for example.

317
00:25:48,529 --> 00:25:57,691
Like uh like you have some number of strikes or you have some number of points and it's
unclear like how many points you get for what.

318
00:25:57,691 --> 00:26:07,055
And also like how many points you get for um like various like behavioral like strikes
because

319
00:26:07,055 --> 00:26:07,500
Right.

320
00:26:07,500 --> 00:26:13,371
I I think that's not that's not something, at least for an individual actor, that's not
something they'll necessarily like think about hard.

321
00:26:13,371 --> 00:26:16,873
They'll just think about like content, like content, content.

322
00:26:16,873 --> 00:26:26,545
Uh but instead, like if we can detect, okay, like who are they, who's the scene like
they're trying to target, is their pattern like just someone interacting with their

323
00:26:26,545 --> 00:26:33,877
friends, or is it someone like trying to find like new like like new marks to scan?

324
00:26:34,938 --> 00:26:36,108
that's uh

325
00:26:36,214 --> 00:26:41,197
I think that's a lot harder for peop for people to kind of uh reverse engineer.

326
00:26:41,197 --> 00:26:51,853
And like going back to to like like the cucumber an analogy, like totally sidesteps this
like trying to go trying to like level up in in like abstract space because there's like

327
00:26:51,853 --> 00:26:58,556
no there there's no like content you can change if if like we're not detecting the
content.

328
00:26:58,745 --> 00:26:59,526
Right.

329
00:26:59,526 --> 00:27:09,854
Well, there's another aspect here that I I just thought of, which is that if changing the
validation or classification mechanism to better identify malicious actors on on the

330
00:27:09,854 --> 00:27:13,709
platform, just causes them to change their behavior.

331
00:27:13,709 --> 00:27:19,012
It doesn't actually reduce the amount of spam or like unwanted messages that you have to
process.

332
00:27:19,012 --> 00:27:27,335
So it's almost in some way better off to let those messages be sent, not block the user,
and then just block like send them into the abyss.

333
00:27:27,335 --> 00:27:27,655
Right.

334
00:27:27,655 --> 00:27:35,619
Like it's it's fine if someone performs you know, a violation on a platform if no one is
around to witness it, right?

335
00:27:35,619 --> 00:27:40,160
And if you don't ban them because of it, they'll never find out that it was a problem.

336
00:27:40,160 --> 00:27:42,632
Yeah, that that's another I I think it's another good one.

337
00:27:42,632 --> 00:27:56,672
A good technique for from I'll call it um like the actual uh actually passing on d
judgment, which is like throttling or or or shadow banning where we okay, we think

338
00:27:56,672 --> 00:28:00,104
someone's like trying to spam a bunch on on Facebook.

339
00:28:00,104 --> 00:28:07,690
Well like just let them, but but like kind of kind of like pull back on the reach uh a a a
little bit.

340
00:28:07,690 --> 00:28:08,830
Of course like

341
00:28:09,850 --> 00:28:13,593
Um if you get it wrong that th you're kind of like torturing them.

342
00:28:13,593 --> 00:28:17,186
in so you definitely don't want to do to to do that.

343
00:28:17,186 --> 00:28:19,858
Not that I um on Facebook it's not so bad.

344
00:28:19,858 --> 00:28:24,881
Like people definitely like complain about it on on on say like YouTube.

345
00:28:24,881 --> 00:28:33,047
For I don't I don't know whether it's actually like abuse views or not, but I I I
frequently see like people on YouTube say, Oh, I didn't I don't have as many views as I

346
00:28:33,047 --> 00:28:36,820
did like uh like a year a year ago.

347
00:28:36,876 --> 00:28:38,668
Like is is YouTube punishing me?

348
00:28:38,668 --> 00:28:45,984
And again, I don't know if it's 'cause they did something bad with their content or not,
or if they just like fell off or the audience fell off or YouTube changed their algorithm

349
00:28:45,984 --> 00:28:47,635
in some arbitrary way.

350
00:28:47,635 --> 00:28:57,753
But um for you can imagine like if if they if they aren't like actually like doing bad
stuff, i it it w it wouldn't like feel bad.

351
00:28:57,753 --> 00:29:06,270
Like at at like at best, like you feel like your friends hate you now 'cause now they
don't interact with your post, whereas they they once did.

352
00:29:06,617 --> 00:29:09,832
Well and maybe maybe this is the thing where

353
00:29:09,859 --> 00:29:17,034
you're better off lying about the engagement mechan like the engagement metrics for your
for that particular user.

354
00:29:17,034 --> 00:29:19,836
Like don't tell them that they're getting less reach.

355
00:29:20,177 --> 00:29:29,564
I mean, it doesn't doesn't really matter, uh what's in but I guess you're you're sort of
have this hole to deal with, which is that if users' fundamental goal is to get a larger

356
00:29:29,564 --> 00:29:37,209
reach, you still want to maybe you want to actually provide them the tools to understand
why they're not reaching uh

357
00:29:37,220 --> 00:29:40,472
their actual personal goals or whatever their goal is on the platform.

358
00:29:40,472 --> 00:29:43,213
And so you have to deal with this duality.

359
00:29:43,213 --> 00:29:51,757
Whereas you want to help people who who are improving the platform, but you don't want to
help people who are diminishing value of it.

360
00:29:51,757 --> 00:29:57,540
And so now you have this whole issue of how do we sort of lie to the people who we don't
want there.

361
00:29:57,540 --> 00:30:06,144
So they continue to do things that are easy for us to detect and avoid and and sanitize
out and help those that that are out elsewhere.

362
00:30:06,624 --> 00:30:13,761
Another way of looking at it, and I I guess I can say this 'cause I don't work in uh I
don't work at Facebook anymore, even though I still still like the company.

363
00:30:13,761 --> 00:30:19,846
Um I I I guess a good way to a good analogy for what you're what you're saying is um like
ads.

364
00:30:20,408 --> 00:30:24,091
Ads in a really uncharitable way are just like like legal spam.

365
00:30:24,091 --> 00:30:27,075
Like spam the like Facebook like

366
00:30:27,075 --> 00:30:28,377
I don't know if I say le legal.

367
00:30:28,377 --> 00:30:29,660
It is for sure spam.

368
00:30:29,660 --> 00:30:32,284
We can maybe leave the the legal part out of it.

369
00:30:33,709 --> 00:30:36,001
Yeah.

370
00:30:36,001 --> 00:30:39,450
and um and what happens when it's like allowed spam that you pay for?

371
00:30:39,450 --> 00:30:40,500
Uh

372
00:30:42,242 --> 00:30:48,493
Well, f like Facebook like does everything they can to try and like increase your reach
for that you're you're like paying you're paying for it.

373
00:30:48,493 --> 00:30:54,182
Like they'll they'll um I don't have you ever like ran Facebook ads or or anything like
that before?

374
00:30:54,584 --> 00:30:56,005
Or like any pain?

375
00:30:56,753 --> 00:30:59,980
me personally, uh, no for a couple of reasons.

376
00:30:59,980 --> 00:31:02,204
but I don't know if that's worth getting into.

377
00:31:02,644 --> 00:31:02,994
Okay.

378
00:31:02,994 --> 00:31:15,718
Well, if if if you do, like you can kind of uh something interesting that i is once once
you like first put out an ad, there's it's kind of like running the algorithm on it.

379
00:31:15,718 --> 00:31:19,045
It's like testing it on on like a small subset of of audiences.

380
00:31:19,045 --> 00:31:22,387
It's like trying to figure out like who's it gonna reach well well.

381
00:31:22,768 --> 00:31:31,714
And then based on your parameters, like minimum spend uh or like maximum spend, like who
are you who are you trying to reach, what goal do you want.

382
00:31:31,714 --> 00:31:36,597
Like maybe you're selling a thing and you want them to hit the purchase button, or maybe
you just want them to look at the thing.

383
00:31:36,597 --> 00:31:41,340
Uh it it will like optimize that for you.

384
00:31:41,340 --> 00:31:46,943
And and so so that's like the total like flip side of of of like trying to kill spam.

385
00:31:46,943 --> 00:31:56,668
It's trying to figure out like what makes your advertisement like seen by like seen more
by more people, like interactively more, like clicking the button at the end of that.

386
00:31:56,668 --> 00:31:59,820
I didn't work on that part, but it is kind of funny how.

387
00:32:00,536 --> 00:32:05,018
They're they're like two like two sides of of of the same coin, i if you think about it.

388
00:32:05,018 --> 00:32:08,826
It's like what content do we what content do people pay for?

389
00:32:08,826 --> 00:32:09,908
Uh

390
00:32:11,608 --> 00:32:18,830
try and and and do the best we can to like be good business partners and and and boost
that as much as we can.

391
00:32:18,830 --> 00:32:29,264
And then what kind of stuff do we feel like is like a harm to the platform and is is kind
of especially like s people like selling like bad stuff.

392
00:32:29,264 --> 00:32:31,725
Um and then how do we like do the opposite?

393
00:32:31,725 --> 00:32:33,005
How do we throttle that?

394
00:32:33,005 --> 00:32:35,656
How do we even even like kick those people off?

395
00:32:36,015 --> 00:32:36,705
Yeah.

396
00:32:36,705 --> 00:32:46,978
Well, I I think the uncharitable view of this is it'd be totally solved if users could
control their their I I you what, I know this is such a ridiculous statement, uh and we

397
00:32:46,978 --> 00:32:50,559
know no platform could ever support this, but just have full control over their feed.

398
00:32:50,559 --> 00:32:57,181
Uh and then they would be able to say that they don't want certain things in there and
that would be the end of it.

399
00:32:57,181 --> 00:33:00,612
Uh like, you know, I only want to see messages from my friends.

400
00:33:00,612 --> 00:33:05,833
And if one of them starts posting pictures of cucumbers, I can probably guess that their
account was hacked.

401
00:33:05,837 --> 00:33:09,158
And I can I can stop following them and that's the end of it.

402
00:33:09,419 --> 00:33:17,713
but I I I do appreciate that in order to make money, companies want to poison your feed
with content and from that regard you have this problem.

403
00:33:17,713 --> 00:33:19,674
you also have the problem of of public groups, right?

404
00:33:19,674 --> 00:33:28,797
Uh where you allow anyone to post, and which case you th those groups by nature ha have
this problem where malicious actors can come in and and and post by default.

405
00:33:28,797 --> 00:33:29,508
Um

406
00:33:29,508 --> 00:33:34,592
interesting thing we uh we worked on was um like WhatsApp integrity.

407
00:33:34,592 --> 00:33:43,812
And I I I I talked a lot about like uh you have to look beyond the content to make uh to
to make a uh the best judgment.

408
00:33:43,812 --> 00:33:48,301
Well in WhatsApp you have to because WhatsApp content is like totally encrypted.

409
00:33:48,301 --> 00:33:53,885
Uh and so y like you can't like read the messages even if you w even if you wanted to.

410
00:33:53,885 --> 00:33:55,816
In some circumstances they can like

411
00:33:56,204 --> 00:33:59,065
Someone can else can like report a message and then and then you can see it.

412
00:33:59,065 --> 00:34:05,157
But that that's like at that point they they're they did your job well already for you and
you're just like confirming it.

413
00:34:05,238 --> 00:34:15,542
Um but for for for WhatsApp, uh you have to figure out everything based purely on like
behavioral features, like who's messaging who, like groups and and so on.

414
00:34:15,542 --> 00:34:23,885
Uh and um I guess the most obvious thing is I don't know if you have like WhatsApp, but
every so often I get added to like

415
00:34:25,868 --> 00:34:27,879
Like crypto token like pump group.

416
00:34:27,978 --> 00:34:30,120
Like like number number ninety eight.

417
00:34:30,120 --> 00:34:40,884
Like I I don't even like trade crypto, but I say add to these groups and then they say
like we're gonna b all buy this token and then Yeah, and then you're you're missing out

418
00:34:40,884 --> 00:34:47,146
you don't buy as well because uh you'll be a millionaire if you buy at the same time and
then sell when we w when we tell you to.

419
00:34:47,383 --> 00:34:48,245
Only a millionaire?

420
00:34:48,245 --> 00:34:52,001
I mean with what the US dollar is at, I feel like that's that's shooting pretty low there.

421
00:34:52,001 --> 00:34:56,370
uh you know, anything anything less than than ten or a hundred million.

422
00:34:56,370 --> 00:34:58,533
I I mean what what what's even what's even the point?

423
00:34:58,533 --> 00:35:00,615
Uh

424
00:35:00,615 --> 00:35:05,421
I I mean for for for the for for like the shit coins that they try and pump.

425
00:35:05,421 --> 00:35:09,516
I I don't even think they get to a cap of of like over a hundred million.

426
00:35:09,583 --> 00:35:10,494
Yeah.

427
00:35:10,758 --> 00:35:11,702
Yeah, for sure.

428
00:35:11,702 --> 00:35:12,965
I mean and

429
00:35:15,729 --> 00:35:23,771
Ugh, I mean that's pretty interesting about the the behavioral the behavioral metadata
that's associated with the messages is actually what you're performing the analysis on.

430
00:35:24,331 --> 00:35:37,475
so uh one of the things I wanted to do, especially because of your your uh pr professional
pivot now to being the CTO at a TextQL, I I'm sort of interested in I I just if I had to

431
00:35:37,475 --> 00:35:45,609
guess, you've ported some of your learnings and knowledge from working at meta
classification over to providing a similar solution to

432
00:35:45,741 --> 00:35:47,946
um other companies that are doing something.

433
00:35:47,946 --> 00:35:53,387
And like I'm sort of curious, like what is the the depth of the technology at TechSQL
doing?

434
00:35:53,664 --> 00:36:07,535
Um so uh if if you haven't if you haven't heard of of it or or or looked into it, PACGL
like kind of simply is is um like uh it's kind of like chat GBT but for querying like data

435
00:36:07,535 --> 00:36:15,392
warehouses and databases and big data systems, like doing any of like data engineering,
data analysis, data science.

436
00:36:15,392 --> 00:36:23,858
And um even if you're like non technical and specifically like even able to like work on
on like messy data and and stuff like that.

437
00:36:23,886 --> 00:36:32,354
So can think of it as and I can go into like why is this a why is this something you want
you even because e even over just like Claude by itself.

438
00:36:32,354 --> 00:36:44,224
Um but you I guess you can hopefully you can imagine like the problem or like why someone
who doesn't like know SQL or programming might want this capability themselves.

439
00:36:44,224 --> 00:36:51,510
Um like kind of in addition to or in substitute of like a full-time like data.

440
00:36:52,032 --> 00:36:56,245
engineering data a analysis team of o of like humans.

441
00:36:56,305 --> 00:36:58,336
Well let's talk about the technical side then potentially.

442
00:36:58,336 --> 00:37:05,393
Maybe there's there's some value in it, especially because I think the audience here is
much more technical, so it could give them some inspiration into what they're dealing

443
00:37:05,393 --> 00:37:05,644
with.

444
00:37:05,644 --> 00:37:17,004
I think historically a lot of companies had some sort of business analytics and that was
uh replicated data MS SQL databases if you were lucky or Oracle databases if you were

445
00:37:17,004 --> 00:37:17,925
unlucky.

446
00:37:18,125 --> 00:37:27,019
And then there were teams who basically wrote SQL on top of that or managed the SQL, the
DB admins, and then teams who didn't understand the SQL and used like SQL crystal reports

447
00:37:27,019 --> 00:37:28,990
or whatever to expose the data.

448
00:37:28,990 --> 00:37:33,802
And I think the natural evolution is, of course, to throw uh an LLM-based query at it.

449
00:37:33,802 --> 00:37:37,354
And it the LM will generate you some SQL to run against your database.

450
00:37:37,354 --> 00:37:47,539
And I think what we realized collectively in the last six years or so is that it's
incredibly expensive to figure out how to do the semantic search on your database with all

451
00:37:47,539 --> 00:37:48,042
the columns.

452
00:37:48,042 --> 00:37:48,372
whatnot.

453
00:37:48,372 --> 00:37:57,536
And it's also super expensive to do to put that data in a rag database or in a vector
database in order to uh perform the embeddings on all of that data that you have for your

454
00:37:57,536 --> 00:38:04,239
entire company, as well as perform the embeddings on every single request or query that's
coming in and match it to what's in the database.

455
00:38:04,459 --> 00:38:14,184
And now I feel like where the world is at, and maybe you can correct me, my my limited
understanding is that especi the most of the advan more advanced companies have switched

456
00:38:14,184 --> 00:38:15,714
to some sort of semantic layer.

457
00:38:15,714 --> 00:38:17,385
Whereas instead of

458
00:38:17,829 --> 00:38:30,065
Performing embeddings on the original query and what goes into the database and using
embeddings to do the similarity match, you just parse the schema and run that through your

459
00:38:30,065 --> 00:38:33,648
embeddings engine basically and then match incoming LLM

460
00:38:33,837 --> 00:38:38,129
requests or prompts that users have about the data to specific queries.

461
00:38:38,129 --> 00:38:40,770
Like you're you're not dynamically matching in the database.

462
00:38:40,770 --> 00:38:45,462
You're dynamically matching a particular query to be able to even interact with the
database in the first place.

463
00:38:45,462 --> 00:38:49,374
It's a and so you don't have to really, you don't have to parse your data.

464
00:38:49,374 --> 00:38:51,655
You don't have to run it through an embeddings model.

465
00:38:51,655 --> 00:38:59,369
You don't have to run it through an embeddings model every single time you change your
embeddings model, which I think historically has been a huge challenge.

466
00:38:59,389 --> 00:39:04,191
And I don't I don't know how much of this you're actually doing, but I I find the space
now to be much more

467
00:39:04,911 --> 00:39:05,352
Yeah.

468
00:39:05,352 --> 00:39:14,166
Um no, I I I think your evaluation of of like where did the industry start out and and
where um where has it gone is is is totally spot on.

469
00:39:14,166 --> 00:39:28,812
Kind of the current the the current like conventional wisdom is to kind of decompose your
schema into a bunch of like metrics and dimensions and then kind of construct these rails

470
00:39:28,812 --> 00:39:34,324
such that like you can search over them and then any combination of like metrics and
dimensions you have.

471
00:39:34,714 --> 00:39:39,998
is kind of like correct by construction, uses like the right formula and stuff like that.

472
00:39:40,419 --> 00:39:51,888
Um it's interesting you go there because uh kind of like our um like our approach to Text
Ul is trying to um get to the next stage of of where that is.

473
00:39:51,888 --> 00:40:01,386
And why we need a next stage, because the the the main problem like as I see it with
semantic layers is time to value.

474
00:40:01,386 --> 00:40:04,320
I mean kind of the same with with with with like rag

475
00:40:04,320 --> 00:40:06,771
and and like vector databases to a degree.

476
00:40:07,051 --> 00:40:20,644
In order to have it's once you have like this correct by construction layer, then you get
this like next level of correctness uh and assurance that okay the language model isn't

477
00:40:20,644 --> 00:40:23,055
gonna go totally off the rails.

478
00:40:23,055 --> 00:40:28,137
Um the kind of downside to that is you have to set it up first.

479
00:40:28,137 --> 00:40:32,948
And that can take a long, a long time and a lot of validation.

480
00:40:33,260 --> 00:40:41,187
And also like the kind of the what happens if someone like mentions a metric that's like a
slight variation on one that's in the semantic layer.

481
00:40:41,187 --> 00:40:45,902
Well now it can't really it's kind of broken as it can't it it's like stuck on the rails.

482
00:40:45,902 --> 00:40:46,449
Um

483
00:40:46,449 --> 00:40:48,829
let's I I want I want to really dive into that.

484
00:40:48,829 --> 00:40:59,244
Um maybe first get your perspective on how you would define semantic layer because I know
my my definition is totally convoluted and not accurate for someone that is not an expert

485
00:40:59,244 --> 00:41:07,737
in in the area, and then also understand more about the complexities that you currently
see in trying to have the semantic layer work correctly.

486
00:41:07,956 --> 00:41:16,456
Um so I I I guess like so I mean um I guess I'm neglected to go into into this, but like
my background is in programming languages.

487
00:41:16,456 --> 00:41:22,182
And so my my definition of a semantic layer would be kind of a uh

488
00:41:23,744 --> 00:41:29,310
A configuration of of like of I'll I'll call it like

489
00:41:30,964 --> 00:41:35,735
um analytical definitions on on top of a database.

490
00:41:35,735 --> 00:41:39,456
So that's one part, like your config layer of definitions.

491
00:41:39,456 --> 00:41:51,900
And then the second part is kind of a do a domain-specific language, like a very simple
programming language language on top of that config, such that like if you can express a

492
00:41:51,900 --> 00:41:59,782
query in that, it is like ostensibly correct if you provide that you like did the
configuration right.

493
00:41:59,782 --> 00:42:00,602
If if that makes

494
00:42:00,602 --> 00:42:07,245
like m it's almost like you're saying that SQL isn't the best language for querying a a a
pool of data.

495
00:42:07,306 --> 00:42:11,649
Um well it I I I guess like it is and it it is and isn't.

496
00:42:11,649 --> 00:42:17,032
Um so I I'm I I I've been like uh a semantic layer isn't necessarily good.

497
00:42:17,032 --> 00:42:21,864
Um but but it is good in the sense that like

498
00:42:23,922 --> 00:42:35,008
uh it is good in the sense that like if you have a query and you can write it in the
semantic layers DSL, it'll probably be right, or at least like you can be assured of of

499
00:42:35,008 --> 00:42:39,530
like its structural correctness to a degree that you can't with like raw SQL.

500
00:42:39,551 --> 00:42:48,736
The problem is that you uh have to define everything you want to refer to in this like
configuration layer.

501
00:42:48,736 --> 00:42:52,778
And that takes and and that means you have to write the configuration layer.

502
00:42:52,972 --> 00:42:54,863
And then that takes a lot of time.

503
00:42:54,984 --> 00:43:05,713
The good thing about SQL is that um it's it's like practically like a Turing complete
language on on top of like your tables and columns.

504
00:43:05,773 --> 00:43:13,700
That means you can express anything, which means you don't have the guardrails, but also
like what are LLMs good at?

505
00:43:13,700 --> 00:43:20,600
They're good at like looking, they're they're good at like trying many hypotheses, they're
good at like at

506
00:43:20,600 --> 00:43:25,272
creativity and flexibility and coming up with things you haven't thought about.

507
00:43:26,353 --> 00:43:37,538
And so that's like that's kind of uh one of the other like by having this like rigid
configuration layer, you're kind of like killing some of the juice that makes people

508
00:43:37,538 --> 00:43:44,681
really like using language models, w which is the ability to take on like any task in a
super flexible manner.

509
00:43:44,681 --> 00:43:46,782
Now you're kind of straight jacketing it.

510
00:43:46,972 --> 00:43:47,993
I like that flavoring.

511
00:43:47,993 --> 00:44:00,244
I mean it sounds like right, if we rely on I so in in a previous episode where we talked
about embeddings a little bit, uh one of the things that w I sort of identified was that

512
00:44:00,244 --> 00:44:09,829
when you do a similarity match in a vector database, so you you get in all your data, you
run it through an embeddings model, you get out a bunch of binary numbers.

513
00:44:09,829 --> 00:44:18,875
I like to think of this as the transformation between like the coordinate system, like X
and Y, for packets versus the Fourier transform in the frequency domain.

514
00:44:18,875 --> 00:44:20,556
So you're changing domains basically.

515
00:44:20,556 --> 00:44:28,650
And then you're doing the similarity match by running the same embedding model on the new
query that's coming in to figure out whether or not there's data in your database that

516
00:44:28,650 --> 00:44:29,401
matches.

517
00:44:29,401 --> 00:44:36,825
Now, with that, the thing that was sort of identified is that the values in similarity is
uh

518
00:44:36,879 --> 00:44:47,758
An inherent property of the embedding model that you're utilizing, that how close two
words are or their hypothetical meaning is going to be based off of whatever, as you said,

519
00:44:47,758 --> 00:44:51,621
special creativeness is in the in the model itself.

520
00:44:51,642 --> 00:45:02,281
And so while it's expensive and it may be problematic to manage, especially at scale,
switching off of that and moving the flexibility to the SQL layer means you're losing

521
00:45:02,281 --> 00:45:04,965
whatever core aspect, whatever.

522
00:45:04,965 --> 00:45:12,647
the I hate to say value or soul was in the original embedding model that allowed it to
even perform the similarity in the first place.

523
00:45:12,887 --> 00:45:14,627
So I I get that.

524
00:45:14,918 --> 00:45:26,571
there's also this complexity where it sounds like we were saying that with a semantic
layer where we're dynamically generating SQL from im incoming prompts, that the accuracy

525
00:45:26,571 --> 00:45:29,031
of the SQL generation is still potentially problematic.

526
00:45:29,031 --> 00:45:34,991
I mean you can of course uh try to improve it using guardrails, et cetera, but at the end
of the day you could still end up

527
00:45:34,991 --> 00:45:39,100
with a syntactically invalid SQL that you're running against the database.

528
00:45:39,100 --> 00:45:40,565
I I think that's what you're saying.

529
00:45:40,684 --> 00:45:44,576
Um so I I'm I I guess I'm I'm I'm kind of describing like two worlds.

530
00:45:44,576 --> 00:45:51,101
Like one world is is um like LM like writes any SQL it wants.

531
00:45:51,121 --> 00:45:55,464
It's the exact same as like giving it your Snowflake credentials in in like cloud code.

532
00:45:55,464 --> 00:46:07,222
And then the second world is like make it like pick the the the things in the can in the
semantic layer configuration that it wants, submit that to the semantic layer program, and

533
00:46:07,222 --> 00:46:08,873
then get back correct SQL.

534
00:46:08,873 --> 00:46:10,284
Um now that's

535
00:46:10,562 --> 00:46:17,066
Now that SQL will be like structurally at least like structurally correct, provide you did
the configuration right.

536
00:46:17,547 --> 00:46:21,729
the problem is yeah, sure.

537
00:46:21,729 --> 00:46:28,053
You could like refer to stuff that like doesn't exist or like totally misinterpret like
the user's question.

538
00:46:28,282 --> 00:46:31,105
Right, but I mean that accuracy is a different problem for sure.

539
00:46:31,947 --> 00:46:32,187
Right.

540
00:46:32,187 --> 00:46:46,544
And and and I guess like I'm contrasting these two worlds is like uh the semantic layer
world, like you get structural rigidity, uh which is bad for for kind of language model

541
00:46:46,544 --> 00:46:51,286
creativity, um, but good for kind of predictability.

542
00:46:51,666 --> 00:46:55,968
Exact same as like moving from embeddings to like deterministic text search.

543
00:46:55,968 --> 00:47:00,140
Embeddings is like fluid, but you have no idea like how it'll act.

544
00:47:00,330 --> 00:47:03,281
and and it's super sensitive to like small changes in your pipeline.

545
00:47:03,281 --> 00:47:16,154
Whereas like text search is like it it some text search can be complicated, but at the end
of the day, you feel a lot more confident and you can kind of like pick out if two things

546
00:47:16,154 --> 00:47:23,937
are close together, where exactly in the pipeline of of like your algorithm did they come
to be close together?

547
00:47:23,937 --> 00:47:29,718
Whereas like semantic or or not embedding search, you're kind of just like throwing up
your hands and saying,

548
00:47:30,156 --> 00:47:39,873
I think open the eye or whoever, like change a good model and uh because it's a good
model, these two things are are close and so they're they're that's that's probably like

549
00:47:39,873 --> 00:47:41,846
actually correct that they should be close.

550
00:47:44,294 --> 00:47:44,924
No, I I get it.

551
00:47:44,924 --> 00:47:50,367
So I mean it does seem like uh where we're going is that even so there's a problem with
the current state, right?

552
00:47:50,367 --> 00:47:54,058
That we've we've removed some of the you're calling creativity.

553
00:47:54,058 --> 00:48:03,863
I don't know if I love that term, but there is something here that may have captured the
in intuitiveness that whoever labeled the original data, whoever refined the data sets

554
00:48:03,863 --> 00:48:09,437
that were used to train the LLM in the first place, uh, knew subconsciously.

555
00:48:09,437 --> 00:48:11,729
And included that in the creation of the model.

556
00:48:11,729 --> 00:48:18,102
But as we shift layers up, we we lose some of that understanding fundamentally of how
these two things are related.

557
00:48:18,283 --> 00:48:21,985
And like we were talking about the context of the eggplant versus the cucumber, right?

558
00:48:22,966 --> 00:48:24,807
and since some contexts, those are very similar.

559
00:48:24,807 --> 00:48:26,688
And other ones, they're fundamentally different.

560
00:48:26,688 --> 00:48:27,883
Uh

561
00:48:27,883 --> 00:48:29,924
And I I think this keeps gone going up though.

562
00:48:29,924 --> 00:48:33,075
I mean, you mentioned the sort of accuracy of the generated SQL.

563
00:48:33,075 --> 00:48:35,785
it sounds like an inevitable direction we're going.

564
00:48:35,785 --> 00:48:40,296
And I I hate that you said that SQL is a Turing complete language, and so that's good.

565
00:48:40,797 --> 00:48:52,480
but there there is this aspect where we can say that a a DSL that is internally consistent
and guaranteed to be objectively correct in what we're generating is still better if it's

566
00:48:52,480 --> 00:48:54,581
a dynamic DSL DSL.

567
00:48:54,761 --> 00:48:55,601
And

568
00:48:55,671 --> 00:49:00,946
using the appropriate language and then converting that to something that the database
understands in order to do the query in the first place.

569
00:49:01,187 --> 00:49:08,854
Is that the eventuality of where we're going with technology in in the space to be able to
search databases effectively at at scale?

570
00:49:08,854 --> 00:49:12,050
Or is there something on top of that that you're already envisioning?

571
00:49:12,050 --> 00:49:17,770
Um so I I think if um so if that existed, like of course that'd be way better.

572
00:49:17,770 --> 00:49:26,034
I I guess like the problem I'm trying to I'm trying to highlight is um what does it take
to actually bring this layer into existence?

573
00:49:26,034 --> 00:49:38,217
It kind of requires defining like what structurally correct means for like your database
or like your business or your organization or it and s and stuff like that.

574
00:49:38,578 --> 00:49:40,438
That's like not really um

575
00:49:40,738 --> 00:49:42,529
That's not really like a programming problem.

576
00:49:42,529 --> 00:49:55,288
That's kind of um I'll I'll call it like a some combination of like a social problem and
like a UX problem and uh and a project management problem.

577
00:49:55,529 --> 00:50:05,506
And and that's kind of like where I feel like uh all of the all of the minds uh you might
accidentally step on in like s in the semantic layer world might lay.

578
00:50:05,506 --> 00:50:09,438
Uh and and so that I I think like

579
00:50:09,826 --> 00:50:17,530
Kind of like when we're selling TaxUL, like the thing we hammer on is like like time to
value, like time to value, time to value.

580
00:50:17,530 --> 00:50:21,652
Like how fast can you like get in someone's hands and happily using it?

581
00:50:21,893 --> 00:50:31,642
Semantic layers, uh if you use them naively, unfortunately, are are kind of like
counterproductive of this because they they they kind of make you define everything before

582
00:50:31,642 --> 00:50:32,426
you use it.

583
00:50:32,426 --> 00:50:35,430
And and that's also like kind of why I brought up creativity.

584
00:50:35,430 --> 00:50:39,242
Like what if you what if you want the language model to think about something.

585
00:50:39,330 --> 00:50:46,552
you haven't like exactly defined already, well, then you can't if if you're only using the
semantic layer, like no, you can't do it.

586
00:50:46,612 --> 00:50:51,853
Um hence what we try and do is kind of like like semantic modeling is great.

587
00:50:51,874 --> 00:50:53,714
Correct by construction is great.

588
00:50:53,714 --> 00:50:56,055
Let's try and get there like incrementally.

589
00:50:56,055 --> 00:50:58,995
Like write some some like SQL.

590
00:50:59,616 --> 00:51:01,836
It'll uh it'll be wrong sometimes.

591
00:51:01,856 --> 00:51:08,078
It'll be right like more than often than you think though, because language models are
pretty smart nowadays, especially if you can if you like connect them

592
00:51:08,078 --> 00:51:12,740
to all the definitions and business documents they actually need to do their work.

593
00:51:12,740 --> 00:51:24,854
Um and then like over time, like incrementally build that up in the same way that like if
you're like vibe coding, you're incrementally building up like an application from maybe

594
00:51:24,854 --> 00:51:29,576
start with some front end stuff, then you add a database, then you add some endpoints, all
all of that.

595
00:51:31,835 --> 00:51:42,845
So what so from this regard, you're obviously trying to approach the innovation in this
space in how we're doing searching in large scale databases.

596
00:51:42,845 --> 00:51:45,240
Um some companies call them data lakes.

597
00:51:45,240 --> 00:51:52,848
I guess most of them are actually data data swamps where nothing of value is actually
stored in them whatsoever.

598
00:51:55,984 --> 00:52:11,493
Where is the challenge today to actually do the query design or semantic search or
embedding based search or the correct by construction DSL generation to be able to do the

599
00:52:11,493 --> 00:52:11,744
query?

600
00:52:11,744 --> 00:52:14,880
Like where is the biggest challenge in even being able to do that?

601
00:52:14,880 --> 00:52:25,880
Um I think the um I I think right now the the the natural way of thinking is okay, what's
like the best system?

602
00:52:25,880 --> 00:52:29,203
Like what's the best retrieval system for like data assets?

603
00:52:29,203 --> 00:52:30,804
What's the best DSL?

604
00:52:30,804 --> 00:52:40,433
Like what's the best um like how do we define like a correct like semantic layer with all
of our domains mapped out?

605
00:52:40,433 --> 00:52:42,494
Um kind of like

606
00:52:43,394 --> 00:52:59,448
the the the direction I want to move the the market here is like your agentic analyst or
really any AI system sorry excuse me um your agentic analyst any or really like any AI

607
00:52:59,448 --> 00:53:10,691
system is actually like a dynamic one that like starts out with nothing and then
eventually like maybe like us for now eventually has like all the components you want from

608
00:53:10,691 --> 00:53:13,342
from it and kind of like

609
00:53:13,504 --> 00:53:29,203
A lot of are you a is your organization or are you a successful user of AI is not really
like how well designed is your end state, but how like how good is your system at like

610
00:53:29,203 --> 00:53:32,024
every intermediate state between now and the end state?

611
00:53:32,024 --> 00:53:42,020
You kind of brought it up earlier with like what's the um embeddings are great, like
semantic search and rag are are are great, but there's this like large cost in setting

612
00:53:42,020 --> 00:53:42,770
them up.

613
00:53:42,892 --> 00:53:54,031
And so if you like go about that wrong, you're gonna like take a a ton of time to get to
this like state where finally everything is like raggable.

614
00:53:54,112 --> 00:54:01,729
And like pray that you didn't do it wrong because if you did, now you have to like go back
to the start and then do it all do it all over again.

615
00:54:01,729 --> 00:54:10,526
Um kind of like I w uh the the thing I I I would like people to think about more is like

616
00:54:10,700 --> 00:54:18,841
All the states in in between like zero and and and one hundred and making sure you have a
good product like at every every point in in in that

617
00:54:18,841 --> 00:54:19,261
with that.

618
00:54:19,261 --> 00:54:24,474
ah Getting people to actually evaluate or they have a good product.

619
00:54:24,474 --> 00:54:26,064
That's a that's for sure a challenge.

620
00:54:26,064 --> 00:54:26,925
I I'm sort of curious.

621
00:54:26,925 --> 00:54:34,507
Like so while you're building TextQL, are you taking insight and uh innovation from your
from your days at Meta?

622
00:54:34,507 --> 00:54:43,872
Is there something specific that you are working on internally that you like right now
that seems like it's the it's the showstopper or the next biggest innovation that you're

623
00:54:43,872 --> 00:54:46,593
currently working on to achieve?

624
00:54:46,816 --> 00:54:59,860
Um so the I I I guess like the the biggest like new thing we've done is I'll I'll call it
like our take on the semantic layer, which we call or we call it an ontology, but our

625
00:54:59,860 --> 00:55:05,331
biggest thing is like it's the fastest time to value and time to build semantic layer.

626
00:55:05,331 --> 00:55:15,784
And so if you take my analogy of like going from zero to one hundred, um we make it so
that uh let's say you have like no documentation whatsoever in your data.

627
00:55:16,016 --> 00:55:16,546
Yeah.

628
00:55:16,546 --> 00:55:17,426
it's like pretty decent.

629
00:55:17,426 --> 00:55:21,367
It'll help you like build out the semantic layer on its own using AI.

630
00:55:21,507 --> 00:55:30,808
Let's say you're like in the middle, it's still okay because maybe it it like tries to
answer one question and it can do it with the semantic layer and then tries to answer an

631
00:55:30,808 --> 00:55:32,249
another one and it can't.

632
00:55:32,249 --> 00:55:41,883
That's okay because we can like break down break out into the unmodeled part of the
database and then write like a raw SQL query and then notify you that it was a raw SQL

633
00:55:41,883 --> 00:55:42,553
query.

634
00:55:42,553 --> 00:55:46,088
Um as to like what did I use from meta?

635
00:55:46,088 --> 00:55:59,125
Um kind of in interestingly enough, like a big part of uh a a big part of like doing
classification work is um user experience.

636
00:55:59,185 --> 00:56:11,712
The users here being defined as like all of the ML scientists and like policy experts and
like single region experts who are trying to define these policies without necessarily

637
00:56:11,712 --> 00:56:14,714
being like expert programmers

638
00:56:14,732 --> 00:56:24,006
Or maybe they are an expert like ML programmer, but they're not like an expert in terms of
like orchestrating something to be run at like the millions of requests per second, like

639
00:56:24,006 --> 00:56:25,086
QPS.

640
00:56:25,402 --> 00:56:32,289
You can think of like if you combine scale plus audience, you run into like a
representation problem.

641
00:56:32,289 --> 00:56:42,473
Like how do you represent or how do you provide like the best represent representational
interface for a non expert to get as close to expert results as possible?

642
00:56:42,613 --> 00:56:44,340
At meta, the the the

643
00:56:44,340 --> 00:56:49,873
expert results were like perfect classification of spam and not spam.

644
00:56:50,093 --> 00:57:03,191
I I I guess at a text UL, the re the the perfect representation is trying to get to as
close to expert level like data science, data engineering, with as little actual like

645
00:57:03,191 --> 00:57:07,003
knowledge of of the inner workings of those as as as possible.

646
00:57:07,003 --> 00:57:09,244
So you can think of them as actually like pretty similar.

647
00:57:09,244 --> 00:57:12,626
And again like my my background i is

648
00:57:13,356 --> 00:57:17,269
Like I'm I've been super into like programming languages and functional programming.

649
00:57:17,290 --> 00:57:25,038
Kind of like even if that's not like even if you're not designing a programming language
for these things, um that's the toolkit.

650
00:57:25,038 --> 00:57:29,782
The the toolkit is to think about like how you represent information the the best.

651
00:57:31,134 --> 00:57:37,170
and then how you do that given your audience isn't the same between like customer to
customer.

652
00:57:39,384 --> 00:57:52,637
you s you spoiled this a a bit before during the um pre uh before we went live, uh you
were spoiling a bit for me how you were secretly at heart a Haskell engineer, and that you

653
00:57:52,637 --> 00:57:54,238
absolutely love Haskell.

654
00:57:54,598 --> 00:58:03,320
and I I feel like you're not the first guest on the show for um in in the recent time that
has suggested how great functional based programming is.

655
00:58:03,320 --> 00:58:07,833
And I'm wondering if uh LLMs have inspired a new level of uh

656
00:58:07,833 --> 00:58:18,257
lack of control over the world and needing to be more refined in how we're actually
communicating with the systems that we're building or passing on our expectations.

657
00:58:18,257 --> 00:58:26,776
And if this is just our cry for help in in the night to be like, you know what, I'm gonna
use a functional based programming language because it's gonna make me feel more secure in

658
00:58:26,776 --> 00:58:27,933
what what I'm building.

659
00:58:27,933 --> 00:58:29,661
Uh any thoughts there?

660
00:58:31,643 --> 00:58:35,196
I'm well you you said I'm secretly a Haskell pro program at heart.

661
00:58:35,196 --> 00:58:37,242
Like I don't it's not really that secret.

662
00:58:37,242 --> 00:58:42,412
Like you can look at my my GitHub and see that like almost all the repos are in are are in
Haskell.

663
00:58:42,412 --> 00:58:55,372
But um I mean I I think you're um I I I I I think you are getting at at at at something,
uh and and that's kind of um

664
00:58:57,752 --> 00:59:09,132
The the amount of of of just like output, whether that be like text or code, um that an
LLM can do compared to even a team of humans, is like is is absurd.

665
00:59:09,132 --> 00:59:10,973
Like maybe ten to a hundred X.

666
00:59:10,973 --> 00:59:14,776
And that's if you're like being responsible uh about usage uh of it.

667
00:59:14,776 --> 00:59:26,534
Um and like I I I think we've all kind of thrown up our hands and said, like, okay, mi
like technically we have code review and and I do try and like look at everything.

668
00:59:26,534 --> 00:59:27,968
technically we have that.

669
00:59:27,968 --> 00:59:29,179
as as as as possible.

670
00:59:29,179 --> 00:59:37,882
But at like the end of the day, like people's attention like does slip from from from like
time to time.

671
00:59:37,943 --> 00:59:45,856
And and like the more volume, the higher pressure there there is for your attention to
slip when you're like checking an L L M's code or trying to figure out like how something

672
00:59:45,856 --> 00:59:55,190
was was was done or or like whether you just like wrote a huge L L PR and and if you wanna
like look really look at every line before submitting it.

673
00:59:57,068 --> 01:00:09,807
The thing I I guess like the good thing about like typed functional programming and other
like very structural ways of of like outputting code or or or text is that they're like

674
01:00:09,807 --> 01:00:20,373
principled like uh and and highly rigorous and and so having that structure grants a level
of of assurance.

675
01:00:20,634 --> 01:00:27,010
Maybe we overstate that assurance because uh like maybe the principles

676
01:00:27,010 --> 01:00:30,912
that like the LLM built the Haskell on are also like suspect.

677
01:00:30,912 --> 01:00:48,249
But um at at the end of the day, it is um it is kind of I I I think reassuring to to see
that okay I submitted this like 2000 line PR and according to this like according to the

678
01:00:48,249 --> 01:00:56,252
compiler that that um unless and compilers have bugs sometimes, but much less than like
than like LLM code.

679
01:00:56,438 --> 01:01:08,953
according to this compiler, everything between like the assumptions declared in like the
interface and the as and the assumptions declared in the code are the exact same.

680
01:01:08,953 --> 01:01:15,146
And and I I think that um I think that makes people feel a lot safer.

681
01:01:17,709 --> 01:01:19,860
you're definitely getting at a uh sore point for sure.

682
01:01:19,860 --> 01:01:28,945
I think early on, maybe a couple of years ago, there was uh sort of a controversy where
some of the core Linux modules were converted from C to Rust.

683
01:01:28,945 --> 01:01:38,510
And I believe uh the goal was to avoid uh all the sort of problems that Rust solves,
memory related, etc., as far as vulnerabilities go.

684
01:01:38,510 --> 01:01:41,672
And we can be sure that none of those exist in the compiled modules.

685
01:01:41,672 --> 01:01:46,125
But the problem was that new issues showed up through the transformation process.

686
01:01:46,125 --> 01:01:47,226
And so I think

687
01:01:47,226 --> 01:01:54,172
know you're definitely on to something there to say that we are avoiding certain concerns
and maybe it doesn't actually solve all of them.

688
01:01:54,272 --> 01:02:04,411
But on the flip side, I I I think that I am also a little bit more optimistic where we're
able to define the the semantics or the invariance with our our code or our business logic

689
01:02:04,411 --> 01:02:08,665
in such a way that actually does give us additional guarantees there.

690
01:02:08,841 --> 01:02:11,204
So I I can appreciate uh this migration.

691
01:02:11,204 --> 01:02:20,814
I know uh my own preference is Rust when I can, um, because I feel like using Rust does
avoid some of the complexities uh that show up in larger systems.

692
01:02:20,814 --> 01:02:23,706
I also think that yeah.

693
01:02:23,706 --> 01:02:30,634
like poison people's brains, but the the the kind of feeling of of like if it compiles it
works is is definitely there.

694
01:02:30,634 --> 01:02:32,181
Of of course it's not always true.

695
01:02:32,181 --> 01:02:41,413
Like you can you can add two when you when you meant to add one, but it it uh there's
definitely like a a high level of reassurance there.

696
01:02:41,413 --> 01:02:42,733
Yeah, I definitely I definitely agree.

697
01:02:42,733 --> 01:02:52,887
Uh actually this came up in the episode that we were recording earlier earlier with uh
Cassidy Williams on basically that there's correctness in what we're what we're generating

698
01:02:52,887 --> 01:02:56,008
and a a commit to getting the correct answer.

699
01:02:56,008 --> 01:03:07,147
But more importantly, is that uh there are vulnerabilities, just safety checks or security
issues in what we're building today when we're not using one of these

700
01:03:07,523 --> 01:03:09,805
languages that just compiles.

701
01:03:09,805 --> 01:03:13,488
Uh and if it compiles, it it's correct and it it works to some degree.

702
01:03:13,488 --> 01:03:18,361
And such that we'll migrate to them because there's clear wins in doing that.

703
01:03:18,402 --> 01:03:22,165
And from there, there are still innovations that we can have on top of that.

704
01:03:22,165 --> 01:03:27,433
I think one of them is known as formal verification, which we just can't do with

705
01:03:27,611 --> 01:03:30,222
like a scripting language or even a a weekly type language.

706
01:03:30,222 --> 01:03:41,735
And this is the aspect of ensuring that what we built or the code that's running is not
only syntactically correct, but and avoids these sort of vulnerabilities that come up

707
01:03:41,735 --> 01:03:46,727
because of memory leak leakage, et cetera, but is actually doing what the business
declared.

708
01:03:46,727 --> 01:03:51,368
There's some aspect to the code which makes it objectively correct on a higher level.

709
01:03:51,589 --> 01:03:53,045
And I think we'll get there.

710
01:03:53,045 --> 01:04:07,829
you think about it, um if you say like um define like your reasonably definable invariance
like in the types and then have your program like off the types, well even with like um

711
01:04:07,829 --> 01:04:17,878
ten thousand LLM written lines of code per day, you can be rather sure that like the
invari the type invariance will be a much smaller service area than than like all of the

712
01:04:17,878 --> 01:04:18,908
business logic.

713
01:04:19,161 --> 01:04:22,893
Yeah, I I I mean I I definitely I definitely like the optimism there.

714
01:04:22,893 --> 01:04:30,086
One of the problems is that like I still think that LMs tend to generate new stuff rather
than pulling the semantics out of what have already been generated.

715
01:04:30,086 --> 01:04:39,090
Like we have the same type for a sixteen character string generated over and over again,
except this sixty this one's a sixteen character string that represents, I don't know, the

716
01:04:39,090 --> 01:04:39,741
order ID.

717
01:04:39,741 --> 01:04:43,843
And this one's a sevent fifteen to seventeen character string that represents the invoice
ID.

718
01:04:43,843 --> 01:04:48,685
When both of them, you know, still have to go in the same column in the database because
it's an auditing

719
01:04:48,961 --> 01:04:49,794
Mm-hmm.

720
01:04:50,130 --> 01:04:50,856
Yeah.

721
01:04:50,856 --> 01:04:53,418
key ID, you know, actually has to be sixteen characters.

722
01:04:53,418 --> 01:04:55,357
And when it gets it seventeen characters is gonna be a problem.

723
01:04:55,357 --> 01:04:59,045
And so well thank you LLM for generating the invariant that says the invoice ID

724
01:04:59,045 --> 01:05:03,398
to break out of it likes to break out of um like centralized abstractions.

725
01:05:03,398 --> 01:05:09,012
Have you ever heard of this like uh debate or tension between like

726
01:05:10,988 --> 01:05:14,303
Locality of behavior and don't repeat yourself.

727
01:05:16,060 --> 01:05:16,731
yeah, absolutely.

728
01:05:16,731 --> 01:05:25,209
It's one of the I I think it's one of the biggest struggles I've had in uh mentoring
engineers for my last twenty years for sure.

729
01:05:26,892 --> 01:05:27,162
Right.

730
01:05:27,162 --> 01:05:40,563
So I I think um it it's definitely like w it it's a debate that that's like come up in
engineering, like stuff like and and I I think like the case um it was like all don't

731
01:05:40,563 --> 01:05:41,834
repeat yourself for thirty years.

732
01:05:41,834 --> 01:05:45,597
And now I think there's some advocates for locality behavior.

733
01:05:45,722 --> 01:05:56,346
Um I fortunately or unfortunately, um an extremely large advocate of locality of behavior
or large language models who prefer to do everything within

734
01:05:56,546 --> 01:06:00,766
got like literally one hundred percent of the logic defined within their context window.

735
01:06:02,320 --> 01:06:09,248
I I think this is one of the things that the pendulum swings back and forth to one of the
extremes over and over again.

736
01:06:09,297 --> 01:06:18,701
uh just for context here, uh, for anyone who's not familiar with these ideas, the the
don't repeat yourself or or dry of the solid principles basically says if you're doing the

737
01:06:18,701 --> 01:06:26,105
same thing in multiple locations, you should abstract out some sort of abstraction or
function or method or class that encapsulates that functionality.

738
01:06:26,105 --> 01:06:32,228
So like a sum method if you're adding numbers over and over again, and then just call the
sum method.

739
01:06:32,228 --> 01:06:39,281
The the problem is that in practice you get extra complexities that are added into it,
like um, what are the

740
01:06:39,281 --> 01:06:40,291
Parameters that are allowed in?

741
01:06:40,291 --> 01:06:41,262
Are they just strings?

742
01:06:41,262 --> 01:06:46,316
And does it parse strings to in so are doubles or floats before it does the arithmetic?

743
01:06:46,316 --> 01:06:47,527
What about overflows?

744
01:06:47,527 --> 01:06:48,677
What about negative numbers?

745
01:06:48,677 --> 01:06:51,229
What happens with irrational or complex numbers?

746
01:06:51,229 --> 01:06:59,865
And so do you end up with a single method that's just called sum that and I'm sure some
mathematicians have a opinion here about how you define a group and the operations on the

747
01:06:59,865 --> 01:07:05,629
group uh or on the set in order to decide, you know, whether or not it is a group and
whether or not that's the appropriate function.

748
01:07:05,629 --> 01:07:08,150
And I'm sure I just lost everyone when I said that.

749
01:07:08,210 --> 01:07:09,251
so welcome to

750
01:07:09,251 --> 01:07:13,837
uh real analysis for for mathematics and how or group theory really.

751
01:07:14,359 --> 01:07:15,351
And so I think that's one thing.

752
01:07:15,351 --> 01:07:23,332
Uh the other the other thing is that realistically when when we are deciding where the

753
01:07:23,905 --> 01:07:33,510
aspects or complexity of our program should go, it really does require a little bit of a
design philosophy on whether or not it makes sense to do the subtraction and have one

754
01:07:33,510 --> 01:07:38,772
infinitely configurable method, which then loses all its value, or have something
incredibly opinionated.

755
01:07:38,772 --> 01:07:47,796
And obviously the optimal is somewhere in between, whereas the locality of the behavior of
the function defines where it's like, well, in this place, we only need to add two

756
01:07:47,796 --> 01:07:48,296
integers.

757
01:07:48,296 --> 01:07:52,418
So we will just write add integers and that we'll be done with that.

758
01:07:52,418 --> 01:07:53,922
And we won't care about the all these

759
01:07:53,922 --> 01:08:03,095
cases, but you lose some of the understanding and the expertise that has been brought by
building up your single abstraction of the sum method over time.

760
01:08:03,135 --> 01:08:11,527
And so I think what's interesting here is to be aware of the controversy or the
discrepancy and the duality of the opinions in the space, and then figure out what

761
01:08:11,527 --> 01:08:12,238
actually makes sense.

762
01:08:12,238 --> 01:08:15,909
And I feel like LLMs just this is one of the areas where they fail.

763
01:08:15,909 --> 01:08:23,061
And I feel like this stems from all the failures that LLM generated code always has, which
it doesn't fully grasp.

764
01:08:23,095 --> 01:08:26,578
Why to pick one of these solutions over the other one?

765
01:08:26,578 --> 01:08:34,815
I mean it will if you ask it when should I do this, like should I use this A or B, it will
give you a whole list of things, but it's not going to pick the right one for sure.

766
01:08:35,076 --> 01:08:36,523
And uh

767
01:08:36,523 --> 01:08:42,764
I I my pick in a previous episode was this paper that compares the Linux operating system
to E.

768
01:08:42,764 --> 01:08:55,482
coli bacteria, as far as the DNA for protein creation, where it says evolution has
concluded that the most critical functions are highly replicated throughout the DNA, which

769
01:08:55,482 --> 01:09:00,904
in a way represents similarly to what we see with LLMs with the locality of behavior for
the functions.

770
01:09:00,904 --> 01:09:06,387
Because if a function is so critical, if there's a mutation in that function or a bug.

771
01:09:06,387 --> 01:09:10,330
That's introduced or a regression, then the whole organism ceases to function.

772
01:09:10,330 --> 01:09:17,854
But if you have that same function replicated everywhere, uh only where it's necessary,
and it's a different instance of that function.

773
01:09:17,854 --> 01:09:26,940
If one of them introduces a mutation in the DNA, which creates a protein, which means that
that thing can no longer function, the rest of the creature or organism still can survive

774
01:09:26,940 --> 01:09:28,877
as long as it's not like.

775
01:09:28,877 --> 01:09:32,072
I don't know, ha handling cell division in some way.

776
01:09:32,072 --> 01:09:33,964
Or, you know, it does get a cancer and and die.

777
01:09:33,964 --> 01:09:38,030
And so like there are some critical failure modes, but for most of them, it can still
survive.

778
01:09:38,030 --> 01:09:41,355
And maybe that mutant form is actually better than what was previously.

779
01:09:41,355 --> 01:09:45,636
And I feel like the real challenge is identifying, okay.

780
01:09:45,636 --> 01:09:52,110
Are we in a scenario where we're creating an abstraction that directly duplicates that
functionality and has to be the same everywhere?

781
01:09:52,111 --> 01:09:56,234
Or are we the Linux operating system where, you know, there's only one version of that
module?

782
01:09:56,234 --> 01:10:02,148
And if there is a failure there, then that means that every single version of Linux is now
susceptible to a security vulnerability because of it.

783
01:10:02,742 --> 01:10:03,622
wow.

784
01:10:03,722 --> 01:10:12,325
Yeah, I never really thought about um like redundancy in the code as a potential defense
mechanism.

785
01:10:12,325 --> 01:10:19,808
I mean, like you might just say like, don't screw up like the the the single version of of
of uh of the function.

786
01:10:19,808 --> 01:10:27,676
But um I I guess like the need for defense i is um i is probably a hundred times more than
than than ten years ago.

787
01:10:28,966 --> 01:10:29,666
yeah, for sure.

788
01:10:29,666 --> 01:10:38,799
I I think the the wisdom that I've shared here a lot is look at the function that you're
creating and decide, is there another function that has the same core values and

789
01:10:38,799 --> 01:10:43,425
expectations for how that function should evolve over time as the new one you're creating?

790
01:10:43,439 --> 01:10:51,333
And that's not a simple thing to just answer on the fly, but I feel feel like that's
fundamentally the aspect of doing software development, software engineering is doing this

791
01:10:51,333 --> 01:10:52,193
activity.

792
01:10:52,193 --> 01:10:54,384
And so we still have to make those decisions.

793
01:10:54,384 --> 01:11:03,478
And I feel like I feel like people who spend more and more time generating more code uh
are increasingly avoiding making those decisions and understanding what's going on there

794
01:11:03,478 --> 01:11:04,719
and skipping that part of the review.

795
01:11:04,719 --> 01:11:13,785
And so we're going to end up with a lot of the locality of functionality, uh, but avoiding
the question of does this actually have to work the same?

796
01:11:13,785 --> 01:11:15,688
And those are where bugs creep in.

797
01:11:16,930 --> 01:11:18,495
Yeah, no, totally.

798
01:11:20,549 --> 01:11:23,223
I have a lot of controversial controversial theories on this.

799
01:11:23,223 --> 01:11:25,026
actually, uh maybe I'll throw one more at you.

800
01:11:25,026 --> 01:11:35,883
Uh the value uh that is the business value that is created by software you're writing is
proportional to the time the a human has spent uh creating that software.

801
01:11:38,198 --> 01:11:39,398
Hmm.

802
01:11:39,860 --> 01:11:44,526
I would say um at the um

803
01:11:46,509 --> 01:11:47,326
Well

804
01:11:49,482 --> 01:12:00,887
It it it might be true if you define it as like an upper bound because there there's
certainly like um like like random like like Java product with a hundred thousand

805
01:12:00,887 --> 01:12:07,340
engineers versus like more pleasurable to use product with like two hundred right.

806
01:12:07,340 --> 01:12:15,593
Like it's not the I mean it it might be written in Java too, but but definitely the the
the legacy one was written in Java.

807
01:12:15,593 --> 01:12:17,334
um

808
01:12:18,806 --> 01:12:24,776
Yes, I I it it's it's if you define it as like an upper bound, that's I I think that's
definitely true.

809
01:12:24,776 --> 01:12:31,967
Like keeping in mind that you you can't you can totally like waste human hours uh on
making the the the software better.

810
01:12:31,967 --> 01:12:33,692
Um but uh

811
01:12:33,692 --> 01:12:37,437
organizations waste human hours in the name of software development.

812
01:12:37,437 --> 01:12:40,640
So I I think I think for sure that that that has to be true.

813
01:12:40,768 --> 01:12:52,086
but yeah, I I'm I'm like leaning towards uh I I'm like leaning towards yes, the the th
this seems to be true because uh

814
01:12:54,198 --> 01:13:01,640
I I think like the there are claims floating around about like how this or that person is
like a hundred times more productive with like AI software engineering.

815
01:13:01,640 --> 01:13:11,803
Um I I I'm not sure I I see like a hundred times better like software products compared to
like fat compared to like ten years ago.

816
01:13:11,803 --> 01:13:21,605
Like ostensibly, um like Jira now has like seventy percent of the or or more of Jira
developers all use like AI in their in their coding.

817
01:13:21,605 --> 01:13:24,120
Um is like the

818
01:13:24,120 --> 01:13:36,196
the the Jira like profit or or like or customer satisfaction or anything like tripled what
it was com compared to to like eight years ago b before anyone was doing AI coding.

819
01:13:36,196 --> 01:13:38,877
Like I like I I really think so.

820
01:13:38,877 --> 01:13:41,658
I I it so it seems like um

821
01:13:43,520 --> 01:13:55,534
AI helps a a lot, but there is some like underlying variable that that like equally
influences quality and and does not seem to be moved and and AI like anything or automated

822
01:13:55,534 --> 01:13:57,035
anything doesn't seem to move the needle.

823
01:13:57,035 --> 01:14:07,558
The only thing that moves the needle there is is like um lots of of like thoughtful like
human like groundwork, at least with the current level of AI capability.

824
01:14:07,929 --> 01:14:08,520
Yeah, you know what?

825
01:14:08,520 --> 01:14:13,957
Honestly, I'm I'm like this close to writing a blog post called the like Warren's Laws.

826
01:14:13,957 --> 01:14:19,485
Uh sort of like Maxwell's laws for the electromagnetism and the nature of the universe.

827
01:14:19,485 --> 01:14:22,919
Uh to like this one is one of them.

828
01:14:22,919 --> 01:14:26,009
The the upper bound on the uh

829
01:14:27,427 --> 01:14:29,958
relative gain is proportional to to the human.

830
01:14:29,958 --> 01:14:39,612
I I I do think that there's something to be said about there there's clearly some value
here with using LLMs and I feel like understanding c the core of of what it is is super

831
01:14:39,612 --> 01:14:40,633
important.

832
01:14:40,654 --> 01:14:45,709
but maybe that's context for uh an uh potentially another episode.

833
01:14:47,031 --> 01:14:55,767
One of the things I do wanna sort of circle back around to, and maybe this is a a question
that's still relevant in the space, is so it it it seems to me that you know you're being

834
01:14:55,767 --> 01:15:05,834
very careful about which where you're using LLMs in the software development process and
how you're interacting with them for the benefit of the customer ingestion uh for the

835
01:15:05,834 --> 01:15:10,737
data, for how you're generating DSLs or the complete

836
01:15:13,189 --> 01:15:14,330
Sorry, what was the term used?

837
01:15:14,330 --> 01:15:21,535
Um for the generated queries that you're making that are provably correct.

838
01:15:21,535 --> 01:15:24,276
Um what yeah.

839
01:15:24,957 --> 01:15:26,258
Correct by construction.

840
01:15:26,258 --> 01:15:28,079
Uh I I I I I love that term.

841
01:15:28,079 --> 01:15:35,364
Um do you see that there is a corresponding challenge for scale?

842
01:15:35,364 --> 01:15:42,448
Like if you only had to perform a hundred requests per day on against the database versus

843
01:15:42,448 --> 01:15:47,662
I think uh somewhere uh in the classification architecture for Facebook, you were at 50
million.

844
01:15:47,662 --> 01:15:52,305
Does this change your approach for interacting with that data?

845
01:15:52,305 --> 01:16:00,070
Or do you is that sort of like a different orthogonal axis that just talks about how to
make the the solution reliable?

846
01:16:00,070 --> 01:16:08,095
Like is it, okay, well, we're doing 50 million, we need to make sure we're using different
software languages or or doing some sort of performance testing, or is there a meaningful

847
01:16:08,095 --> 01:16:10,927
difference in how we're approaching doing the

848
01:16:10,927 --> 01:16:13,250
the development or the interaction with the L LMs.

849
01:16:13,250 --> 01:16:14,150
Hmm.

850
01:16:14,690 --> 01:16:18,071
Frankly, I don't have a don't have a super good answer to to this.

851
01:16:18,071 --> 01:16:21,152
Like theoretically, um theoretically, like yes.

852
01:16:21,152 --> 01:16:34,866
Like like in like I was just writing SQL, like at at Facebook to do analytics, like and
under like underneath the hood, like if you have correct like primitives like like

853
01:16:34,866 --> 01:16:42,408
sharding for for example, like everything should like work structurally the same at uh on
on like

854
01:16:42,602 --> 01:16:46,006
one gigabyte to to like a million gigabytes.

855
01:16:46,908 --> 01:16:47,369
Right.

856
01:16:47,369 --> 01:17:00,296
Um the like yeah, th theoretically, but it's it's uh it it's kind of hard to say because
always it's always the case that like one like

857
01:17:02,240 --> 01:17:11,234
when you actually go to the million gigabyte or more scale, like implement like tiny
details do always like creep in.

858
01:17:11,234 --> 01:17:23,969
Like you have to you have to think about um you have to like think about how data is
distributed like not just the shard, like s on on like some for some function of of of

859
01:17:23,969 --> 01:17:24,459
like the shard.

860
01:17:24,459 --> 01:17:28,821
So you kind of already broke the mental model with with w w with that.

861
01:17:29,401 --> 01:17:32,362
Uh and um kind of like going to the flip side.

862
01:17:32,556 --> 01:17:38,649
Like what if you only had to deal with like ten gigabytes, which is like more often you
think.

863
01:17:38,649 --> 01:17:45,571
I that's probably like years for for TextQL where all the relevant information could fit
in in in like ten gigabytes.

864
01:17:45,571 --> 01:17:56,076
Um there's actually like a lot of power that you can do that you can get just by saying
like screw um like sc screw scale, like I don't I'm not gonna think about this.

865
01:17:56,076 --> 01:17:58,457
Let's say I'm I'm at like ten gigabytes forever.

866
01:17:58,457 --> 01:18:01,958
Now I can like use um

867
01:18:02,146 --> 01:18:04,927
what what I call like laptop tooling for for everything.

868
01:18:04,927 --> 01:18:17,531
Like m like SQLite, like DuckDB, like random like PHP server like like living on on on on
like my my laptop, like bash scripts and and and so on.

869
01:18:17,531 --> 01:18:27,173
And um the amount of composability you get from like living on on like one small machine
is is like absolutely insane.

870
01:18:27,173 --> 01:18:29,614
Like I I think you can do the same

871
01:18:30,150 --> 01:18:39,508
Again, it it's theoretically just SQL in at Facebook too, but somehow if it's just like
bash and SQLite, I can do the same thing like a thousand times faster just on my computer.

872
01:18:40,697 --> 01:18:42,088
Yeah, I I I think you're absolutely right.

873
01:18:42,088 --> 01:18:49,655
Uh if you only have ten gigabytes, you're like, well, I can just load this all into memory
and then we can do an optimized investigation of whatever the data is.

874
01:18:49,655 --> 01:18:51,447
Like querying in memory is just so much easier.

875
01:18:51,447 --> 01:18:56,191
We have all the data constructs that we want to use or whatever data structures that make
sense.

876
01:18:57,012 --> 01:18:57,643
Right, exactly.

877
01:18:57,643 --> 01:19:02,457
Uh I mean, obviously the challenge is like how do we load the data in from the database so
that it's uh like

878
01:19:02,529 --> 01:19:06,291
Exactly in the data structure which makes it easier or faster to query over and over
again.

879
01:19:06,291 --> 01:19:10,783
But it's gonna be so much faster if we just do that at the start than trying to read from
disk.

880
01:19:10,783 --> 01:19:21,719
And obviously at at scale, you have s some of these other problems where there is a real
latency between uh a particular kind of request and another one if one shard is in this

881
01:19:21,719 --> 01:19:26,766
data center and another shard is in a data center that's you know halfway across the
country or something like that.

882
01:19:26,766 --> 01:19:38,220
to or you're like trying to query in like the absolute worst way that like like takes a
tiny scoop from every single shard, which is probably like much easier than than you

883
01:19:38,220 --> 01:19:38,910
think.

884
01:19:39,203 --> 01:19:39,684
Mm.

885
01:19:39,684 --> 01:19:39,924
Right.

886
01:19:39,924 --> 01:19:47,656
I mean, you just name your indexes wrong or you don't you didn't even think about doing
that in the first place and why w you know, of course some data will end up getting

887
01:19:47,656 --> 01:19:49,879
sharded uh ineffectively.

888
01:19:49,879 --> 01:19:51,560
And ha you'll have to deal with that then.

889
01:19:51,560 --> 01:20:01,274
And I the problem is that I see often a lot of times these show up at scale but are sort
of orthogonal to utilizing LLMs or the technology that we've built up recently, vector

890
01:20:01,274 --> 01:20:02,304
databases, et cetera.

891
01:20:02,304 --> 01:20:11,438
Uh it's like almost completely separable until we get to the point where like, oh, we want
to have LLMs write the source code for us or the schema or, you know, where the indexes

892
01:20:11,438 --> 01:20:18,411
are, because then it will get that wrong, or generate the queries for us because it won't
take into account that whatever our infrastructure is.

893
01:20:18,411 --> 01:20:19,772
And so I I do see

894
01:20:19,772 --> 01:20:28,978
that there is still some sort of meaningful distinction still, but it's not as important
as just understanding that at scale there are certain things that we've already figured

895
01:20:28,978 --> 01:20:30,128
out and have to do.

896
01:20:30,722 --> 01:20:31,032
Right.

897
01:20:31,032 --> 01:20:45,060
I'm like uh I I guess like the promise of of like having the cloud and and being able to
stand up an entire app from like the Ib US SDK is that like it's kind of like the correct

898
01:20:45,060 --> 01:20:53,805
by construction version of like like DevOps or or infra where oh hey hey like uh I'm I'm
using like ECS.

899
01:20:53,805 --> 01:20:57,958
Uh it it appears to horizontally scale like right out of the box.

900
01:20:57,958 --> 01:20:59,146
Or hey, I'm using like

901
01:20:59,146 --> 01:21:01,548
or like or or or Aurora or or something.

902
01:21:01,548 --> 01:21:05,911
Hey, that also like horizontally scales, like right out of the box or or or like Dynamo D
B.

903
01:21:05,911 --> 01:21:13,696
But then like you run into all of these like it's so easy to like accidentally like make
some kind of locality assumption.

904
01:21:13,696 --> 01:21:17,078
And then like you go up one order of magnitude and everything explodes.

905
01:21:17,252 --> 01:21:18,202
Yeah.

906
01:21:18,203 --> 01:21:30,316
I I mean you say that and then I I just think back to I think it was a couple episodes ago
where we were discussing about uh RTO and RPO, uh about handling either malicious

907
01:21:30,316 --> 01:21:35,674
attackers like encrypting all your data and how do you recover from that or losing a data
center and how you recover from that.

908
01:21:35,735 --> 01:21:40,698
And the the clouds don't give me that burprining.

909
01:21:41,967 --> 01:21:45,479
I mean, it's w it's not even like you pay for it and then it happens.

910
01:21:45,479 --> 01:21:53,599
You have to actually invest in understanding what the building blocks are that the cloud
provider offers you so that you can even utilize them correctly.

911
01:21:53,599 --> 01:22:00,475
Like you mentioned, like RDS versus Aurora serverless versus DynamoDB in AWS.

912
01:22:00,475 --> 01:22:07,568
Some of one of those provides you multi region backups and active active configurations at
scale.

913
01:22:07,568 --> 01:22:08,178
By default.

914
01:22:08,178 --> 01:22:12,259
The other two make it incredibly painful to make it happen.

915
01:22:12,339 --> 01:22:14,420
And in practice, it could be problematic.

916
01:22:14,420 --> 01:22:21,362
So I feel like, yeah, I do agree that the c w going to the cloud does solve some of the
problems with scale.

917
01:22:21,442 --> 01:22:28,664
I think the original promise was more on the lines of the zero to one velocity to make
that happen.

918
01:22:28,664 --> 01:22:33,956
And over time, I think they've gotten better with adding the necessary building blocks to
go further than that.

919
01:22:33,956 --> 01:22:34,628
But

920
01:22:34,628 --> 01:22:41,322
I don't think the user the the usability or the user experience there uh helps make people
make the right decisions.

921
01:22:42,183 --> 01:22:47,315
That being said, maybe that's uh more of a tangent for one of the previous episodes that
we had.

922
01:22:47,315 --> 01:22:51,868
So maybe now's a good time for us to switch over to picks for the episode.

923
01:22:51,868 --> 01:22:55,470
So uh Mark, what did you bring for the audience today?

924
01:22:55,942 --> 01:23:04,628
Um sure, I I I think the thing we we we discussed beforehand was um like burpees or hotel
room exercise in general.

925
01:23:04,628 --> 01:23:15,356
Like I'm in the hotel room right now for all kinds of like events and like selling, like I
guess like perks or what happens when you're when you sell to other businesses, you have

926
01:23:15,356 --> 01:23:17,357
to be on the road a lot.

927
01:23:17,357 --> 01:23:24,762
Um I like to exercise, but I used to always like fall back on uh or it I was I always

928
01:23:25,740 --> 01:23:30,853
let it go to the wayside whenever I was like on the road or especially busy or or
anything.

929
01:23:30,853 --> 01:23:34,934
And uh to solve that I just started doing burpees in my ho hotel room.

930
01:23:34,934 --> 01:23:35,765
Uh that's it.

931
01:23:35,765 --> 01:23:41,877
I mean it's a it's a workout where you're just like standing up and like jumping up and
down and and and doing push ups.

932
01:23:41,938 --> 01:23:51,132
Good enough for me for for like a couple day uh couple day road trip and and it means like
when I do get do get back home, like I'm not out of the habit anymore.

933
01:23:52,582 --> 01:23:57,711
I feel like there's always sort of enough room somehow to do a burpee in the hotel room.

934
01:23:58,006 --> 01:23:58,356
Right.

935
01:23:58,356 --> 01:24:08,786
Unless you're in uh unless you're in some kind of like hostel with with that's like six
bunk beds in a two hundred square foot room, then yeah, yeah, definitely you'll have

936
01:24:08,786 --> 01:24:09,086
space.

937
01:24:09,086 --> 01:24:12,038
And even if not, like go to the sidewalk and then and then do them.

938
01:24:12,740 --> 01:24:14,380
I the hands on the sidewalk.

939
01:24:14,380 --> 01:24:17,472
I I I think I I I think would be a little bit of a challenge.

940
01:24:17,472 --> 01:24:18,913
I mean, for some people, I'm sure that's fine.

941
01:24:18,913 --> 01:24:24,975
Honestly, the hallways in the hostels always were incredibly wide and s for some reason.

942
01:24:24,975 --> 01:24:34,069
It's like the rooms incredibly small and you know, double or triple bunks or whatever, but
in the h the hallways super wide, you could absolutely uh you know, just jump up and down

943
01:24:34,069 --> 01:24:34,489
like that.

944
01:24:34,489 --> 01:24:35,790
I I love it honestly.

945
01:24:35,790 --> 01:24:41,962
B I mean when I when I travel, I feel like I definitely use that as an excuse not to
exercise.

946
01:24:43,276 --> 01:24:46,081
Right, well now you don't have one anymore.

947
01:24:46,130 --> 01:24:51,968
Uh you've you've ruined my my whole travel life.

948
01:24:51,968 --> 01:24:55,263
My whole my whole conference speaking circuit was always like, this is for me.

949
01:24:55,263 --> 01:25:03,333
I travel, I enjoy talking with people and uh discussing new topics, and I just get to
chill out and

950
01:25:03,566 --> 01:25:08,348
regress in a lot in a lot of ways and and now you're like, well, you've got no excuses
left.

951
01:25:08,348 --> 01:25:16,912
There were always so a long time ago when I tr when I was traveling, I used to try to go
if there was a swimming pool, uh I used to try to utilize it in the hotel.

952
01:25:16,912 --> 01:25:18,653
Um and if there was an exercise room.

953
01:25:18,653 --> 01:25:23,265
But there was definitely a huge barrier to have to overcome in order to do that.

954
01:25:23,265 --> 01:25:28,318
Like finding the time in the day to uh add that in was definitely problematic.

955
01:25:28,318 --> 01:25:33,460
And as you mentioned, if you're traveling a lot, like especially in a sales position, you
may be traveling three

956
01:25:33,460 --> 01:25:35,793
three months, uh three weeks out of every month.

957
01:25:35,793 --> 01:25:38,558
And then at that point you're like, well I'm I'm pretty much just not exercising ever.

958
01:25:38,558 --> 01:25:40,871
Uh that week I get back, that's for me.

959
01:25:40,871 --> 01:25:43,936
I have to then relax and recover during during that time.

960
01:25:43,936 --> 01:25:48,116
So you you pretty much completely avoid exercising completely uh in that regard.

961
01:25:48,116 --> 01:25:56,349
the thing about or like whatever exercise you can do, just like standing up in in in like
the space you are right now, is that it like totally removes the friction.

962
01:25:56,349 --> 01:26:09,712
Like and like motivation and like love for for um like doing an activity, whether that be
like exercise or even like like programming even, it it like wanes.

963
01:26:09,712 --> 01:26:10,953
Sometimes you don't feel like doing it.

964
01:26:10,953 --> 01:26:15,234
But um what's the best way of continuing to do that regardless?

965
01:26:15,234 --> 01:26:15,970
Um

966
01:26:15,970 --> 01:26:17,636
Just like taking out all the fiction.

967
01:26:18,414 --> 01:26:24,028
Is there a whole Mark Hayes exercise routine for traveling hotels or is Burpees it?

968
01:26:25,854 --> 01:26:36,717
so I I mean at at minimum the the like frictionless thing is is like just just like
burpees, do do like a hundred or or something or or like split it up uh as well.

969
01:26:36,737 --> 01:26:49,281
And then if there's an exercise room, then I can build up the motivation to go go there
and and like uh do do like pull ups or like exercises or run or or something.

970
01:26:49,281 --> 01:26:52,352
All the stuff that like burpees doesn't like cover super well.

971
01:26:52,352 --> 01:26:53,562
But um

972
01:26:54,506 --> 01:27:03,322
Once you once you realize you can do exercise like right in your room, then all that
stuff's just a cherry on top and you already you already like passed the test, so to

973
01:27:03,322 --> 01:27:03,843
speak.

974
01:27:03,843 --> 01:27:08,410
And now it's just about passing to to going to like an a uh an A or B.

975
01:27:08,856 --> 01:27:09,407
Mm.

976
01:27:09,407 --> 01:27:12,519
Do you do you change your exercise routine based off of where you are?

977
01:27:12,519 --> 01:27:17,706
Like if you're not traveling, you do one set of things and when you're traveling, you do
something separate, or did

978
01:27:18,073 --> 01:27:23,187
because uh I I I guess like because once I'm home, I actually have a gym and and stuff.

979
01:27:23,187 --> 01:27:25,669
I definitely go for like way more d diversity.

980
01:27:25,669 --> 01:27:30,373
Like if I was only doing burpees forever, that might be like bad.

981
01:27:30,373 --> 01:27:33,935
I it seems like you you get some kind of muscle imbalance from from doing that.

982
01:27:33,935 --> 01:27:43,132
So I guess compensate on the other end by trying to mix it up as much as possible when
when when I when I'm when I do have the motivation do other stuff.

983
01:27:43,676 --> 01:27:46,160
Do you also exercise while you're on vacation?

984
01:27:46,754 --> 01:27:49,015
Uh not nearly as much.

985
01:27:49,015 --> 01:28:01,982
May maybe a couple of days, but um vacation here at least like for for the past couple for
me, I I'm like walking so much that I it seems like a reasonable enough substitute.

986
01:28:03,411 --> 01:28:04,392
Yeah, I like to think.

987
01:28:04,392 --> 01:28:07,574
Or I'm just like physically exhausted by that the butt point.

988
01:28:07,574 --> 01:28:08,894
I don't know if I can.

989
01:28:09,535 --> 01:28:17,610
but I also pick like vacation destinations where there's like a lot of like tr temperate
climate or something very exhausting and far away from infrastructure.

990
01:28:17,610 --> 01:28:20,622
So yeah, I I I totally get you.

991
01:28:21,643 --> 01:28:22,271
I like it.

992
01:28:22,271 --> 01:28:24,111
I like I like the recommendation.

993
01:28:24,111 --> 01:28:26,025
Um okay.

994
01:28:26,246 --> 01:28:32,796
I actually wasn't sure what I wanted to share, uh, but I found for my pick a article.

995
01:28:32,796 --> 01:28:35,782
Called I Left Port Twenty Two Open for Fifty Four Days.

996
01:28:35,782 --> 01:28:44,303
Uh it's by Armin Hussein, and there's some interesting conclusions uh based off of leaving
uh

997
01:28:44,303 --> 01:28:46,808
is like anyone could SSH in there.

998
01:28:48,454 --> 01:28:49,178
No, no.

999
01:28:49,178 --> 01:28:51,180
I I'm uh I'm just like confirming.

1000
01:28:51,180 --> 01:28:52,079
That's the idea.

1001
01:28:52,079 --> 01:28:53,009
yeah, yeah, yeah, for sure.

1002
01:28:53,009 --> 01:28:57,742
So like SSH server running on port twenty two and you get to see what people are doing.

1003
01:28:57,742 --> 01:29:01,874
Like what like what what a p what what does the internet try to do with with your port?

1004
01:29:01,974 --> 01:29:08,738
And what's really interesting about like the conclusions that he finds from it, like I I
don't know what you would expect to have happen.

1005
01:29:09,088 --> 01:29:11,039
so it wasn't a real computer.

1006
01:29:11,039 --> 01:29:16,682
It was uh like a virtual machine that was specifically set up to capture and respond to
certain

1007
01:29:16,882 --> 01:29:17,719
commands and events.

1008
01:29:17,719 --> 01:29:19,680
never like read this article.

1009
01:29:20,421 --> 01:29:26,836
Um I I guess like someone would would like do some like Bitcoin mining program on on it or
like ransomware.

1010
01:29:27,051 --> 01:29:36,276
Well that's the thing, is like you try as a as an attacker, uh you know, you it it's it
starts you figure out what the optimal thing is for you to do um with a machine that you

1011
01:29:36,276 --> 01:29:38,437
find that you can SSH into.

1012
01:29:38,778 --> 01:29:48,344
And that means from a defense standpoint, you may be interested, what are people trying to
do with my server so that for my actual servers, if I wanted to uh configure them in some

1013
01:29:48,344 --> 01:29:52,764
way to deal with malicious attackers, I can look for a same sort of signatures.

1014
01:29:52,764 --> 01:29:55,736
So a question is like, well, what did he actually find?

1015
01:29:55,736 --> 01:30:06,303
And the interesting thing is that most people, most of the uh attacks that were done, like
Bitcoin miners or whatever, were just basically an automated command that was run straight

1016
01:30:06,303 --> 01:30:10,165
against the machine, like SSHN, trying any password.

1017
01:30:10,165 --> 01:30:18,791
They get the password, it works, and then they just run a single command, usually uh CNC
based command or C2 infrastructure, basically, it would be a single command which would

1018
01:30:18,791 --> 01:30:22,833
pull stuff from a third party server that the c attacker owned to run.

1019
01:30:22,965 --> 01:30:25,588
infrastructure on that machine, like a Bitcoin miner or whatever.

1020
01:30:25,588 --> 01:30:33,435
And if you run uh, you know, Bitcoin miner.start, uh the the fake server would just
respond, okay, you know, like it's running, right?

1021
01:30:33,435 --> 01:30:34,525
And uh and that's it.

1022
01:30:34,525 --> 01:30:35,336
End of story.

1023
01:30:35,336 --> 01:30:40,891
And so the attacker, you know, thinks it worked and then they go away and you capture the
signature of what uh these attackers were doing.

1024
01:30:40,891 --> 01:30:50,171
And so you can capture all of the C2 uh command and control servers that were uh in effect
that were you know currently being used at this time by proportion and whatnot.

1025
01:30:50,171 --> 01:30:56,796
And what he found is that like 99% of the attackers, the visitors that went there, never
went on beyond like a single command.

1026
01:30:56,796 --> 01:31:04,453
Basically, they found the open server, they attempted to authenticate with one of like a
thousand most common used pass username and passwords.

1027
01:31:04,453 --> 01:31:06,303
Then they run the result and that's it.

1028
01:31:06,303 --> 01:31:06,765
They're done.

1029
01:31:06,765 --> 01:31:09,517
They they disconnect and the server just keeps on running.

1030
01:31:09,517 --> 01:31:16,063
Some of those actually attacks didn't even run the script in the background.

1031
01:31:16,063 --> 01:31:18,924
So as soon as they disconnected, it just stopped.

1032
01:31:19,449 --> 01:31:22,830
The attack, like the miner just would stop at that point.

1033
01:31:22,830 --> 01:31:24,240
Like the command wasn't even sufficient.

1034
01:31:24,240 --> 01:31:29,502
Uh, but like in the top one percent, there was actually some really interesting stuff
going on.

1035
01:31:29,502 --> 01:31:37,394
Like, you could he could tell how sophisticated an attacker was based off of the type of
command that they were doing.

1036
01:31:37,394 --> 01:31:42,726
Like, were they running something that would get saved in the user's bash history for
authentication?

1037
01:31:42,726 --> 01:31:45,477
So, like, if you went in later, you'd be able to see this in the history.

1038
01:31:45,477 --> 01:31:47,057
Well, there was like some.

1039
01:31:47,548 --> 01:31:58,687
Clearly state sponsored actors, uh, some from the um probably Fen French government with
French IP addresses that were you know disabling bash history and using direct uh port

1040
01:31:58,687 --> 01:32:03,838
protocol to handle messages and whatnot to completely avoid getting anything logged.

1041
01:32:03,838 --> 01:32:06,193
Uh most of it was crypto related in some way.

1042
01:32:06,193 --> 01:32:12,558
People trying to run Bitcoin miners or because Bitcoin is worthless, uh Solana, which is
much cheaper.

1043
01:32:12,558 --> 01:32:14,870
You don't need infrastructure to run Bitcoin miners.

1044
01:32:14,870 --> 01:32:16,847
Uh I think it's it costs.

1045
01:32:16,847 --> 01:32:20,940
I think right now it costs more to mine a Bitcoin than Bitcoin is worth.

1046
01:32:20,940 --> 01:32:29,805
So people who mine Bitcoin are only doing it on with malicious uh intentions, which I
think is absurd.

1047
01:32:32,228 --> 01:32:33,648
yeah, yeah, for sure, right?

1048
01:32:33,648 --> 01:32:35,530
you get some cloud credits and you run it.

1049
01:32:35,530 --> 01:32:37,621
You hijack someone's account and you run it, right?

1050
01:32:37,621 --> 01:32:38,632
You're not doing it.

1051
01:32:38,632 --> 01:32:43,315
I mean, if you're really smart, you use Monero instead, because it's untraceable.

1052
01:32:43,315 --> 01:32:46,457
So no one can track that these are from the same.

1053
01:32:46,655 --> 01:32:49,127
account or uh potential threat actor.

1054
01:32:49,127 --> 01:32:50,568
But a lot of it was complete garbage.

1055
01:32:50,568 --> 01:32:53,290
Uh that wasn't very smart.

1056
01:32:53,290 --> 01:32:55,441
They weren't doing very s something very interesting.

1057
01:32:55,441 --> 01:32:57,873
Some people were just curious, like, I found a server.

1058
01:32:57,873 --> 01:33:01,565
I'm gonna SSH in here and see what I can do and then run some interesting commands.

1059
01:33:01,565 --> 01:33:04,007
A lot of them were just running LLS to start.

1060
01:33:04,007 --> 01:33:04,898
Like where am I?

1061
01:33:04,898 --> 01:33:06,769
What do I have access to in this directory?

1062
01:33:06,769 --> 01:33:07,170
I don't know.

1063
01:33:07,170 --> 01:33:09,471
I just I found this article so interesting.

1064
01:33:09,922 --> 01:33:12,085
Wow, yeah, that that is cool.

1065
01:33:12,085 --> 01:33:18,794
It it's cool how um I guess this the sophistication four follows kind of like a power law.

1066
01:33:18,794 --> 01:33:19,535
That that's one.

1067
01:33:19,535 --> 01:33:22,098
Um two, it's kinda funny.

1068
01:33:22,098 --> 01:33:28,586
I i it's not clear that anyone like benefited significantly from the successful attack on
on on on the server.

1069
01:33:31,069 --> 01:33:31,603
yeah.

1070
01:33:31,603 --> 01:33:32,574
no, for for sure not.

1071
01:33:32,574 --> 01:33:33,527
Um I mean

1072
01:33:33,527 --> 01:33:36,310
enrolled maybe got enrolled into like a botnet or something.

1073
01:33:36,310 --> 01:33:40,564
That'd probably be the best outcome out of what I can imagine for the attacker.

1074
01:33:40,793 --> 01:33:51,591
Yeah, well there are there there is a so outside of the Bitcoin or crypto mining, uh there
is a small section where they were basically just recording metrics about the actual

1075
01:33:51,591 --> 01:33:52,042
machine.

1076
01:33:52,042 --> 01:34:05,072
So like where it was, its IP address, what it was actually running, uh, which version of
Linux or uh distribution, uh Ubuntu, Kubuntu, whatever, uh Gentu, et cetera, uh, for what

1077
01:34:05,072 --> 01:34:07,087
presumably would be a later attack.

1078
01:34:07,087 --> 01:34:11,570
So that later they could come back and be like, you know what, we want a machine in this
region with this IP address so that we could use.

1079
01:34:11,570 --> 01:34:14,713
And so it was just about fingerprinting the actual machine.

1080
01:34:14,713 --> 01:34:17,565
And a lot of the messages were were from that.

1081
01:34:17,565 --> 01:34:18,455
Yeah.

1082
01:34:18,455 --> 01:34:19,771
so like I said, really interesting.

1083
01:34:19,771 --> 01:34:20,487
It's not very long.

1084
01:34:20,487 --> 01:34:21,978
It's a like a 20-minute read.

1085
01:34:21,978 --> 01:34:24,009
Uh pretty interesting article.

1086
01:34:24,632 --> 01:34:25,447
Cool.

1087
01:34:27,237 --> 01:34:34,135
Well, thank you, Mark, for joining us today and telling us all about the classification at
Facebook and what's next in the semantic layers.

1088
01:34:34,135 --> 01:34:38,430
Uh I I already hear that semantic layers are are over and done.

1089
01:34:38,430 --> 01:34:40,543
We're already on the next the next great thing.

1090
01:34:40,543 --> 01:34:44,557
So always interesting to hear that the latest tech is now obsolete.

1091
01:34:45,964 --> 01:34:46,316
Right.

1092
01:34:46,316 --> 01:34:48,192
Um just until un until the next one.

1093
01:34:48,192 --> 01:34:53,298
Uh until your next guest tries to uh yeah, another generation.

1094
01:34:53,647 --> 01:34:58,011
until next week where we find out that the replacement for semantic layers is already
obsolete.

1095
01:34:58,011 --> 01:35:01,213
Uh I don't remember what next week's episode is at the moment.

1096
01:35:01,213 --> 01:35:06,697
Um but thanks all the listeners for tuning in for this week and I hope to see everyone
back next week.