1
00:00:07,778 --> 00:00:12,920
Welcome back to Adventures in DevOps, where we frequently question if a spreadsheet can be
a microservice.

2
00:00:12,920 --> 00:00:21,104
Today's guest speculated over twenty years ago that the web was being designed
suboptimally, where it only took four years of maintaining an Oracle ERP system to burn

3
00:00:21,104 --> 00:00:24,325
out, found a new company on his speculations, and become successful.

4
00:00:24,325 --> 00:00:26,296
The CEO of Ragic, Jeff Quo.

5
00:00:26,296 --> 00:00:27,176
Welcome to the show.

6
00:00:27,176 --> 00:00:28,467
Hello everyone.

7
00:00:28,587 --> 00:00:33,149
Yeah, so I gotta say, what old what it what was old is new again.

8
00:00:33,149 --> 00:00:35,598
You were advocating for the semantic web

9
00:00:35,598 --> 00:00:36,922
Twenty years ago or so.

10
00:00:36,922 --> 00:00:38,788
And now we're basically back to it.

11
00:00:38,788 --> 00:00:39,438
uh

12
00:00:39,438 --> 00:00:40,048
Okay, yeah.

13
00:00:40,048 --> 00:00:45,492
Semantic web was my the the area of study of my master's thesis like twenty five years
ago.

14
00:00:45,492 --> 00:00:59,412
Wow, okay, that's my master's thesis was twenty five years ago and it started out as some
studies in AI and and semantic web and it just kind of evolved into this kind of

15
00:00:59,412 --> 00:01:07,798
prototype, like creating a system that can be semantically integrated with others and
while being really fast.

16
00:01:07,842 --> 00:01:08,452
to build.

17
00:01:08,452 --> 00:01:14,125
You can build simple database systems really quickly using this kind of technology.

18
00:01:14,125 --> 00:01:21,269
So that's kind of the the background of of that, what also l in later years turned into
Reag.

19
00:01:21,269 --> 00:01:29,534
So yeah, in my in my master thesis defense I I even demoed a prototype, the like the very
earliest version of Reagic there.

20
00:01:29,534 --> 00:01:30,086
Yeah.

21
00:01:30,086 --> 00:01:40,192
it's really interesting because it reminded me a lot about what is being used even in the
last decade or so as like JSON LD in like metadata headers within HTML pages.

22
00:01:40,192 --> 00:01:49,117
Like I think it's this real real idea where programmatic systems do not want to have to
scrape HTML and figure out like what is the important parts of that document.

23
00:01:49,117 --> 00:01:58,486
And I feel like it took until a bunch of AI companies were harassing every single website
out there for understanding the semantics of what's

24
00:01:58,486 --> 00:01:59,468
What's the point of the company?

25
00:01:59,468 --> 00:02:00,772
What does the product do?

26
00:02:00,772 --> 00:02:06,626
All the documentation pages for someone to stand up and say, like, we need to do something
better.

27
00:02:06,626 --> 00:02:20,684
But with the with the L L AI coming, it just kind of in a way erase the purpose of the
semantic web because you know agents they can't understand the semantics and they can

28
00:02:20,684 --> 00:02:26,778
aggregate all those they don't really need the oncology and to to to to categorize them.

29
00:02:26,778 --> 00:02:31,480
They can just read it and categorize it in their in their models.

30
00:02:31,480 --> 00:02:35,118
So yeah, that's something we we never ex really expect.

31
00:02:35,118 --> 00:02:36,879
Can we fight on this on this point?

32
00:02:36,879 --> 00:02:38,731
Uh I I want to think sure.

33
00:02:38,731 --> 00:02:40,762
I want to take the opposite perspective.

34
00:02:40,763 --> 00:02:52,982
I feel like I see I've seen in a bunch of the IETF working groups a bunch of companies
that quote unquote are AI companies struggling to parse effectively uh HTML to actually

35
00:02:52,982 --> 00:02:53,953
truly understand.

36
00:02:53,953 --> 00:02:56,305
So correct parsing is a is an issue.

37
00:02:56,305 --> 00:03:03,241
I think we end up with a lot of hallucinations because it is there's no like correct
answer that can be ex like extrapolated from that.

38
00:03:03,241 --> 00:03:05,100
And the other one is token costs.

39
00:03:05,100 --> 00:03:10,031
that companies are trying to reduce as much as they can uh what they're consuming.

40
00:03:10,031 --> 00:03:23,285
And so the d there's been a desire to shift from uh human visual language, like a U UI or
GUI, to using like LLM's text or something that was basically markdown for for processing

41
00:03:23,285 --> 00:03:29,037
because it's more optimized for actually I hate using the word understanding, but I guess
I'll use it here.

42
00:03:29,037 --> 00:03:32,048
So that LMs can understand what the page is actually doing.

43
00:03:32,048 --> 00:03:33,656
So I agree that there is a

44
00:03:33,656 --> 00:03:36,128
capability to completely parse what's there.

45
00:03:36,128 --> 00:03:38,190
There's no reason to necessarily change.

46
00:03:38,190 --> 00:03:48,988
I feel like there is a quite an opportunity to really go back and challenge the everything
we've built up and since the dot com bust uh about how we're doing it and instead go back

47
00:03:48,988 --> 00:03:58,255
to those roots that you sort of identified and really from that perspective, like what
should actually be contained in a web page in order to make it usable for the entities

48
00:03:58,255 --> 00:03:59,394
that interact with it.

49
00:03:59,394 --> 00:03:59,814
Yeah.

50
00:03:59,814 --> 00:04:04,596
Well I haven't really been in the semantic web field for many, many years.

51
00:04:04,596 --> 00:04:15,301
But I I I do agree that in in detail it's it's still different from you know, having the
the correct ontology set with LM just parsing it.

52
00:04:15,301 --> 00:04:24,504
But I do kind of believe that LM can be immensely helpful in automating this process to to
generate these graphs.

53
00:04:24,504 --> 00:04:26,915
But of course that will cost a lot of tokens.

54
00:04:26,915 --> 00:04:28,810
So yeah, it's

55
00:04:28,810 --> 00:04:31,098
It it's kind of like half the answer.

56
00:04:31,098 --> 00:04:31,576
Yeah.

57
00:04:31,576 --> 00:04:42,774
So took this really the the goal of the semantic web, which I I don't know was entirely
realized in the in the public, and you converted it to and forgive me for my lack of

58
00:04:42,774 --> 00:04:48,354
understanding here, it seems like a much better version of Airtable is what you've built
at Ragik.

59
00:04:48,354 --> 00:04:54,236
Yeah, that's kind of kind of the intended effect and it we're not really happy about this.

60
00:04:54,236 --> 00:04:57,658
We actually started before Airtable and they got so successful.

61
00:04:57,658 --> 00:05:00,089
But anyway, but and yeah.

62
00:05:00,089 --> 00:05:12,884
Um we we we we are under the same assumption that and spreadsheets are kind of the de
facto interface that business people like to um interact with data.

63
00:05:12,884 --> 00:05:17,614
So in the very beginning, in the very beginning, Ragic is nothing about spreadsheets.

64
00:05:17,614 --> 00:05:20,996
So in my master's thesis, there's nothing about spreadsheets.

65
00:05:20,996 --> 00:05:30,552
We we adopted the the Species interface like two or three years in when when we s when we
started to really commercialize this this product.

66
00:05:30,552 --> 00:05:40,169
So the spreadsheet interface was kind of added later to make the whole database building
process easier to understand for business people.

67
00:05:40,169 --> 00:05:42,210
Because when we are demoing our product,

68
00:05:42,210 --> 00:05:45,662
They also they they always say, can you do this just like on Excel?

69
00:05:45,662 --> 00:05:56,368
Could you like drag this and could you like click that and and use hotkey to change focus
on different fields or okay, so you're basically asking me to build an a special

70
00:05:56,368 --> 00:05:56,884
interface.

71
00:05:56,884 --> 00:06:02,671
So okay, so why why don't I just build a special interface and every everybody could be
happy?

72
00:06:02,671 --> 00:06:09,645
So yeah, so we when we reach the market we gradually and realize that, you know, this is
kind of what people want.

73
00:06:09,645 --> 00:06:11,626
And we're just lucky that w

74
00:06:11,626 --> 00:06:16,898
Underneath it we have the data structure, the flexibility to support something like this.

75
00:06:16,898 --> 00:06:18,258
Yeah, no, I could totally understand that.

76
00:06:18,258 --> 00:06:27,741
I think this is one thing that a lot of products get wrong when they are coming up with a
new model, is they are envisioning what the user experience should be, and users aren't

77
00:06:27,741 --> 00:06:31,062
necessarily comfortable or would have to go through the process of learning that.

78
00:06:31,062 --> 00:06:39,864
So it's it's sort of like a it's like an unfortunate conclusion that you're like, we have
the perfect, you know, user interface that would be beneficial here, but they're stuck on

79
00:06:39,864 --> 00:06:42,445
a a legacy understanding of how it would work.

80
00:06:42,445 --> 00:06:45,466
Uh so in your conversion from going

81
00:06:45,650 --> 00:06:53,097
from whatever interface you had to more of a spreadsheet, like did you ever consider
having multiple different layers where you could optimize for the particular workflows

82
00:06:53,097 --> 00:07:02,025
specifically and let users decide if they want the sort of first level spreadsheet
interface or if there's like a more workflow centric view and like let them switch back

83
00:07:02,025 --> 00:07:02,485
and forth?

84
00:07:02,485 --> 00:07:07,362
Or did you just go all in on like, well, no, we're spreadsheets, you know, may actually be
the right answer.

85
00:07:07,362 --> 00:07:18,445
Yeah, we basically went all in saying that no Sprechy is the answer because for most
database builders, like traditional database builders, you can say Microsoft Access or

86
00:07:18,445 --> 00:07:23,206
FileMaker, they they usually have a two layer architecture.

87
00:07:23,206 --> 00:07:34,290
So they have a database uh underneath it and you can design the tables and you could kind
of write the SQL or they generate a SQL for you and you design the the interface pages.

88
00:07:34,290 --> 00:07:35,840
So um

89
00:07:35,840 --> 00:07:47,895
it's usually like two layered and we we feel like and this is kind of unnecessary
complication for non technical people because for them it's the same thing.

90
00:07:47,895 --> 00:07:54,668
So we would like to have them to manipulate the data model and the interface at the same
time.

91
00:07:54,668 --> 00:07:58,499
Because for in in their mental model it's it's the same thing.

92
00:07:58,499 --> 00:08:03,461
So we want to match how what they think about the date the data.

93
00:08:03,461 --> 00:08:05,882
So especially in the beginning

94
00:08:06,186 --> 00:08:14,561
It it's kind of interesting that technical background people who have written database
applications first get a little bit confused when using Regix.

95
00:08:14,561 --> 00:08:17,913
So okay, so where do I create the tables?

96
00:08:17,913 --> 00:08:19,214
How do I add the fields?

97
00:08:19,214 --> 00:08:21,155
How do I add an index?

98
00:08:21,155 --> 00:08:33,172
So it's a little bit confusing for technical people, but for business people, for some
people, they just get it like real really quickly and they just don't worry at worry about

99
00:08:33,407 --> 00:08:36,279
like changing from one to one to one to many relationships.

100
00:08:36,279 --> 00:08:40,111
But but database people they'll say, No, you you can't do this.

101
00:08:40,111 --> 00:08:41,721
That that's not right.

102
00:08:42,162 --> 00:08:48,095
It's always that but then they'll but then they're gonna be be like, w whoa, this just how
the hell do you do that?

103
00:08:48,095 --> 00:08:50,976
Just because it's just structured differently.

104
00:08:51,050 --> 00:09:01,725
I think this is the challenge for uh technical um and engineers or even technical business
people who like really understand how data works because you know that there is like sort

105
00:09:01,725 --> 00:09:04,746
some sort of lie that's been propagated that's unavoidable.

106
00:09:04,746 --> 00:09:10,588
And it's like most people don't need to know about that, but they get stuck on like, wait,
but how can I do this with my data?

107
00:09:10,588 --> 00:09:11,959
Like, are you doing something weird?

108
00:09:11,959 --> 00:09:13,820
It's like, yeah, actually we're doing something weird.

109
00:09:13,820 --> 00:09:16,491
I mean, it's clever and weird to deal with the business problem.

110
00:09:16,491 --> 00:09:19,414
Uh so you you must have had quite the

111
00:09:19,414 --> 00:09:28,561
Opportunity to learn about those weird oddities that you'd have to put into your product
over time, especially building up spreadsheet services are not are like not simple.

112
00:09:28,561 --> 00:09:30,042
I mean, the UI looks simple, right?

113
00:09:30,042 --> 00:09:34,345
That's the sort of lie we tell ourselves that, the implementation must also be simple.

114
00:09:34,345 --> 00:09:43,611
But since you've had such a long go at this, have there been some cornerstone cases or
challenging or controversial things that have happened in the last two decades or so?

115
00:09:43,611 --> 00:09:47,468
Good learnings or challenges that you've found with the architecture that you've built up?

116
00:09:47,468 --> 00:09:59,886
Yeah, I think the biggest difference between Ragic and other database builders like
traditionally like Microsoft Access or FileMaker is that it's not based on um SQL.

117
00:09:59,886 --> 00:10:01,648
It's not based on database tables.

118
00:10:01,648 --> 00:10:03,089
It's not relational.

119
00:10:03,089 --> 00:10:03,359
Okay.

120
00:10:03,359 --> 00:10:06,011
So it's basically based on graph.

121
00:10:06,011 --> 00:10:08,062
Like because it came from semantic web.

122
00:10:08,062 --> 00:10:10,714
I I d was designing it as a graph.

123
00:10:10,714 --> 00:10:13,876
So it it has a lot more flexibility.

124
00:10:13,876 --> 00:10:17,428
So you you don't really need to change the data structure.

125
00:10:17,432 --> 00:10:21,785
when you go from one to one to one to many or change it to many to many.

126
00:10:21,785 --> 00:10:24,786
So w that's one of the greatest strengths.

127
00:10:24,806 --> 00:10:28,869
But um it's just really hard to implement.

128
00:10:28,869 --> 00:10:35,423
And in the beginning, I was just I was just a a master student, graduate student.

129
00:10:35,423 --> 00:10:39,715
So so it it took me quite a long time to to get it right.

130
00:10:39,715 --> 00:10:44,394
So the the first version, um in the beginning I I tried to

131
00:10:44,394 --> 00:10:46,074
Remember this is before spreadsheets.

132
00:10:46,074 --> 00:10:53,937
So we're still trying to build this super flexible data model, graph based data model that
can work on any kind of business data.

133
00:10:53,937 --> 00:11:01,539
So so in the very beginning, I I I tried to use Hibernate because at that time it's kind
of like the the the standard.

134
00:11:01,539 --> 00:11:08,021
So everything well if you have theta you should probably use Hibernate and O O R to do the
mapping and do everything.

135
00:11:08,021 --> 00:11:13,858
So I use Hibernate and I try to implement it with Hibernate and it doesn't really work.

136
00:11:13,858 --> 00:11:24,667
Because yeah, because I'm what I'm trying to build is to customize as you can if as you
can imagine, if you try to build a a flexible database builder, um it's not gonna be ri

137
00:11:24,667 --> 00:11:27,559
that easy to make all those kind of changes in Hibernate.

138
00:11:27,559 --> 00:11:31,693
So I think okay so I I I have I have to write raw SQL.

139
00:11:31,693 --> 00:11:35,436
So I began to I changed to H SQL, hypersonic SQL.

140
00:11:35,436 --> 00:11:39,079
It was a like a tiny database that I can embed in the application.

141
00:11:39,079 --> 00:11:42,742
So I began to write it in SQL in a lot of

142
00:11:42,742 --> 00:11:43,252
joins.

143
00:11:43,252 --> 00:11:54,509
So I I began to write real, really large SQL and try to write code to generate a lot of
almost like recursive joins to to search through the data.

144
00:11:54,509 --> 00:11:59,741
And that kind of worked, but and that created a lot of performance problems.

145
00:11:59,741 --> 00:12:06,455
So there there was um kind of a big prop performance and barrier that that we're trying to
break.

146
00:12:06,455 --> 00:12:11,734
So there's like one day I was just talking to some of my tech friends and he was talking

147
00:12:11,734 --> 00:12:22,939
about how great this this this thing, the Berkeley Berkeley D B, uh how great this D B
that a lot a lot of people has never heard of and he has he has been saying, how great

148
00:12:22,939 --> 00:12:25,300
this is and I was like, Okay, I'm I I'll try this out.

149
00:12:25,300 --> 00:12:34,123
So so th so then I think yeah, because I I think it solves a lot of my problem when doing
these queries on the graph.

150
00:12:34,123 --> 00:12:39,305
Because at that time there's no tool like Neil for J that a graph database.

151
00:12:39,305 --> 00:12:40,456
Yeah, we don't we don't have that.

152
00:12:40,456 --> 00:12:41,376
It's it's

153
00:12:41,718 --> 00:12:44,799
quite a few years after my initial development.

154
00:12:44,799 --> 00:12:53,121
So I began trying out Berkeley DB and and slowly to to to see that this could could
actually solve our problem.

155
00:12:53,121 --> 00:13:00,784
So because what we need is a lot of very, very small quick queries to the database.

156
00:13:00,784 --> 00:13:09,006
We need to quickly query a lot of nodes on the gr on the graph and query like what links
they have and if these two nodes have links.

157
00:13:09,006 --> 00:13:09,706
So

158
00:13:09,706 --> 00:13:14,850
Every single one of those queries are very fast, uh very simple, but we need to do a lot
of them.

159
00:13:14,850 --> 00:13:23,838
So wh how we end up doing is that um Berkeley D B can be embedded into the the there's a
Java edition can be embedded into the process.

160
00:13:23,838 --> 00:13:29,423
So it's actually all those queries are l are just API calls and within the same process.

161
00:13:29,423 --> 00:13:30,544
So it's super fast.

162
00:13:30,544 --> 00:13:30,895
Yeah.

163
00:13:30,895 --> 00:13:37,580
So but after we we tried to adopt um Berkeley DB, the the performance was like

164
00:13:38,050 --> 00:13:43,404
ten a hundredfold from the original ones using the the SQL.

165
00:13:43,404 --> 00:13:45,556
So it was it made a huge difference.

166
00:13:45,556 --> 00:13:50,430
But but Berkeley D B is not exactly a a full database.

167
00:13:50,430 --> 00:13:52,602
It's like half a database.

168
00:13:52,602 --> 00:13:57,686
So the the problem is that you have to basically implement half the database.

169
00:13:57,908 --> 00:14:02,965
uh well, I I think today there still are no graph good graph database options available.

170
00:14:02,965 --> 00:14:09,336
And so have you switched off of using the Berkeley D B or are you are you still using that
to power most of your technology?

171
00:14:09,336 --> 00:14:11,247
We're still we're still using that.

172
00:14:11,247 --> 00:14:11,627
Yeah.

173
00:14:11,627 --> 00:14:12,766
If it ain't broke, don't fix it.

174
00:14:12,766 --> 00:14:20,839
Which it it took there are s still lots of weird stories that we we face while using
Berkeley DB trying to fix that.

175
00:14:20,839 --> 00:14:27,561
There there's still lots of problems because you know, when you're building half a
database, there are just tons of unforeseen issues.

176
00:14:27,561 --> 00:14:31,922
And I I was just a gr gr grad student, so what do I know?

177
00:14:31,922 --> 00:14:33,683
So I thought, wow, this works.

178
00:14:33,683 --> 00:14:36,364
It it it's it's like ten times, a hundred times faster.

179
00:14:36,364 --> 00:14:37,684
I'm I'm dedicated to that.

180
00:14:37,684 --> 00:14:38,358
So

181
00:14:38,358 --> 00:14:40,581
I I I just feel that this is the greatest thing.

182
00:14:40,581 --> 00:14:44,296
This will this will actually make our product commercially viable.

183
00:14:44,296 --> 00:14:45,427
So yeah.

184
00:14:45,427 --> 00:14:50,434
So we stick to that and we we've been sticking to Berkeley D B until until today.

185
00:14:50,434 --> 00:14:50,836
Yeah.

186
00:14:50,836 --> 00:14:59,992
If you had designed the the company and the the product from the ground up starting today,
would you st would you go for that again or would you consider one of the other maybe a

187
00:14:59,992 --> 00:15:09,016
like a a relational data model or even a key value store for storing the graph nodes in or
another or an actual first class um graph database provider?

188
00:15:09,016 --> 00:15:13,147
Well Berkeley D B is actually basically a name value store.

189
00:15:13,168 --> 00:15:18,060
And honestly, yeah, I would probably still use that.

190
00:15:18,060 --> 00:15:18,710
Cool.

191
00:15:18,710 --> 00:15:28,894
Because yeah, even if I use something like Neo4J, like this already graph based, but
because we write our own in the indexes, we we design our own index.

192
00:15:28,894 --> 00:15:32,516
We could do s all sorts of cool, weird things with it.

193
00:15:32,516 --> 00:15:38,072
So if we because we have so much low low level control of the database, we just it just

194
00:15:38,072 --> 00:15:39,753
gives us a lot more flexibility.

195
00:15:39,753 --> 00:15:51,802
But it it it takes time to work through all those issues and and oddities of the of
writing your own no ha writing your own index, writing your own own half a database.

196
00:15:51,802 --> 00:15:55,205
But we've worked through that and it it turns out pretty well.

197
00:15:55,205 --> 00:15:58,738
So so we we do think it's kind of worth it.

198
00:15:58,738 --> 00:15:59,628
Yeah.

199
00:15:59,662 --> 00:16:00,473
It's really amazing.

200
00:16:00,473 --> 00:16:06,448
Like for every for every story that there's, you know, we successfully built basically a
database.

201
00:16:06,448 --> 00:16:15,336
There's a hundred more of that was the worst mistake we ever made as a company, thinking
that, you know, we could just spin up a file system, you know, write some blobs to it or

202
00:16:15,336 --> 00:16:19,000
blocks to it, and we'd be totally good and and not run into problems.

203
00:16:19,000 --> 00:16:28,032
I mean, it sounds like, you know, in your scenario, given the nuances that are contained
in how the product actually works and the specialty and the experience.

204
00:16:28,032 --> 00:16:34,736
that this is one of those areas where it was much more valuable to actually learn gain
that g knowledge and and insight and go from that.

205
00:16:34,736 --> 00:16:37,748
And you know, it's interesting you bring this up for a couple of reasons.

206
00:16:37,748 --> 00:16:44,372
The first one is that uh my company has a has a product where there is relationships
between nodes that we want to capture.

207
00:16:44,372 --> 00:16:52,037
And we evaluated all of the Graph DB products and all of the relational databases and key
value stores and stuff like that to figure out how to store stuff.

208
00:16:52,037 --> 00:16:55,048
And we we had a very complex model to start out with.

209
00:16:55,176 --> 00:17:04,390
And when we were evaluating stuff based off of like P ninety nines of what customers
needed or what we even needed to support queries and whatnot, most graph databases would

210
00:17:04,390 --> 00:17:12,234
be slower by a factor of magnitude than performing repeated queries or joins on relational
or multiple queries to a key value store.

211
00:17:12,234 --> 00:17:20,178
If things were simple, you maybe can get away with that one clever query or three clever
queries or a clever join.

212
00:17:20,178 --> 00:17:21,218
And while

213
00:17:21,248 --> 00:17:23,239
Every engineer who looks at it is like, that's disgusting.

214
00:17:23,239 --> 00:17:25,201
I can't believe you would have that in your source code.

215
00:17:25,201 --> 00:17:31,765
Uh the reality is graph databases were never designed as like speed optimal things.

216
00:17:31,765 --> 00:17:38,089
And so you end up with this huge challenge of if you want your solution to be fast, you do
an ugly thing in your database.

217
00:17:38,310 --> 00:17:46,876
And I feel like if you're met with those two options, both it's gonna be ugly if you do
this with a relational database or a key value store, and it's gonna be, but it's gonna be

218
00:17:46,876 --> 00:17:50,210
faster versus it's gonna be slow, but maybe it's slightly better.

219
00:17:50,210 --> 00:17:53,563
But we can make it even better if we if we write our own database.

220
00:17:53,563 --> 00:17:55,782
You know, I I'd be really curious to know.

221
00:17:55,782 --> 00:18:01,081
Um, I mean, you basically have the experience now to potentially go out and just write the
whole database from scratch.

222
00:18:01,081 --> 00:18:11,128
Have you ever considered making that final leap, like using the Berkeley DB as sort of a
starting point and reconfiguring it in a way that would be optimal for the the business?

223
00:18:11,128 --> 00:18:16,281
You know, with today's AI assisted coding, I don't think that's that far off.

224
00:18:16,281 --> 00:18:21,123
The the honest take is that we don't really see a a huge need for that.

225
00:18:21,264 --> 00:18:27,507
Well we we see a little bit of demand for for something that's like inside Berkeley D B
that we cannot change.

226
00:18:27,507 --> 00:18:38,853
But I I do think that for now we don't w we still feel it's kind of still kind of a big
effort and yeah, w we we don't really see the benefit there yet.

227
00:18:38,853 --> 00:18:41,088
But maybe maybe someday

228
00:18:41,088 --> 00:18:42,558
Maybe someday we we will.

229
00:18:42,558 --> 00:18:53,382
But but really like a few years ago when we were tackling with all those problems in in
Berkeley DB, and yeah, we were thinking about oh come on, w why why don't we just write

230
00:18:53,382 --> 00:18:55,193
our own write our own database?

231
00:18:55,193 --> 00:19:05,657
Because there were some odd bugs, some pretty weirdly serious bugs in especially earlier
versions of Berkeley DB edition.

232
00:19:05,657 --> 00:19:07,002
Yeah, no, please share.

233
00:19:07,002 --> 00:19:08,088
Anything that comes to mind.

234
00:19:08,088 --> 00:19:08,619
Yeah.

235
00:19:08,619 --> 00:19:09,069
Okay.

236
00:19:09,069 --> 00:19:21,760
Well yeah, one of the most serious one in in the earlier version of Berkeley DB is that
when the database gets large, like hundreds of gigabytes large, and and when it's under

237
00:19:21,760 --> 00:19:30,136
like a lot of load, somehow we we cannot reproduce this, but we we know that it's under
high load and a lot of data.

238
00:19:30,337 --> 00:19:33,019
It deletes the wrong data file.

239
00:19:34,561 --> 00:19:36,362
Like, yeah.

240
00:19:36,362 --> 00:19:38,663
It deletes the wrong data file.

241
00:19:38,663 --> 00:19:44,905
And that is just that is the the biggest problem that that we we face.

242
00:19:44,905 --> 00:19:49,476
Like it's it's for for the before the version four.

243
00:19:49,476 --> 00:19:51,376
So right now we're using version seven.

244
00:19:51,376 --> 00:20:03,394
So the version seven's we've never seen this on version seven, but on version four it it
just hits us like once every few months and it just

245
00:20:03,394 --> 00:20:09,917
horrible because we we have to use like automated backup to restore or try to, you know,
fix the data.

246
00:20:09,917 --> 00:20:19,161
So yeah, we we even try to write our own fixer f programs trying to fix this, but just
it's just so difficult because they delete the the wrong data.

247
00:20:19,161 --> 00:20:23,383
And yeah, it it it just helpless when it actually deletes it.

248
00:20:23,383 --> 00:20:31,462
So in the end we we we came up with a solution is that on those version four databases
they they have a flag is to

249
00:20:31,462 --> 00:20:33,213
not actually delete data files.

250
00:20:33,213 --> 00:20:37,565
Just mark it with a s a suffix, a dot DEL.

251
00:20:37,565 --> 00:20:37,885
Okay.

252
00:20:37,885 --> 00:20:40,047
And you write your own program to delete it.

253
00:20:40,047 --> 00:20:43,358
So in the end we we we end up using this flag.

254
00:20:43,358 --> 00:20:46,550
So Berkeley DB never deletes any data file.

255
00:20:46,550 --> 00:21:00,057
And we just periodically scan the whole database and data file folder and move the the
file with the DEL and suffix into another disk, another larger, cheaper disk, and for a

256
00:21:00,057 --> 00:21:01,218
few days and

257
00:21:01,218 --> 00:21:04,201
We make sure that the database is still working all right.

258
00:21:04,201 --> 00:21:12,329
And if we find any of those issues, if that aha, I know this file is fist missing, so I
can go back to that disk and find a missing file.

259
00:21:12,329 --> 00:21:15,825
Yeah, and this this mechanism has been a lifesaver.

260
00:21:15,825 --> 00:21:21,919
It's just a a very wacky solution to a a bug that we cannot just fix for them.

261
00:21:21,919 --> 00:21:25,442
So yeah, that that saved us a lot of time.

262
00:21:25,442 --> 00:21:27,813
I think it's ingenious, honestly, as a as a solution.

263
00:21:27,813 --> 00:21:36,326
And I I think anyone who's questioning this has to realize that there is no technology
that is 100% reliable, one and for databases 100% durable.

264
00:21:36,326 --> 00:21:38,327
You're gonna get a failure in some way.

265
00:21:38,327 --> 00:21:43,639
And usually it's around the writing to the journal and then or through the replication
process.

266
00:21:43,639 --> 00:21:53,143
And then if you have a failover event or a critical failure in your main uh DB cluster for
if you have a um like a main writer or write node and you and the rest of them are

267
00:21:53,143 --> 00:21:54,924
readers, like when you fail over there.

268
00:21:54,924 --> 00:22:00,858
There tends to be stuff that is in the journal but isn't written or isn't even in the
journal because it's still in memory and process.

269
00:22:00,858 --> 00:22:01,999
And how do you deal with that?

270
00:22:01,999 --> 00:22:12,176
And so uh the fact that you like, you know, you found those, you ran into that issue and
you thought about it consistently of how to solve it in a way which doesn't cause any sort

271
00:22:12,176 --> 00:22:13,187
of data corruption.

272
00:22:13,187 --> 00:22:18,740
I mean, it's not great that you were forced into it by, you know, what seems like a bug in
in the in the software.

273
00:22:19,142 --> 00:22:19,493
Yeah.

274
00:22:19,493 --> 00:22:22,546
It doesn't it doesn't only happen at a checkpoint.

275
00:22:22,546 --> 00:22:33,255
It doesn't happen only w on replication and just randomly, seldomly, deletes old uh some
some data file just by mistake.

276
00:22:33,255 --> 00:22:35,457
Just it's just very, very painful.

277
00:22:35,457 --> 00:22:36,378
That's

278
00:22:36,632 --> 00:22:47,525
I read all these posts today that for one reason or another that we're not going to get
into, decide that going to an on-prem solution is the right thing to do off the cloud.

279
00:22:47,525 --> 00:22:50,346
And we we can get into this um in in this episode.

280
00:22:50,346 --> 00:23:00,459
But what I wanna what I wanna point out is that they often say, yeah, the capital the
capitalization or the capital expenditures rather than the opex for buying data center

281
00:23:00,459 --> 00:23:04,189
resources is cheaper uh in the long run if you know what you're doing.

282
00:23:04,190 --> 00:23:05,782
And I question

283
00:23:05,782 --> 00:23:06,713
Do you know what you're doing?

284
00:23:06,713 --> 00:23:17,322
Because are you prepared to deal with it's not just hard drives failing, but neutrinos
flying through the air and flipping bits in your non-ECC RAM uh or on your hard drive for

285
00:23:17,322 --> 00:23:18,713
your database cluster?

286
00:23:18,713 --> 00:23:20,334
And these things will happen.

287
00:23:20,334 --> 00:23:23,617
And you probably aren't designing software with that in mind.

288
00:23:23,617 --> 00:23:27,190
Like uh most people that are doing this haven't been running data centers.

289
00:23:27,190 --> 00:23:31,544
Like it's one thing if you're in a cloud provider and you're like, you know what, I'm
gonna quit.

290
00:23:31,544 --> 00:23:34,386
I'm gonna start my own company that's its own data center.

291
00:23:34,486 --> 00:23:42,755
data center as a service in a particular region and I have the experiences of building and
running that not just the hardware part but the interface layer, the software.

292
00:23:42,755 --> 00:23:44,057
So I know what to expect.

293
00:23:44,057 --> 00:23:47,360
But most companies and most people who work with them, they for sure don't.

294
00:23:47,360 --> 00:23:53,280
So what I will ask is are you running in a cloud or is all your technology on prem
somewhere?

295
00:23:53,280 --> 00:23:55,272
No, we're we're running on cloud on cloud.

296
00:23:55,272 --> 00:23:55,592
Yeah.

297
00:23:55,592 --> 00:24:00,266
So even wh wh when we're running on cloud, we also have the on prem version for our
customers to use.

298
00:24:00,266 --> 00:24:12,547
But it's still kind of I do feel like the on prem is kind of the safer well, the cloud is
the safer solution for them because no, when they actually see a problem like the system

299
00:24:12,547 --> 00:24:19,610
magically deletes a a data file, I'm I'm there to help them for the on on the on the on
the on on the cloud.

300
00:24:19,610 --> 00:24:21,218
I I went through like

301
00:24:21,218 --> 00:24:29,402
two three almost sleepless days to trying to salvage their data, but they don't even have
the ability to do that if they wanted to.

302
00:24:29,402 --> 00:24:35,845
So it's it's still nice to have someone to help help you out help you out with that.

303
00:24:35,845 --> 00:24:43,358
And and for infrastructure we are also experienced a and some pretty incredible failures
there.

304
00:24:43,358 --> 00:24:44,919
So yeah.

305
00:24:44,919 --> 00:24:47,484
On each layer there are just some difficulties.

306
00:24:47,484 --> 00:24:47,970
Yeah.

307
00:24:47,970 --> 00:24:56,220
Uh, who wouldn't want their the the CEO of their vendor being on call to respond to every
critical incident that they're you know, you know that there's a problem that you're

308
00:24:56,220 --> 00:24:59,353
waking up an executive to go debug that issue for you.

309
00:24:59,353 --> 00:25:01,474
Like it that that's just another level.

310
00:25:01,474 --> 00:25:10,632
Yeah, but it just I I really can't go to sleep knowing that that they're d they have data
corruption and and and they can't access their their their data.

311
00:25:10,632 --> 00:25:14,646
It just just uh just it it's not right.

312
00:25:14,646 --> 00:25:15,987
I I know, I I totally understand.

313
00:25:15,987 --> 00:25:17,358
I I can totally commiserate with you.

314
00:25:17,358 --> 00:25:19,520
Uh well I may not be able to do anything to help.

315
00:25:19,520 --> 00:25:28,577
I feel like I have some responsibility to be awake and field um support calls or even
translate or communicate with that customer with what's actually happening at that moment.

316
00:25:28,577 --> 00:25:36,294
Because I and I know some people that are more better at this than me, who've like calm
composure, do not panic or anything like that.

317
00:25:36,294 --> 00:25:40,727
But honestly I'm just like, God, like we need to have a solution for this when it happens.

318
00:25:40,727 --> 00:25:43,429
And it's not it's not a fun time for for sure.

319
00:25:43,429 --> 00:25:44,270
You

320
00:25:44,362 --> 00:25:47,703
mentioned that you're running on the cloud, it's easier to bug.

321
00:25:47,703 --> 00:25:56,666
And I'm totally with you because you just have access to a bunch of tools where you can
extract data, store it, or even investigate really in a strategic way.

322
00:25:56,666 --> 00:26:02,468
Whereas on-prem stuff you have to be asking, well, are they running it on some weird
on-prem cluster?

323
00:26:02,468 --> 00:26:03,898
What are they even using for a cluster?

324
00:26:03,898 --> 00:26:07,709
How about their hardware or connection, network cables, et cetera?

325
00:26:07,709 --> 00:26:10,742
Like maybe there's some RJ forty five jack that's

326
00:26:10,742 --> 00:26:18,585
misbehaving or power fluctuations that's causing not enough power to be delivered to the
hard drive are just incredible things to or challenging to debug.

327
00:26:18,585 --> 00:26:24,768
I remember previously in one company I was in, there was an issue in one of the data
centers with the Wi-Fi signal.

328
00:26:24,768 --> 00:26:34,051
And people would be moving from one area to another one and that would cause an issue with
some of the data which wasn't being validated correctly because they were using like YAML

329
00:26:34,051 --> 00:26:40,734
and YAML you don't know when the end of the stream is all the time because it's not
encoded there's no end encoding.

330
00:26:40,846 --> 00:26:43,108
so it can be dangerous, you know, if something gets cut off.

331
00:26:43,108 --> 00:26:49,963
And uh yeah, there were for sure some issues that are very difficult to debug when they're
not in a data center that you can control.

332
00:26:49,963 --> 00:26:54,676
Uh so I I totally understand the aspect of running on on-prem though.

333
00:26:54,676 --> 00:27:03,052
You know, there are some customers that absolutely have certain concerns, whether they're
uh security related or usually regulatory reasons for for doing that.

334
00:27:03,052 --> 00:27:04,933
Are you building the same product basically?

335
00:27:04,933 --> 00:27:09,559
Like are you just shipping them what you're running in the cloud or are there fundamental
differences between

336
00:27:09,559 --> 00:27:10,872
These two versions.

337
00:27:10,872 --> 00:27:13,253
They're they're off the same code trunk.

338
00:27:13,253 --> 00:27:15,444
So they're basically the same thing.

339
00:27:15,645 --> 00:27:15,955
Yeah.

340
00:27:15,955 --> 00:27:21,868
And we try to keep everything with one main main version so that it's by configuration.

341
00:27:21,868 --> 00:27:24,820
It behaves a little bit differently, but by configuration.

342
00:27:25,294 --> 00:27:31,726
Do you have challenges trying like uh there are some scenarios where you may want to
optimize because of the cloud provider?

343
00:27:31,726 --> 00:27:35,317
Can I can I ask you using like AWS or GCP or something specific to Taiwan?

344
00:27:35,317 --> 00:27:38,198
Both okay, cool.

345
00:27:38,198 --> 00:27:42,199
So I I mean well, you know, we're gonna hear that.

346
00:27:42,199 --> 00:27:50,112
I I feel like there's a thing if you're just using one cloud provider, there's an
opportunity for optimizations utilizing the primitives that are available from that cloud

347
00:27:50,112 --> 00:27:50,754
provider.

348
00:27:50,754 --> 00:27:53,857
But as soon as you go to two, you start losing that capability.

349
00:27:53,857 --> 00:27:55,749
So you're not able to make those same changes.

350
00:27:55,749 --> 00:27:57,661
So it's a real trade-off to go multi-cloud.

351
00:27:57,661 --> 00:28:03,486
So I guess what I'll ask you is what what went wrong that caused you to have to go down
this path?

352
00:28:04,192 --> 00:28:04,813
Okay, yeah.

353
00:28:04,813 --> 00:28:10,688
But ear early in the days in in Ragik we were actually hosted on a a service called
Linode.

354
00:28:10,688 --> 00:28:11,738
I think they're still around.

355
00:28:11,738 --> 00:28:12,289
Yeah.

356
00:28:12,289 --> 00:28:14,831
And it it was hosted on Linode.

357
00:28:14,951 --> 00:28:20,135
So it was year I think it's around two thousand and fourteen.

358
00:28:20,436 --> 00:28:31,525
There's uh if you if it if you Google it, there there's there around Christmas time, there
was a DDoS attack a a whole series of DDOS attack around Christmas time, two thousand and

359
00:28:31,525 --> 00:28:33,006
f I think fourteen.

360
00:28:33,108 --> 00:28:36,260
And the basically the whole data center went down.

361
00:28:36,260 --> 00:28:40,103
So we are not able to have any access to any of our data.

362
00:28:40,103 --> 00:28:42,205
So that I was on vacation.

363
00:28:42,205 --> 00:28:45,927
So that was just yeah, that was just hell.

364
00:28:46,088 --> 00:28:57,436
And yeah, I was trying to calm our calm our customers down and trying to trying to tell
tell them because we we still have so little visibility from uh Dino because Dino say,

365
00:28:57,436 --> 00:29:00,152
we're under d h very, very, very heavy

366
00:29:00,152 --> 00:29:04,984
D DOS attacks, so we're doing everything we can, but they can't cannot give us any
visibility.

367
00:29:04,984 --> 00:29:12,467
So during that weekend, because it it's not like a matter of hours, it's kind of like a
couple of days.

368
00:29:12,467 --> 00:29:15,819
I think it's it went down for like almost like twenty-four hours.

369
00:29:15,819 --> 00:29:16,385
Oh wow.

370
00:29:16,385 --> 00:29:21,391
And we Yeah, and we we just thought that yeah, we really, really can't take this anymore.

371
00:29:21,391 --> 00:29:26,103
So because they're kind of uh remedying on and off.

372
00:29:26,103 --> 00:29:28,914
So we kind of start to get

373
00:29:28,950 --> 00:29:37,370
about like thirty minutes of time that we can access the the database the the the servers
and we then we get an hour and it's down again.

374
00:29:37,370 --> 00:29:40,032
So I so I just decided, yeah, that's it.

375
00:29:40,032 --> 00:29:41,575
We we have to get out of there.

376
00:29:41,575 --> 00:29:43,356
We have to move out of there.

377
00:29:43,356 --> 00:29:52,120
So we kind of decided and migrated all our services from Lino to GCP over one weekend.

378
00:29:52,120 --> 00:29:53,890
So over that that weekend.

379
00:29:53,890 --> 00:29:55,481
So I was on vacation.

380
00:29:55,481 --> 00:29:57,102
So I was not even in the office.

381
00:29:57,102 --> 00:29:57,816
So I was

382
00:29:57,816 --> 00:29:59,176
kind of like in a museum.

383
00:29:59,176 --> 00:30:08,149
I was sitting there with trying to use the museum Wi Fi and then my kids were just like
off off like in visiting in the museum.

384
00:30:08,149 --> 00:30:14,931
I was just in the in in in the the lobby and trying to m move everything from Linux to G C
P.

385
00:30:15,031 --> 00:30:20,702
And gladly or kind of nice that we don't have that much data back then.

386
00:30:20,702 --> 00:30:23,793
So just the like a couple of servers that we need to move.

387
00:30:23,793 --> 00:30:27,394
So we kind of just moved everything to G C P

388
00:30:27,626 --> 00:30:29,127
over one weekend.

389
00:30:29,127 --> 00:30:40,276
And and later on we we just feels like we we need the flexibility to switch services
whenever there's uh we don't want to be buying to one vendor.

390
00:30:40,276 --> 00:30:41,376
That's just too dangerous.

391
00:30:41,376 --> 00:30:44,329
We want to have the ability to move anywhere.

392
00:30:44,329 --> 00:30:52,805
We just I just told myself we have to get we have to have we we're not going to be buying
to one single uh one single vendor.

393
00:30:52,805 --> 00:30:54,410
And yeah I know that

394
00:30:54,410 --> 00:31:01,963
No, most people would use leverage tools on these like AWS and G C P to scale their
services, to automate things.

395
00:31:01,963 --> 00:31:10,137
But basically we just roll out our own and service instance management with custom code.

396
00:31:10,137 --> 00:31:19,370
We just write custom code and they can even like SSH to different instances and t and to
do the routine management and to to do whatever we want.

397
00:31:19,370 --> 00:31:23,362
So we we basically just write up our own

398
00:31:23,379 --> 00:31:27,692
um service management system with no connecting by SSH.

399
00:31:27,692 --> 00:31:32,662
I mean, once you've suffered those those particular traumas, there's there's no going back
at that point.

400
00:31:32,662 --> 00:31:37,395
Yeah, yeah, that yeah, that was very, very very stressful.

401
00:31:37,666 --> 00:31:39,726
That's really early days for G C P too.

402
00:31:39,726 --> 00:31:41,579
I I didn't even know they were out at that point.

403
00:31:41,579 --> 00:31:50,737
And so like that must have been a huge risk even to decide like, okay, not only are we
going to go to a cloud provider, but we don't have a lot of options and they haven't been

404
00:31:50,737 --> 00:31:51,828
around that long.

405
00:31:51,828 --> 00:31:54,039
you know, are we ready to even trust them with that regard?

406
00:31:54,039 --> 00:31:57,282
But it was enough for you to actually even make that switch.

407
00:31:57,343 --> 00:32:06,330
I I know I mean Lino was definitely is focused more on the VMware the VM side of the house
and uh definitely a huge challenge to get even

408
00:32:06,388 --> 00:32:14,767
more reliability if you want, say, like a database in in any particular m way which has
backup or um reliability set up.

409
00:32:14,767 --> 00:32:18,641
Are you using those primitives in the cloud providers though?

410
00:32:18,641 --> 00:32:20,443
Or are you managing the data?

411
00:32:20,443 --> 00:32:24,737
I mean, you're running your own database that isn't offered as a primary option.

412
00:32:24,737 --> 00:32:28,802
Like there is no Berkeley D B offered by G C P or AWS today.

413
00:32:28,802 --> 00:32:29,676
If they st

414
00:32:29,676 --> 00:32:37,416
tomorrow stood up and say, Hey, you know, we're gonna offer to manage version of that,
would you switch to that or would you still run it within the the service management mesh

415
00:32:37,416 --> 00:32:41,794
that you've you've created for the um that particular provider?

416
00:32:41,794 --> 00:32:50,161
I don't think it's possible at with today's technical architecture because the database
runs within the same process.

417
00:32:50,161 --> 00:32:56,266
I was th I would I would think that anything outside of the process would be a lot slower.

418
00:32:56,266 --> 00:33:01,080
So we don't r even really put database on a different server.

419
00:33:01,080 --> 00:33:11,642
They just run on the same process and they use a shared and they usually use a shared
memory pool inside the JVM to to for multi-tenancy.

420
00:33:11,642 --> 00:33:19,179
So yeah, it's it's kind of different from the other database architectures because we're
based on Berkeley DB.

421
00:33:19,179 --> 00:33:20,450
There are ad advantages.

422
00:33:20,450 --> 00:33:31,130
There are actually quite a few advantages of using that because when we're building
multi-tenant application with Berkeley DB, and it's actually a very nice tech

423
00:33:31,130 --> 00:33:33,858
infrastructure to build multi-tenant applications.

424
00:33:33,858 --> 00:33:41,091
They have shared um memory pools for all these d different database instances and the
database synthesis are physically separated.

425
00:33:41,091 --> 00:33:49,015
They're not just logically separate, they're you know physically separated so that makes
it easy to do hot di hot backup and restore.

426
00:33:49,015 --> 00:33:51,148
So it's just quite suitable.

427
00:33:51,148 --> 00:34:00,380
Yeah, I I suppose the other perspective is that the needs of something that stores data
could be fundamentally different from the um I.O.

428
00:34:00,380 --> 00:34:08,383
or HTTP required ports or sockets that are available uh for a application server.

429
00:34:08,383 --> 00:34:13,534
And so com combining those needs falls into one of two cap like buckets.

430
00:34:13,534 --> 00:34:17,686
The first one is we perfectly match all the resources available to the VM.

431
00:34:17,686 --> 00:34:21,789
Some part of the resources are dedicated to the application and the other part is
dedicated to the database.

432
00:34:21,789 --> 00:34:25,732
However, the alternative strategy, and so we have like full utilization.

433
00:34:25,732 --> 00:34:30,096
The competing argument is that this the way in which they scale is different.

434
00:34:30,096 --> 00:34:37,252
When we some queries or requests require more scaling on the database side or the
database, the resources dedicated for the database.

435
00:34:37,252 --> 00:34:41,666
And so separating them is valuable because it's not a one-to-one match.

436
00:34:41,666 --> 00:34:46,840
More requests or more complicated requests may not translate to exactly the same thing in
the database.

437
00:34:46,840 --> 00:34:51,877
Have you found that you're at a sweet spot where the utilization is still incredibly high?

438
00:34:51,877 --> 00:35:03,466
Or uh are you sort of wasting some uh capacity or monetary value by having it on one
machine, but making it up with the the simplicity of having it all in one place?

439
00:35:03,466 --> 00:35:15,117
I do feel that having it in the same machine kind of improves the utilization because, you
know, when you spread it out, it's more likely to have unused um resources.

440
00:35:15,117 --> 00:35:22,824
So so basically for the application and the database, they're using the the same memory in
the same JVM.

441
00:35:22,824 --> 00:35:29,986
So w we can actually configure the percentage of um memory to be allocated to the database
or

442
00:35:29,986 --> 00:35:31,807
to be allocated to the application.

443
00:35:31,807 --> 00:35:35,488
That's something we can configure or even change dynamically.

444
00:35:35,488 --> 00:35:44,612
So that's a a a pretty good thing because we can decide how much memory I want to allocate
to each database instance when it starts because it's multi-tenant.

445
00:35:44,612 --> 00:35:54,496
So we can kind of determine, we look at how large the data set is and we look at and how
much usage they have, how many users they have, and we can decide like how many how much

446
00:35:54,496 --> 00:35:58,966
memory that we want to allocate to them and according to the current memory use.

447
00:35:58,966 --> 00:36:10,523
So I think it actually gives us a lot of flexibility to move the resources between and the
database that what would usually be the database server and the application server, that

448
00:36:10,523 --> 00:36:12,715
part is actually pretty nice.

449
00:36:12,715 --> 00:36:18,350
So we we we are able to easily shift the resources between the D B and the AP.

450
00:36:18,350 --> 00:36:26,938
I mean, if you're already in the JVM, have you thought about ever taking the next level
and shifting to having your architecture be specifically on a like a container native

451
00:36:26,938 --> 00:36:28,044
platform instead?

452
00:36:28,044 --> 00:36:32,516
Yeah, they they they have been deployed on Dockers and and containers.

453
00:36:32,516 --> 00:36:32,876
Yeah.

454
00:36:32,876 --> 00:36:37,268
Because for sometimes for our on prem users they they do deploy on Docker.

455
00:36:37,268 --> 00:36:48,142
But for us, because the basically the whole installation is like really, really simple
because and for the application server we use Jetty, which we embed in there.

456
00:36:48,142 --> 00:36:51,944
And for database we use um Berkeley D B which also embedded.

457
00:36:51,944 --> 00:36:57,086
So basically when to set up a server you just install Java and you just

458
00:36:57,086 --> 00:36:59,598
run Ragic and be done with it.

459
00:36:59,598 --> 00:37:04,702
So it it's just not a lot of work for not a lot of benefit for Docker.

460
00:37:04,702 --> 00:37:12,008
But but it but for some some of our on prem users they they like to they always use Docker
so they could still still do that.

461
00:37:12,140 --> 00:37:20,199
One thing that always comes up as a question for me in scenarios where people are
basically running uh all the infrastructure themselves on a virtual machine or even bare

462
00:37:20,199 --> 00:37:23,184
metal is how do you scale testing effectively?

463
00:37:23,184 --> 00:37:28,110
Because from my standpoint, it's always been if you have containers, it's easier to

464
00:37:28,158 --> 00:37:36,595
deploy them, uh, make some configuration changes, understand memory utilization, CPU
utilization, et cetera, and actually watch where they fail, potentially if you're doing

465
00:37:36,595 --> 00:37:37,606
some sort of load test.

466
00:37:37,606 --> 00:37:41,189
Do you have different deployment modes to be able to capture that well?

467
00:37:41,189 --> 00:37:47,532
Or have you found particular sweet spots in how you actually do load testing in in the
model that you're running today?

468
00:37:47,532 --> 00:37:52,105
I don't think there's a lot of special things that we're doing with these load testing.

469
00:37:52,105 --> 00:38:05,514
Yeah, we we just have some test test instances and test servers on G C P and AWS and also
we we also have some local machines in our office that we can use to to do these low

470
00:38:05,514 --> 00:38:06,055
testing.

471
00:38:06,055 --> 00:38:11,499
So yeah, we there're just some we don't really scale those tests.

472
00:38:11,499 --> 00:38:16,822
So yeah, we just do some because they're they're basically um a monolith, so

473
00:38:17,152 --> 00:38:19,880
It's kind of easier to just test it.

474
00:38:19,926 --> 00:38:20,386
Yeah.

475
00:38:20,386 --> 00:38:21,907
No, I I I totally get it.

476
00:38:21,907 --> 00:38:27,089
One thing you did mention, uh you hinted at a little bit was your multi-tenancy model.

477
00:38:27,089 --> 00:38:32,271
And so I am curious about your product, especially if you have on-prem or customers that
are deploying it themselves.

478
00:38:32,271 --> 00:38:42,435
And once you have that and if you have the same basically source running in for those
models as well as in your own, say, cloud environment, then a rather than a multi-tenancy

479
00:38:42,435 --> 00:38:47,338
model, I I found a lot of companies just deploy a different set of instances for every
single customer.

480
00:38:47,338 --> 00:38:48,578
Are you doing that or

481
00:38:48,578 --> 00:38:51,040
Did you mean something different by you're doing something different than that?

482
00:38:51,040 --> 00:38:51,530
Yeah, yeah.

483
00:38:51,530 --> 00:38:58,415
That for one instance we we are running like thousand thousands of tenants on each
instance.

484
00:38:58,416 --> 00:39:08,002
So that that's one of the the good things that I talked that I talked about, like they had
they can use a shared memory pool on Berkeley D B and say we're running thousands of

485
00:39:08,703 --> 00:39:13,187
thousand thousands of in um customers on on the same server.

486
00:39:13,187 --> 00:39:18,008
And that's of of course that's still kind of a a technical challenge that

487
00:39:18,008 --> 00:39:23,663
has caused some problems because like n the the biggest problem would be like the noisy
neighbor problems.

488
00:39:23,663 --> 00:39:32,440
For sure some some accounts they are taking up a lot of resources or than then the others
affecting the service quality of other accounts.

489
00:39:32,440 --> 00:39:45,861
So there we we've learned from experience and and added a lot of those quotas and limits
to to to everything that might take up extra resources and might you know starve other

490
00:39:45,861 --> 00:39:47,168
other neighbor

491
00:39:47,168 --> 00:39:49,519
neighbor accounts, neighbor databases.

492
00:39:49,559 --> 00:40:02,324
And another thing that we do a lot, more and more based on these, is that we try to do
most of the processing using an asynchronous queue, like a blocking queue.

493
00:40:02,324 --> 00:40:11,028
So that, you know, when certain account is trying to execute a hundred or a thousand thing
at the same time, we just try to put that in the queue.

494
00:40:11,028 --> 00:40:14,129
So that it's not going to hog up all the CPU cores.

495
00:40:14,129 --> 00:40:15,904
It's not going to hog up all the

496
00:40:15,904 --> 00:40:16,755
all the memories.

497
00:40:16,755 --> 00:40:28,388
So we will on these kind of multi tenant environments, we have to for everything that
might take too much resource, you probably has to put it in an asynchronous asynchronous

498
00:40:28,388 --> 00:40:32,211
queue so that it's not gonna all fire at the same time.

499
00:40:32,211 --> 00:40:37,777
So that way I didn't start that way.

500
00:40:38,046 --> 00:40:39,608
Yeah, you didn't start that way.

501
00:40:39,911 --> 00:40:43,648
what was the straw that broke the camel's back, so to say?

502
00:40:43,858 --> 00:40:54,041
Oh, I I can't well, to to say I can't even remember, but that back broke long ago because
you hit that problem pr pr pretty pretty early on.

503
00:40:54,041 --> 00:41:00,343
Especially in Ragik's model is that people can build anything on Ragic, any type of
therapy supplication.

504
00:41:00,343 --> 00:41:06,144
So you're we see all sorts of weird use of Ragik pretty early on.

505
00:41:06,144 --> 00:41:11,660
So so in the beginning we we we hit that problem quite quite long ago and and

506
00:41:11,660 --> 00:41:16,603
Yeah, so I started kind of writing my own blocking queue to to to try to to fix this.

507
00:41:16,603 --> 00:41:21,537
And w we had we had our first blocking queue to for for this purpose quite early on.

508
00:41:21,537 --> 00:41:26,651
But but of course a lot of things didn't live on the queue in the beginning.

509
00:41:26,651 --> 00:41:32,585
And then we see people starting to misuse it or accidentally use too much resource.

510
00:41:32,585 --> 00:41:36,168
It was just oh, we need to use the the blocking queue for this too.

511
00:41:36,168 --> 00:41:37,469
So yeah.

512
00:41:37,469 --> 00:41:39,754
We just slowly begin to move

513
00:41:39,754 --> 00:41:43,738
Almost everything that takes resources to the to the queue.

514
00:41:43,738 --> 00:41:44,448
I mean it makes sense.

515
00:41:44,448 --> 00:41:45,369
It's certainly one of the things.

516
00:41:45,369 --> 00:41:54,155
Uh early on when we were designing our solution, we knew that this was gonna be a problem
in some regard and you have to come to the come to the floor, basically.

517
00:41:54,155 --> 00:42:00,700
come to the table already believing that there's something that you're gonna like you have
to have the tools in front of you in order to be able to do something about it.

518
00:42:00,700 --> 00:42:08,096
You don't necessarily need to implement it at the beginning, um, or you need to have a
database that exposes hooks that allows you to build your own technology on top of it to r

519
00:42:08,096 --> 00:42:11,998
relieve uh back pressure uh or, you know, prevent

520
00:42:11,998 --> 00:42:15,440
uh abusing the the database or resources that are available.

521
00:42:15,440 --> 00:42:21,675
There's actually a whole episode where we went into some of the differences between the
single tenant architecture and and multi-tenant architecture.

522
00:42:21,675 --> 00:42:24,116
So that episode will be in the in the link in the description.

523
00:42:24,116 --> 00:42:27,369
So maybe we won't focus too much on that for for now.

524
00:42:27,369 --> 00:42:38,016
But one of the questions that I sort of want to get into is I think na in the last few
years we see a lot of people standing up and saying, oh, I rebuilt MS Excel in one weekend

525
00:42:38,016 --> 00:42:38,736
and it's great.

526
00:42:38,736 --> 00:42:41,518
Or I rebuilt Airtable in one weekend.

527
00:42:41,518 --> 00:42:48,683
Uh, and me personally, I'm rebuilding uh Gmail as a product right now and I can tell you
it's taking a lot more than one week yet.

528
00:42:48,824 --> 00:42:58,212
How is the like current ecosystem and the product climate fundamentally impacting your
organization or like the technical challenges you're you're seeing today versus the ones

529
00:42:58,212 --> 00:43:01,144
that you would have seen in say the last fifteen years prior?

530
00:43:01,144 --> 00:43:01,674
Yeah.

531
00:43:01,674 --> 00:43:08,779
Who what you're saying those those I built something in in a in the weekend because I was
really curious about that.

532
00:43:08,779 --> 00:43:11,922
So let me address that in in like a few months ago.

533
00:43:11,922 --> 00:43:13,703
So I was like, wow, so really?

534
00:43:13,703 --> 00:43:16,165
No no coding experience you can build that over a weekend?

535
00:43:16,165 --> 00:43:17,626
So so I have two kids.

536
00:43:17,626 --> 00:43:19,787
They are um twelve and fourteen.

537
00:43:19,787 --> 00:43:23,730
So they they always ask me to to help to teach them coding.

538
00:43:23,730 --> 00:43:29,750
So okay, so now we have th those L O So I'm gonna teach you how to not code and that's
build a game.

539
00:43:29,750 --> 00:43:35,735
We went up on a small project to build a uh an RPG game, like sort of like a Dragon Quest.

540
00:43:35,735 --> 00:43:39,477
So I I decided I I I I'm not gonna do any coding.

541
00:43:39,477 --> 00:43:42,279
I'm not gonna not gonna even look at the code.

542
00:43:42,279 --> 00:43:44,901
So so we begin trying to create the RPG game.

543
00:43:44,901 --> 00:43:50,215
Like in half an hour, there's a working RPG game that we can play around.

544
00:43:50,215 --> 00:43:51,826
It's just amazing.

545
00:43:51,826 --> 00:43:57,066
And I thought, oh, this is why people say, I build this over the weekend, I build this
over

546
00:43:57,066 --> 00:43:59,137
o just over like like three hours.

547
00:43:59,137 --> 00:44:01,828
So that's kind of a magical feeling.

548
00:44:01,969 --> 00:44:14,475
But the next day and the next week, because the code gets bigger and bigger and just
there's just a lot of problems that begins to to arise because it just takes longer and

549
00:44:14,475 --> 00:44:22,220
longer for for AI to kind of fix a problem and it and often says that it fixed a problem
and it doesn't really fix the problem.

550
00:44:22,220 --> 00:44:26,786
So I begin to look into the code set and it just really

551
00:44:26,786 --> 00:44:30,568
bad spaghetti code and with some really bad designs.

552
00:44:30,568 --> 00:44:35,430
And in the beginning because I I never told it told him what kind of design we want.

553
00:44:35,430 --> 00:44:38,052
I just have my kids telling me I want this, I want that.

554
00:44:38,052 --> 00:44:40,673
So the design is like really bad.

555
00:44:40,673 --> 00:44:43,654
But the amazing thing is it that the game still runs.

556
00:44:43,822 --> 00:44:49,106
So I want to ask is I feel like that is often the response in these areas.

557
00:44:49,106 --> 00:44:54,800
And the feedback that I get in those moments are, well, Warren, it's a skill issue.

558
00:44:54,800 --> 00:44:59,143
If you were better at prompting the LLM, then you wouldn't have spaghetti code in the
first place.

559
00:44:59,143 --> 00:45:01,714
And I don't I don't agree.

560
00:45:01,722 --> 00:45:07,759
I have tried I have tried so hard to make it so it the code can be maintainable at a long
period of time.

561
00:45:07,759 --> 00:45:09,960
And I just I haven't found that to be the case.

562
00:45:09,960 --> 00:45:12,522
And I I'm sure with your experience that

563
00:45:12,522 --> 00:45:18,572
Yeah, you weren't I mean, there's the I'm not looking at it, but in the on the same time
I'm like, I tried to get it to fix it, right?

564
00:45:18,572 --> 00:45:31,661
So I I this this is a fun side project I do with my kids, but at at work I find the the
LLM never been able to do the uh system design correctly.

565
00:45:31,661 --> 00:45:35,694
Um it's probably because our product is kind of different.

566
00:45:35,694 --> 00:45:48,302
Because if you're building like an e-commerce site or like ERP system, I would tend to
believe that they can build something pretty standard and pretty much right.

567
00:45:48,578 --> 00:45:56,740
But I don't think that it has enough background understanding on how to build something
that I want in the database builder.

568
00:45:56,740 --> 00:46:00,551
So it it it knows the concept, but the design will be really bad.

569
00:46:00,551 --> 00:46:12,865
So every day when my working I I still work with LM to help with the all those design all
the time, but just that I have to I I always have to tell them, Oh, this is wrong because

570
00:46:12,865 --> 00:46:16,078
of this and this that so you have to change that and

571
00:46:16,078 --> 00:46:16,518
I don't know.

572
00:46:16,518 --> 00:46:22,444
I I still work with them, although I end up with coming up with all the designs myself.

573
00:46:22,444 --> 00:46:27,138
But you know, talking to them helps me helps me organize the thoughts, I guess.

574
00:46:27,138 --> 00:46:33,094
I and and they can write up all these documentation about our design plans really quickly,
so that's that's also a plus.

575
00:46:33,094 --> 00:46:36,357
But in the end I would look at a design.

576
00:46:36,357 --> 00:46:38,499
They're still made by me.

577
00:46:38,499 --> 00:46:42,082
Their design is usually pretty off.

578
00:46:42,412 --> 00:46:44,735
Yeah, no, that's that's sort of what I've experienced.

579
00:46:44,735 --> 00:46:49,771
And I I've been trying to figure out if different n tools or different harnesses change
the approach here.

580
00:46:49,771 --> 00:46:58,201
But I think the fallacy is sort of the same one that you've run into, which is in my area,
whatever my area is in quotes, it can't be used.

581
00:46:58,201 --> 00:47:02,848
But if you look at other areas, you know, it's no problem for it to be used there.

582
00:47:02,848 --> 00:47:08,860
And I think that's where like there's a devil in the details where like if you actually go
and try to implement an ERP system, I will tell you.

583
00:47:08,860 --> 00:47:13,338
I will tell you how complicated it is and how much the LLM will get it wrong.

584
00:47:13,338 --> 00:47:15,807
Uh and I think that's sort of yeah.

585
00:47:15,807 --> 00:47:24,914
And but I think this is the thing where w the area where we're an expert in feels like it
can't work or does not work very well without a lot of micromanagement.

586
00:47:24,914 --> 00:47:30,594
And when we push it to other areas that are outside of our expertise, it feels like it
does a much better job.

587
00:47:30,594 --> 00:47:35,456
But when you ask experts from those areas, they give you the opposite feedback.

588
00:47:35,656 --> 00:47:38,527
building a spreadsheet tool, I can do that with an LLM.

589
00:47:38,527 --> 00:47:46,500
Matter of fact, I just saw 10 products on uh whatever a product hunt yesterday that were
all you know selling the air table replacement.

590
00:47:46,800 --> 00:47:48,651
so you know that's an interesting perspective.

591
00:47:48,651 --> 00:47:56,585
But with that, maybe this would be a good moment before going further down a tangent to
switch over to picks for the episode.

592
00:47:56,585 --> 00:48:00,406
So so Jeff, what did you bring for the audience today?

593
00:48:00,672 --> 00:48:01,472
Okay, yeah.

594
00:48:01,472 --> 00:48:15,806
It's one of uh I s I saw on your uh intro you could be one of the adventures and and
actually in about two two weeks I'm I'm going on the back backpacking trekking trip to the

595
00:48:15,806 --> 00:48:18,007
the the mountains in in Taiwan.

596
00:48:18,007 --> 00:48:22,648
It's called the and one of the mountains is called um the Nanghu Mountain.

597
00:48:22,648 --> 00:48:30,584
So yeah, the and Taiwan is um is where I I live or where I'm I'm from and Taiwan is like a
a uh

598
00:48:30,584 --> 00:48:34,055
kind of a small island, but it has a lot of mountains.

599
00:48:34,055 --> 00:48:38,096
So like sixty percent of Taiwan is mountainous areas.

600
00:48:38,096 --> 00:48:44,198
And and I've been to like the Rockies, I've been to Alaska, Switzerland to see the
Alpines.

601
00:48:44,198 --> 00:48:48,139
And the mountains in Taiwan are just just very different.

602
00:48:48,139 --> 00:48:51,199
The sceneries are very different but equally beautiful.

603
00:48:51,240 --> 00:49:00,102
And and I'm just really excited about the backpacking trip that I'm embarking on in a
couple weeks with my college friends and

604
00:49:00,126 --> 00:49:10,997
And I would just love to bring up that, you know, if you're if you love hiking, you f if
you love trekking, backpacking and you love, you know, mountaineering, Taiwan is actually

605
00:49:10,997 --> 00:49:19,520
a really, really nice place to to visit because there are just a lot of amazing and
amazing sceneries in the mountain areas.

606
00:49:19,520 --> 00:49:24,798
Is there one particular peak or or trail that you would recommend above all the other
ones?

607
00:49:24,798 --> 00:49:32,679
One of the most popular thing popular area is the the Taruko, the the Taruko National
Park.

608
00:49:32,679 --> 00:49:39,342
So it's it's sort of it it's it's sort of like the Yosemite, but in a different flavor.

609
00:49:39,342 --> 00:49:47,264
So you can kind of expect um kind of the the same valleys but in the but in the Taiwanese
flavor.

610
00:49:47,264 --> 00:49:49,175
So it's it's a really, really nice place.

611
00:49:49,175 --> 00:49:53,366
There are a lot of uh hiking trails there and that you can take.

612
00:49:53,366 --> 00:49:54,286
So

613
00:49:54,316 --> 00:49:56,737
No, it's it's it's a very, very nice place.

614
00:49:56,737 --> 00:50:00,204
There are a lot of uh mountains you can you can climb there.

615
00:50:00,204 --> 00:50:04,987
Wow, it's like compared it to my two favorite places, Switzerland and Yosemite National
Park.

616
00:50:04,987 --> 00:50:08,068
Uh that's that that's a hard hard sell there.

617
00:50:08,068 --> 00:50:18,514
I'm gonna maybe now have to take the transit out there to uh see the the mountain range,
which is just really interesting because I I just finished um the jet lag, the game season

618
00:50:18,514 --> 00:50:25,688
where they're in Taiwan going around and it's interesting because they're only going
around the edge of the map on the rail and there's no you can't get through the center of

619
00:50:25,688 --> 00:50:29,500
of the country because it's just a whole mountain range, which is

620
00:50:29,710 --> 00:50:31,266
Quite quite inspired.

621
00:50:31,266 --> 00:50:34,254
Yeah, it's it's just real it's just really beloved.

622
00:50:34,254 --> 00:50:34,634
Okay.

623
00:50:34,634 --> 00:50:35,274
Okay.

624
00:50:35,274 --> 00:50:37,115
I just well then.

625
00:50:37,115 --> 00:50:38,275
I love I love the pick.

626
00:50:38,275 --> 00:50:42,486
For me, maybe I mine's a little bit less less inspired.

627
00:50:42,486 --> 00:50:45,127
Uh I'm gonna pick the DevOps Days conferences.

628
00:50:45,127 --> 00:50:52,119
So I just got back from the one in Zurich and honestly, they're the best conferences I've
been to anywhere.

629
00:50:52,119 --> 00:50:57,360
they're non commercial, they're run by volunteers, they're very well done, people very
committed to it.

630
00:50:57,381 --> 00:51:02,998
And just I've been to ones in a a lot of different European countries and

631
00:51:02,998 --> 00:51:05,519
Honestly, if you can go, you absolutely should.

632
00:51:05,519 --> 00:51:14,833
And more importantly, if you can sponsor one of them, I also highly recommend it because
really uh out of all the conferences, they f don't just focus on technical aspects.

633
00:51:14,833 --> 00:51:25,097
They find opportunities for uh cultural improvement and having higher level of
conversations than you normally get at just like a a Java or C sharp or JavaScript

634
00:51:25,097 --> 00:51:33,120
conference and way better than what you get at like an AI conference where you're just
discussing the best new skills to throw into all of your agents.

635
00:51:33,326 --> 00:51:35,107
I don't know if you ask for yeah.

636
00:51:35,107 --> 00:51:41,779
I I don't I don't think I don't know if there is one in Taiwan though, but maybe I'll I'll
look it up after the after this call and see yeah see if if they're offering one.

637
00:51:41,779 --> 00:51:44,621
Basically it's up to volunteers to start and and run it.

638
00:51:44,621 --> 00:51:48,032
There's a global organization, but it's all volunteer based.

639
00:51:48,032 --> 00:51:53,214
And it's one of the few conferences that I go not just as a speaker, but I actually pay
pay to attend.

640
00:51:53,214 --> 00:51:54,145
And I don't know what it is.

641
00:51:54,145 --> 00:52:02,328
I don't know if it's about the the branding or or the mentality or the culture of the or
the organiz the global organization, but individually they just they always seem like

642
00:52:02,328 --> 00:52:02,912
they're

643
00:52:02,912 --> 00:52:10,336
way more in tune with what people actually want and what should go on and they're very
careful about who can sponsor and what sort of talks can show up.

644
00:52:10,336 --> 00:52:10,870
Okay.

645
00:52:10,870 --> 00:52:11,473
Yeah.

646
00:52:11,473 --> 00:52:14,144
Just search if the the there's one in Taiwan.

647
00:52:14,144 --> 00:52:15,255
Yeah.

648
00:52:15,255 --> 00:52:19,950
Um well, uh, thank you, Jeff, for being the guest in in today's episode.

649
00:52:19,950 --> 00:52:27,898
I I didn't know we were where we were going initially, but I absolutely love talking about
all the technical things that a uh executive can still get down and and and do.

650
00:52:27,898 --> 00:52:30,584
So thank you so much for for coming on for the episode.

651
00:52:30,584 --> 00:52:31,507
Thank you, Warren.

652
00:52:31,507 --> 00:52:33,058
It's it's a pleasure talking to you.

653
00:52:33,058 --> 00:52:34,440
Well, it it it's been great.

654
00:52:34,440 --> 00:52:43,134
And uh thanks to the audience for tuning in for this week's Adventures in DevOps and
hopefully we'll see everyone back again next week.