1
00:00:08,119 --> 00:00:10,431
Welcome back to Adventures in DevOps.

2
00:00:10,431 --> 00:00:17,276
It's like the world suddenly woke up to realize that maybe testing is a necessity shocker,
really.

3
00:00:17,276 --> 00:00:20,628
And historically one of the most challenging places to do it is in the browser.

4
00:00:20,628 --> 00:00:23,020
So I'm hoping we can get down to it.

5
00:00:23,020 --> 00:00:30,605
And I brought in a longtime test expert, previous engineering manager at Mozilla, and now
head of open source at Browser Stack, David Burns.

6
00:00:30,605 --> 00:00:31,992
Thank you for joining us.

7
00:00:31,992 --> 00:00:33,136
Thank you for having me, Warren.

8
00:00:33,136 --> 00:00:34,786
I'm really excited to be here.

9
00:00:34,786 --> 00:00:42,431
We are in this space where the things that we're building are being utilized in browsers
as a primary mechanism for interaction.

10
00:00:42,431 --> 00:00:51,156
And yet I always have found in my two decades of software engineering experience, actually
validating what shows up there is always the most difficult part.

11
00:00:51,156 --> 00:00:54,650
Like I write a unit test on some back end code and it's it's a joke.

12
00:00:54,650 --> 00:01:03,776
But as soon as it comes to is the CSS working, is uh the components actually showing at
the right time, is their gestures for mobile stuff actually working correctly, I I mean

13
00:01:03,776 --> 00:01:04,626
good luck.

14
00:01:04,636 --> 00:01:06,447
you are not alone in that.

15
00:01:06,447 --> 00:01:12,288
Having worked on browsers for a decade, uh, even browser engineers have that problem.

16
00:01:12,288 --> 00:01:14,559
Um it it's it's such a common problem.

17
00:01:14,559 --> 00:01:22,441
Browsers nowadays, or at least for the last decade, have effectively been operating
systems in their complexity, right?

18
00:01:22,441 --> 00:01:28,113
Um and you can look at it, Firefox OS, Chrome OS, even WebOS before that, right?

19
00:01:28,113 --> 00:01:31,438
And it was amazing what you could do from rendering

20
00:01:31,438 --> 00:01:44,290
to interactions, um, and then you have all of this on multiple operating systems who do
things slightly differently where again, browsers try and do a bit of massaging to it.

21
00:01:44,290 --> 00:01:48,896
Um and then you throw mobile into it and it's painful.

22
00:01:48,896 --> 00:01:50,797
I mean, I love the promise of frameworks, right?

23
00:01:50,797 --> 00:01:55,868
Because it creates this abstraction layer that potentially you could use to get reliable
results everywhere.

24
00:01:55,868 --> 00:02:04,951
But the whole world of how like CSS implementations are just fundamentally different
between uh the different uh frameworks that power each of the browsers and the the the

25
00:02:04,951 --> 00:02:08,852
classes that are built in or or the flags or whatever that you're utilizing.

26
00:02:08,852 --> 00:02:16,404
And you need something like even if you're using a framework, I'm sure too many people are
unfortunately using React, but if you're using something better, you still have to deal

27
00:02:16,404 --> 00:02:17,454
with this like.

28
00:02:17,464 --> 00:02:25,199
post CSS processing where you take what was actually generated and make it work on
literally ever every browser and it d it still doesn't work out of the box for everything.

29
00:02:25,199 --> 00:02:25,606
So

30
00:02:25,606 --> 00:02:30,549
there's a lot of problems that I think when we're developing software that we really
struggle with.

31
00:02:30,610 --> 00:02:33,132
CSS is a fantastic example, right?

32
00:02:33,132 --> 00:02:38,275
Like the Peter Griffin trying to open the the blind, uh, and you see how that goes wrong.

33
00:02:38,275 --> 00:02:39,356
That is CSS.

34
00:02:39,356 --> 00:02:40,027
I'm like you.

35
00:02:40,027 --> 00:02:41,787
I've done a lot of back-end engineering.

36
00:02:41,787 --> 00:02:44,760
I was never good at the UI side of things.

37
00:02:44,760 --> 00:02:50,064
Um, but I like the kind of automation part of it and trying to get it to do things.

38
00:02:50,064 --> 00:02:52,760
And CSS has kind of grown organically.

39
00:02:52,760 --> 00:02:56,783
from the years and then standards got in the way and they started cleaning it all up.

40
00:02:56,783 --> 00:02:58,645
They had to retrofit a lot of stuff.

41
00:02:58,645 --> 00:03:06,021
Then you've got the other side of it is that when you're interacting with all of these
things, you've got JavaScript uh and that runs asynchronously, right?

42
00:03:06,021 --> 00:03:07,983
And the browser's like, you asked me to do it.

43
00:03:07,983 --> 00:03:09,074
I'll let you know when I'm done.

44
00:03:09,074 --> 00:03:11,756
No, never not not before that.

45
00:03:11,756 --> 00:03:12,497
You're absolutely right.

46
00:03:12,497 --> 00:03:15,529
There's there's a lot there where there could just be problems at every level.

47
00:03:15,529 --> 00:03:24,995
The insight of understanding that JavaScript fundamentally only worked on an async level
was a was a huge breakthrough for me in any sort of front-end development.

48
00:03:24,995 --> 00:03:31,359
Because before that it was pretty much this novel aspect of, well, there's this way you
can sort of asynchronously work with stuff.

49
00:03:31,359 --> 00:03:36,523
And maybe if you used any sort of uh strong type language, you you may have encountered
some sort of a weight.

50
00:03:36,523 --> 00:03:41,742
But when it comes to JavaScript, there's there's literally no way to get very far without

51
00:03:41,742 --> 00:03:45,244
uh utilizing this part of the of the technology.

52
00:03:45,244 --> 00:03:52,256
And I agree, like most engineers that interact with a website will not understand what
this fundamentally means or how it works.

53
00:03:52,256 --> 00:04:00,370
And it does cause a lot of tests to end up with set timeout like 100 milliseconds while
you wait for some execution to happen.

54
00:04:00,370 --> 00:04:04,731
You know that this is wrong, but especially now like LLMs will write these tests.

55
00:04:04,792 --> 00:04:11,320
And obviously there are some better ways, but the the default mechanism is not so obvious
to even write the tests.

56
00:04:11,320 --> 00:04:14,513
That you want to validate uh the code in the right way.

57
00:04:14,513 --> 00:04:20,559
In doing some UI development for about five years, I got much better at writing CSS
classes and attributes.

58
00:04:20,559 --> 00:04:27,766
I still am a terrible designer, but I I can at least picture the 2D layout and use Flexbox
to make the right thing happen.

59
00:04:27,766 --> 00:04:31,980
And then there's like a lot of extra complexities on top that I still don't fully
understand.

60
00:04:31,980 --> 00:04:34,653
But now I feel much more confident in making a website.

61
00:04:34,653 --> 00:04:36,524
But the testing is still

62
00:04:37,443 --> 00:04:38,944
Just gotta give up.

63
00:04:39,316 --> 00:04:45,746
like I'll I'll I'll pray that maybe what is it, playwright works out of the box and you
can actually write the appropriate test there.

64
00:04:45,746 --> 00:04:52,391
But I don't have any confidence that actually what gets written uh will match and actually
validate things correctly.

65
00:04:52,391 --> 00:04:56,470
And it's interesting that you said like you would just want something to work out of the
box.

66
00:04:56,470 --> 00:05:04,853
know, you know that there's an issue when the mechanism to actually validate some
particular test you're writing, and I I think the most common one is like use some sort of

67
00:05:04,853 --> 00:05:14,826
automation key IDs in your HTML or your you know, in your attributes that allow you to
find certain elements during test time and and perform the right thing.

68
00:05:14,826 --> 00:05:21,578
So you can do it hopefully m not just deterministically, but reliably, no matter what sort
of elements actually show up there.

69
00:05:21,578 --> 00:05:23,416
And that's sort of different than like

70
00:05:23,416 --> 00:05:27,838
Well, if you grab it and programmatically interact with the elements, is it the same as a
user interacting with them?

71
00:05:27,838 --> 00:05:28,668
Maybe not.

72
00:05:28,668 --> 00:05:33,461
And so like I feel like you brought up a bunch of different technologies, and it may be
good to just uh go over each one of them.

73
00:05:33,461 --> 00:05:36,142
My understanding is Selenium is more of a like a web driver.

74
00:05:36,142 --> 00:05:40,754
Like you use your real browser and it's sort of like interacting with it directly.

75
00:05:40,754 --> 00:05:45,156
And so I can imagine that there's a lot of complexity that goes into that interaction.

76
00:05:45,156 --> 00:05:47,087
Like you're crossing processes.

77
00:05:47,087 --> 00:05:50,138
Obviously, the operating system is a huge uh complexity.

78
00:05:50,138 --> 00:05:53,970
So like Selenium is uh a browser automation framework.

79
00:05:53,970 --> 00:05:57,241
Let me uh do browser automation and do it really well.

80
00:05:57,241 --> 00:06:06,585
Uh so the project works uh with Google, Mozilla, Apple, Egalia, like all the people
working on browsers, right?

81
00:06:06,585 --> 00:06:12,527
And kind of the underlying technology is also used in a thing called web platform tests.

82
00:06:12,527 --> 00:06:19,030
Web platform tests are how all the browser vendors make sure that there is inter
interoperability between each other.

83
00:06:19,030 --> 00:06:19,260
Right.

84
00:06:19,260 --> 00:06:25,455
So the test need to pass in Chrome and Firefox and Safari and on different operating
systems.

85
00:06:25,455 --> 00:06:27,537
And that's going through the W three C.

86
00:06:27,537 --> 00:06:29,729
I chair the the working group on that.

87
00:06:29,729 --> 00:06:30,880
That's how Selenium does it.

88
00:06:30,880 --> 00:06:34,853
But then like Cyprus as an example, they do things slightly differently.

89
00:06:34,853 --> 00:06:39,837
They inject JavaScript into the page, manipulate the page as you want.

90
00:06:39,837 --> 00:06:42,069
So if you want to click, it will do it.

91
00:06:42,069 --> 00:06:49,002
But it does it in in a way that is not how a user would do it, where Selenium focuses on
kind of being more

92
00:06:49,002 --> 00:06:50,123
real to the user.

93
00:06:50,123 --> 00:06:57,018
When you click, like if you're on YouTube or whatever, you click on a link, um, there's
extra attributes added.

94
00:06:57,018 --> 00:07:01,472
Um and then they're normally for security reasons, so you can tell if it's a bot or not.

95
00:07:01,472 --> 00:07:08,287
But Cypress is like, well, we just want people to be able to test things and test them
quickly and not have to worry about setting certain things up.

96
00:07:08,287 --> 00:07:11,479
And there's certain complexities between those two points.

97
00:07:11,479 --> 00:07:15,082
Um and other frameworks try fit in somewhere in that spectrum.

98
00:07:15,082 --> 00:07:15,338
I

99
00:07:15,338 --> 00:07:22,143
the full spectrum as treat every interaction on the operating system as sort of a black
box and use a web driver.

100
00:07:22,143 --> 00:07:30,448
And so you're interacting like through the operating system to get into the browser to the
application that's running in order to interact with it to the extreme on the other side,

101
00:07:30,448 --> 00:07:34,571
which is basically make a fake browser that you're rendering all your components in.

102
00:07:34,571 --> 00:07:39,266
And I think this is like sort of the puppeteer space and I think JS DOM is here and
there's there's other things.

103
00:07:39,266 --> 00:07:45,832
I think in my experience and in the teams that I've run, no matter which way where you go,
it seems like they all have a problem in some way, right?

104
00:07:45,832 --> 00:07:53,648
Like if you're interacting across the operating system, then you have to trust a real
browser, a real process spinning up, and the security layers that are, you know, built

105
00:07:53,648 --> 00:07:57,101
into that, but you get the most real experience there.

106
00:07:57,101 --> 00:08:03,747
And on the extreme other side, you don't even know if JSDOM is rendering your HTML
correctly, let alone, you know, finding the places to click on and whatnot.

107
00:08:03,747 --> 00:08:08,282
And then, as you mentioned, it doesn't necessarily correctly match.

108
00:08:08,282 --> 00:08:12,533
you know the production reality uh because different browsers will of course respond
differently.

109
00:08:12,533 --> 00:08:21,106
And I think the the biggest one that you've mentioned, which has caught me a lot of times,
is things like Safari will block link redirects if they happen asynchronously and take

110
00:08:21,106 --> 00:08:26,227
like after more than like a hundred milliseconds after a user has actually interacted with
the page.

111
00:08:26,227 --> 00:08:31,909
And so if you have these like back end things that are processing and then it makes an API
call and then a redirect, uh good luck.

112
00:08:31,909 --> 00:08:35,862
Because while it will work in every single test you write uh at runtime

113
00:08:35,862 --> 00:08:38,722
Who knows what actually is happening for users using Zavar?

114
00:08:38,722 --> 00:08:40,503
Yeah, and and the same can be said.

115
00:08:40,503 --> 00:08:43,165
Like, I know a lot of people are moving to Playwright.

116
00:08:43,165 --> 00:08:57,614
Um Playwright uh ships its own versions of Chrome and Firefox and uh WebKits, doesn't ship
Safari, and they have different security model to what the proper versions of each of

117
00:08:57,614 --> 00:08:58,374
those have.

118
00:08:58,374 --> 00:09:05,260
You know, if you went to a page and it had a hidden iframe with malware installed on it.

119
00:09:05,260 --> 00:09:10,206
The normal browser would block that speaking to the normal page and then obviously getting
onto your machine.

120
00:09:10,206 --> 00:09:15,832
But like Playwright will go, Well, you know, nine times out of ten, 'cause testing, you
don't care about that thing.

121
00:09:15,832 --> 00:09:16,883
So we'll just turn it on.

122
00:09:16,883 --> 00:09:18,415
It makes testing a little bit easier.

123
00:09:18,415 --> 00:09:22,189
But then you've got this thing where it's like, it's not exactly how people would use it.

124
00:09:22,189 --> 00:09:26,914
And then you get your situation like you said with Safari and you're like, Why is this
happening?

125
00:09:26,914 --> 00:09:30,017
The one that caught me most recently was uh BF cache.

126
00:09:30,017 --> 00:09:31,738
Like I didn't know this was a thing.

127
00:09:31,738 --> 00:09:33,920
And now I'm like, I don't understand.

128
00:09:33,920 --> 00:09:40,545
How how is it like much later that a user leaves a page and comes back to it and they're
in the exact same state that they were?

129
00:09:40,545 --> 00:09:45,419
There was an expectation that you load a web page and you have no memory allocated.

130
00:09:45,419 --> 00:09:51,975
Obviously what's in a local store or in cookies, those are preserved and obviously
whatever state you have in the back end.

131
00:09:51,975 --> 00:09:54,357
But how do they have these things populated?

132
00:09:54,357 --> 00:09:55,748
It's like, well, shocker.

133
00:09:55,748 --> 00:09:56,556
Uh

134
00:09:56,556 --> 00:10:01,751
Now browsers like try to save the actual memory state and repopulate it.

135
00:10:01,792 --> 00:10:07,939
And you know, the introduction of this obviously breaks every single expectation you have
in any test anywhere.

136
00:10:07,939 --> 00:10:12,144
Is everyone just testing their applications wrong in browsers?

137
00:10:12,144 --> 00:10:17,934
Is there or is there like no actual right way to have a conclusive non-flaky test?

138
00:10:17,934 --> 00:10:23,528
Truthfully, I think flakiness is down to a mismatch in expectations.

139
00:10:23,528 --> 00:10:26,500
Uh and some of it is unknown unknowns.

140
00:10:26,500 --> 00:10:27,881
So BF cache, right?

141
00:10:27,881 --> 00:10:30,923
A lot of people have no idea what a BF cache is.

142
00:10:30,923 --> 00:10:35,036
BF cache is this like very specific um cache that browsers have.

143
00:10:35,036 --> 00:10:40,239
Like if you hit the back button, it needs people don't want to have to wait for a page to
reload, right?

144
00:10:40,239 --> 00:10:45,013
And especially if you're on like slow internet, think mobile phones is a perfect example.

145
00:10:45,013 --> 00:10:47,074
It loads super fast, instantly.

146
00:10:47,074 --> 00:10:57,482
The downside is is that like if you were hoping for it to insert new data, like you have
to have code that kind of goes, Oh, I'll run on a like a time and just see if some there's

147
00:10:57,482 --> 00:10:58,843
new data and things like that.

148
00:10:58,843 --> 00:11:01,775
Again, it's a mismatch in information that people have.

149
00:11:01,775 --> 00:11:09,002
Um and I think it comes down to like browsers are super complex and hard, like they're
operating systems, like you know

150
00:11:09,002 --> 00:11:11,623
I can see both sides of the argument, right?

151
00:11:11,623 --> 00:11:18,526
From an application standpoint, if you're engineering a new one, you only want to use the
features that you know about, right?

152
00:11:18,526 --> 00:11:22,188
Unexpected features are likely going to cause a problem at some point for your users.

153
00:11:22,188 --> 00:11:30,371
However, on the flip side, if you know about the functionality that you want to have,
like, over time, we know users are going to leave our page and come back.

154
00:11:30,371 --> 00:11:36,782
It would be great if we stored their state automatically so they didn't lose it the next
time they loaded the page.

155
00:11:36,782 --> 00:11:44,178
Especially for applications where they use it for more than a few minutes a day or hours a
day, you know, maybe it's a professional application, a business application, quickly

156
00:11:44,178 --> 00:11:45,869
loading that state is valuable.

157
00:11:45,869 --> 00:11:51,714
And so now you have all all the applications we have online uh require basically storing
state.

158
00:11:51,714 --> 00:11:59,350
And at that point, you're like, well, wouldn't it be great if every single website needs
to implement this functionality for us to push down the ownership down a level, right?

159
00:11:59,350 --> 00:12:06,466
Rather than having to implement your own store to cookies or local store or whatever, or
you know, some sort of SQLite database.

160
00:12:06,553 --> 00:12:12,073
that's in the browser to automatically repopulate quickly, what if the browser did it and
then it implements that?

161
00:12:12,073 --> 00:12:18,990
But it does make it extra challenging to make sure that the application is actually doing
the right thing and that requires, you know, throwing these tests.

162
00:12:18,990 --> 00:12:29,354
If you're very interested to have a look and see what the Mozilla CI looks like, there's a
website called treeherder.mozilla.org.

163
00:12:29,575 --> 00:12:32,916
It's open to the public because it's Mozilla, everything's open source.

164
00:12:32,916 --> 00:12:37,558
And you can see like how things are being built, how things are being tested.

165
00:12:37,558 --> 00:12:46,382
When I was joined Mozilla, which was early 2010, um, we had uh there was a guy working
there, Joel um Maher.

166
00:12:46,382 --> 00:12:47,586
Um

167
00:12:47,586 --> 00:12:52,428
It was a project called War on Orange because the flaky tests were called Orange Tests.

168
00:12:52,428 --> 00:12:58,030
Uh the key thing that we kind of got to the point was that we can't get a green CI.

169
00:12:58,030 --> 00:12:59,011
It's impossible.

170
00:12:59,011 --> 00:13:03,032
We're running millions of tests every time we make a push.

171
00:13:03,032 --> 00:13:08,955
Mozilla was also trying to like bring down costs for running all of our infrastructure.

172
00:13:08,955 --> 00:13:10,295
So it's not run everything.

173
00:13:10,295 --> 00:13:17,058
And so you might make a change and then eight hours later a full test suite runs and
suddenly there's your a failure.

174
00:13:17,058 --> 00:13:21,142
And you gotta know if it's related to any of those changes or things like that.

175
00:13:21,142 --> 00:13:23,003
And so we kept working on it.

176
00:13:23,003 --> 00:13:26,967
Um and we got to the point where flakiness we knew we were never going to get rid of.

177
00:13:26,967 --> 00:13:28,428
We can't make it any better.

178
00:13:28,428 --> 00:13:30,370
So we'll just monitor that.

179
00:13:30,370 --> 00:13:34,613
And if we see it often enough, then we'll kind of do something about it.

180
00:13:34,613 --> 00:13:38,136
Either turn off the test, get an engineer to rewrite the test.

181
00:13:38,136 --> 00:13:45,442
Because as I said earlier, when you write your test, you're making the assumption that
like you you know where you're starting from and where you're ending.

182
00:13:45,484 --> 00:13:52,440
But three tests above could have forgot to do any cleanup or someone put in a set timeout,
like you got all these things.

183
00:13:52,440 --> 00:13:55,703
You could turn off one test and it could cause another test to go flaky.

184
00:13:55,703 --> 00:13:56,023
Right.

185
00:13:56,023 --> 00:13:57,324
So it's it's whack-a-mole.

186
00:13:57,324 --> 00:14:02,278
So you were we were trying to bring it into a state where you could clean this up in a
meaningful way.

187
00:14:02,278 --> 00:14:07,332
The way we monitored it was a group of people, they were called the sheriffs.

188
00:14:07,332 --> 00:14:08,843
they were watching the CI.

189
00:14:08,843 --> 00:14:13,878
It just became such a point where you no matter what you did, you we knew it wasn't gonna
be green.

190
00:14:13,878 --> 00:14:14,584
So we just

191
00:14:14,584 --> 00:14:17,165
try to go for how how much is the delta?

192
00:14:17,165 --> 00:14:21,797
Because like this test, it should only fail maybe once in a hundred times.

193
00:14:21,797 --> 00:14:23,118
That's okay.

194
00:14:23,258 --> 00:14:27,034
Or m maybe five in a hundred, no more than that.

195
00:14:27,034 --> 00:14:27,820
Right.

196
00:14:27,820 --> 00:14:36,254
Um and if it's that one in a hundred, the effort to go find and fix that problem becomes a
lot more than just letting the test run and fail.

197
00:14:36,254 --> 00:14:39,396
Because we go, yeah, we we're expecting that one in a hundred.

198
00:14:39,396 --> 00:14:39,986
It's okay.

199
00:14:39,986 --> 00:14:40,362
Is it

200
00:14:40,362 --> 00:14:47,457
Actually, unattainable to have a perfectly working test suite continuously for any sort of
large project?

201
00:14:47,457 --> 00:14:49,799
Or is that a like a fool's error?

202
00:14:49,799 --> 00:14:58,795
And like it would take an infinite amount of time to ensure that you had this, or the
complexity that would have to be built to make it happen, or the the running time of the

203
00:14:58,795 --> 00:15:01,727
test would have to be a multiple of what it is.

204
00:15:01,727 --> 00:15:08,184
For instance, you could have a more guaranteed result if you spun each one up in a private
container that is.

205
00:15:08,184 --> 00:15:10,730
completely isolated, is this like the middle ground?

206
00:15:10,730 --> 00:15:18,922
Or is this just a matter of at the time, you know, this is the problem that we had, and
the only way we could take the next step is by watching the del

207
00:15:18,922 --> 00:15:29,148
When I was at Mozilla, I had friends um at Google and Facebook th who I'd regularly speak
to because some of the open source projects that Mozilla was working on, they were working

208
00:15:29,148 --> 00:15:29,802
on it too.

209
00:15:29,802 --> 00:15:40,334
Uh so like there was work on Mercurial, um, the version control system, and like we had
talked about trying to stick these flaky tests into kind of

210
00:15:40,334 --> 00:15:43,387
an area and go like let's see how it often it happens.

211
00:15:43,387 --> 00:15:44,608
Do it all automatically.

212
00:15:44,608 --> 00:15:48,312
So like there's no humans uh really until someone needs to fix it.

213
00:15:48,312 --> 00:15:54,188
And so the like Facebook philosophy was that they were aiming for 100% green as they were
growing.

214
00:15:54,188 --> 00:15:59,913
And after a while they were like, well, you know, we keep turning off as many tests as
we're adding to the system.

215
00:15:59,913 --> 00:16:04,664
Where Mozilla was like, well we're adding to the s system, like we need to have

216
00:16:04,664 --> 00:16:06,916
tests run within a certain time frame.

217
00:16:06,916 --> 00:16:17,304
Our time frame was kind of long compared to like your average web application where we're
talking, you know, potentially eight to twelve hours from you doing a commit to the CI

218
00:16:17,304 --> 00:16:25,601
finishing because you gotta build it on Windows, but Windows is not just one operating
system, it's multiple operating systems with multiple architectures.

219
00:16:25,601 --> 00:16:29,514
And I'd do the same on Mac and on Linux.

220
00:16:29,566 --> 00:16:31,698
And on um Android to solve this.

221
00:16:31,698 --> 00:16:33,820
I think it's becoming a little bit easier nowadays.

222
00:16:33,820 --> 00:16:40,116
Each tab is its own kind of profile in the way it works so that there's no data that's
shared between those.

223
00:16:40,116 --> 00:16:42,808
Um they they're called user context.

224
00:16:42,808 --> 00:16:44,649
Why do any sort of browser testing?

225
00:16:44,649 --> 00:16:47,560
Why does it take an extensive amount of time to sort of execute?

226
00:16:47,560 --> 00:16:55,293
And if it's like, well, spinning up the process is most of the work, or there's some
single overhead and it's the container spin up or user context spin up that's that's, you

227
00:16:55,293 --> 00:16:56,364
know, problematic.

228
00:16:56,364 --> 00:17:03,098
Then if we reduce that cost in that area, then you can more easily use the same tests.

229
00:17:03,098 --> 00:17:11,060
You know, the thought that comes to mind for me is that it's like a completely different
set of challenges for uh building a browser and running the test against it.

230
00:17:11,060 --> 00:17:11,512
Like

231
00:17:11,512 --> 00:17:22,535
desktop software or OS software than uh cloud deployed stuff, where I think about you want
to avoid breaking uh production applications as much as possible, so much so that you end

232
00:17:22,535 --> 00:17:26,006
up slowing teams down by using some sort of release train.

233
00:17:26,006 --> 00:17:29,607
Because you wanna obviously you only want to release one thing at a time.

234
00:17:29,607 --> 00:17:38,059
And at the same time, you want to make sure that you're only releasing it to a small
subset of the total customer base so that you can watch for changes that you wouldn't have

235
00:17:38,059 --> 00:17:39,360
been able to predict after all.

236
00:17:39,360 --> 00:17:40,960
I mean, if every change

237
00:17:40,960 --> 00:17:47,394
only had predictable, you know, potential impacts, you wouldn't need to do this, but there
was always edge cases that that no one really thought of.

238
00:17:47,484 --> 00:17:49,656
and in specific customer environments for sure.

239
00:17:49,656 --> 00:17:55,820
And this feels like a whole like completely orthogonal or mutually exclusive set of
problems with a completely different solution.

240
00:17:55,820 --> 00:17:59,678
The delay is caused by the cost of actually running the test suite

241
00:17:59,678 --> 00:18:12,363
I think a lot of people put their tests in the wrong place of the testing pyramid, where
you have a nice wide uh base of unit tests or small tests, slightly narrower integration

242
00:18:12,363 --> 00:18:18,186
tests or medium tests, and then you have these long end to end tests and you should have
as few as possible.

243
00:18:18,186 --> 00:18:21,427
I remember this from Selenium Conference in London.

244
00:18:21,427 --> 00:18:28,500
Simon Stewart, the creator of WebDriver, he was doing the keynote and he said that what
people should be doing

245
00:18:28,616 --> 00:18:32,930
Is making sure that they have one test at the end, one large test.

246
00:18:32,930 --> 00:18:34,051
No more, no less.

247
00:18:34,051 --> 00:18:41,017
And what he was trying to say really was not that you should have one test, but that one
test should only do one thing really well.

248
00:18:41,017 --> 00:18:48,613
So if it's like your workflow is log in, do three steps and then log out, that's fine,
right?

249
00:18:48,613 --> 00:18:50,074
That's the most important thing.

250
00:18:50,074 --> 00:18:54,268
It stops where bugs being in the riskiest parts of the system.

251
00:18:54,268 --> 00:18:57,164
You can still have another test where it's like, well,

252
00:18:57,164 --> 00:19:00,216
This next risk level is here, right?

253
00:19:00,256 --> 00:19:09,643
But it the key thing is here is not to have 10 or 100 or whatever, is to try think of just
one risk area, have a good test for that.

254
00:19:09,643 --> 00:19:14,346
Not the longest test, because you then have other problems, and then build out your test
suite.

255
00:19:14,346 --> 00:19:20,650
Then people tend to have faster tests and they have faster feedback loops.

256
00:19:20,650 --> 00:19:26,872
I also am a big advocate of trying to see if there's other ways you could get the same
information.

257
00:19:26,872 --> 00:19:28,363
without necessarily testing.

258
00:19:28,363 --> 00:19:34,145
Um I'm sure you've had other people on talking about say observability as an example,
right?

259
00:19:34,145 --> 00:19:35,626
Adding that to your system.

260
00:19:35,626 --> 00:19:39,107
Um and you can add observability into your test.

261
00:19:39,107 --> 00:19:45,410
So you can trace it from your test, or your one test, all the way through the system and
back out again.

262
00:19:45,410 --> 00:19:51,652
Then you you have this like knowledge that you go, I know that that part of the system is
bulletproof.

263
00:19:51,652 --> 00:19:56,194
And then we can still move fairly fast because we've got all these backup layers of

264
00:19:56,460 --> 00:20:04,734
A lot of unit tests, as many integration tests as we can probably get away with, uh, but
not too many they mustn't outnumber the unit tests.

265
00:20:04,734 --> 00:20:09,946
And then uh high quality for the risk areas, end-to-end tests, um, large tests.

266
00:20:09,946 --> 00:20:17,549
And for those who do a lot of UI stuff, you could also, you know, between the integration
and the um end-to-end tests, you can throw in component tests.

267
00:20:17,549 --> 00:20:25,090
If you're doing React or any front-end framework, we'd only test that component alone and
make sure that's doing really well.

268
00:20:25,090 --> 00:20:29,274
No matter what it is or what the advice is, there's always a way to mess it up.

269
00:20:29,274 --> 00:20:35,009
And I I think the the mess up here is uh what exactly makes a unit test versus not.

270
00:20:35,009 --> 00:20:39,204
And I mean, obviously we have canonical definitions of each one of these uh for
recommendation.

271
00:20:39,204 --> 00:20:50,526
Even with a defined setup and teardown for uh a unit test, there's maybe cruft left over
as far as configuration goes, or a desire to refactor and reuse.

272
00:20:50,526 --> 00:20:55,901
And there's some sort of shared memory or dictionary that ends up getting ported and then
it causes a lot of extra complexity.

273
00:20:55,901 --> 00:21:03,848
Because, wouldn't it be easier if we just did this slightly differently so that we
wouldn't have extra code or whatever the reason is?

274
00:21:03,848 --> 00:21:05,439
And you end up with a problem here.

275
00:21:05,439 --> 00:21:14,507
I think another aspect, especially with the test pyramid, which is my preference
personally, uh, because I've heard that there's other um geometric shapes that are often

276
00:21:14,507 --> 00:21:18,264
utilized as a justification for how you should make your your tests.

277
00:21:18,264 --> 00:21:19,675
How many though is it?

278
00:21:19,675 --> 00:21:22,166
Like how many unit tests should you have?

279
00:21:22,166 --> 00:21:24,097
How many component level tests?

280
00:21:24,097 --> 00:21:26,979
And you know, it doesn't really matter what these things are called.

281
00:21:26,979 --> 00:21:29,810
There's the the very cheap tests and the very expensive tests.

282
00:21:29,810 --> 00:21:39,836
and I actually really am behind the only have one because it does tell people like if you
have more than one like total black box test, you probably are doing you're probably doing

283
00:21:39,836 --> 00:21:40,686
too much there.

284
00:21:40,686 --> 00:21:44,068
And it's not going to tell you what you really need about your system when it fails.

285
00:21:44,068 --> 00:21:47,740
Or when it fails, you don't have the sort of observability that you want.

286
00:21:47,896 --> 00:21:53,671
But I haven't really heard any good arguments for what is the right what is the right
percentage of of testing you have?

287
00:21:53,671 --> 00:21:59,905
And is at like at the unit test level, is it ten percent, twenty percent, uh a hundred
percent?

288
00:21:59,905 --> 00:22:02,968
Like w where is this number as a recommendation like

289
00:22:02,968 --> 00:22:07,229
So a lot of people will go, I need uh as many unit tests as possible.

290
00:22:07,229 --> 00:22:08,720
I need every bit covered.

291
00:22:08,720 --> 00:22:11,920
They're only going to take three hundred milliseconds, so it doesn't really matter.

292
00:22:11,920 --> 00:22:19,903
The key thing is that you're trying to strike the balance between refactorability of your
tests and this underlying system.

293
00:22:19,903 --> 00:22:20,243
Right.

294
00:22:20,243 --> 00:22:24,004
So I'm a big uh proponent for test driven development.

295
00:22:24,004 --> 00:22:27,505
I hate it when people go, I need to have a hundred percent test coverage.

296
00:22:27,505 --> 00:22:32,842
I can guarantee you will have a hundred percent test coverage and there will be bugs in
that system.

297
00:22:32,842 --> 00:22:35,044
It's just the way the systems are built.

298
00:22:35,044 --> 00:22:41,499
A guy wrote a website that got a lighthouse score of a hundred, but it was the worst
website.

299
00:22:41,499 --> 00:22:42,359
You couldn't read it.

300
00:22:42,359 --> 00:22:44,171
You couldn't do anything, right?

301
00:22:44,171 --> 00:22:46,973
But it said accessibility is amazing.

302
00:22:46,973 --> 00:22:48,764
It loads super fast.

303
00:22:48,764 --> 00:22:49,144
Everything.

304
00:22:49,144 --> 00:22:51,836
It was just the worst website in the world.

305
00:22:51,836 --> 00:22:57,781
Python had this major problem around like ASCII versus Unicode before it moved to Python
3.

306
00:22:57,781 --> 00:22:59,470
And people were like, I've got 100%.

307
00:22:59,470 --> 00:23:01,092
test coverage, it's amazing.

308
00:23:01,092 --> 00:23:02,324
And then it's like, okay, cool.

309
00:23:02,324 --> 00:23:11,192
And my favorite way to test that was I would put a uh Unicode snowman into a test and then
just watch as the entire system crashes.

310
00:23:11,192 --> 00:23:21,107
There is a good thing about first deciding up front what the risks are, what your risk
model is, where the highest ROI is, and then attacking it directly.

311
00:23:21,168 --> 00:23:25,770
And, you know, my personal thing is like, oh, is there complexity in this unit test?

312
00:23:25,770 --> 00:23:35,616
Like things that you wouldn't expect to have been written a particular way, uh, a need for
the implementation to be something or or something else as far as the business case goes,

313
00:23:35,616 --> 00:23:39,457
or is it capturing like actual expectations on how the business works?

314
00:23:39,698 --> 00:23:40,396
And

315
00:23:40,396 --> 00:23:48,070
If it's not any of those things, or there's like some you know complicated math that's
involved with, like this has to be a a perfect merge sort or whatever, and you've written

316
00:23:48,070 --> 00:23:52,393
that function and you actually want to test it out to make sure it does actually sort
correctly.

317
00:23:52,393 --> 00:23:54,134
Those things make a lot of sense.

318
00:23:54,134 --> 00:23:58,796
I think where things start to break down is as you pointed out, like just asking for 100%
test coverage.

319
00:23:58,796 --> 00:24:02,058
My question is like how much how much are you willing to spend on making that happen?

320
00:24:02,058 --> 00:24:03,609
And it's not just an upfront cost, right?

321
00:24:03,609 --> 00:24:05,140
You said maintainability.

322
00:24:05,140 --> 00:24:09,400
Every additional test you write is first of all, it's extra code.

323
00:24:09,400 --> 00:24:11,512
Which means it's additional stuff to maintain.

324
00:24:11,512 --> 00:24:15,856
And extra code means obviously more space storage updates.

325
00:24:15,856 --> 00:24:22,522
Like if you're cloning using Git or whatever, your more space on your CI, CD, more
processing power, just straight out of the box.

326
00:24:22,522 --> 00:24:24,754
Then there's like understanding of what's even there.

327
00:24:24,754 --> 00:24:27,466
There's the complexity of where are my tests?

328
00:24:27,466 --> 00:24:34,662
Like if you're working in a monolith project versus if you have a microservice
architecture, it's so much easier to work in microservices when you can just control shift

329
00:24:34,662 --> 00:24:35,973
F things.

330
00:24:36,662 --> 00:24:39,013
And find whatever you're looking for just by typing in the search box.

331
00:24:39,013 --> 00:24:44,766
Whereas in a monolith, you'll get like a thousand different like top level projects with
all the exact same thing.

332
00:24:44,766 --> 00:24:46,922
Good luck finding what you're actually looking for.

333
00:24:46,922 --> 00:24:49,269
Uh, and let alone the the tests themselves.

334
00:24:49,269 --> 00:24:55,232
Uh, there's a thing in that I really like from let's say architecture, uh, which deals in
constraints.

335
00:24:55,232 --> 00:25:01,939
Whereas like if you if you or geometry really, if you pick two particular points, then the
rest of your object is constrained in some way.

336
00:25:01,939 --> 00:25:03,667
You you can't define it differently.

337
00:25:03,667 --> 00:25:04,918
It it's only that, right?

338
00:25:04,918 --> 00:25:06,136
If you have like a

339
00:25:06,136 --> 00:25:13,788
A square and a two-dimensional plane, and you pick two points of where the square is, like
how many squares can you draw with those two points being the corners?

340
00:25:13,788 --> 00:25:15,919
And th there's not there's a finite number of answers.

341
00:25:15,919 --> 00:25:17,589
You've constrained the way it works.

342
00:25:17,589 --> 00:25:19,350
And unit tests work the same way.

343
00:25:19,350 --> 00:25:26,759
So I think these are the things that definitely get missed when talking about how many
tests we should have and and really percentages.

344
00:25:26,759 --> 00:25:33,164
I mean, obviously for some people, uh saying we need to have higher unit test coverage, it
may be the right thing.

345
00:25:33,250 --> 00:25:39,478
but at any sort of mature level you end up starting to really look at like, okay, what tet
areas do we really need to test?

346
00:25:39,478 --> 00:25:48,062
I th I think kind of unless you're working in finance or health or aeronautics, a hundred
percent is not where you wanna be.

347
00:25:48,062 --> 00:25:48,562
You know what?

348
00:25:48,562 --> 00:25:57,002
I'll even disagree with that honestly, because it's like if you look at where most of the
issues come from, they're hum human written code or LM written code today.

349
00:25:57,002 --> 00:25:57,194
Right.

350
00:25:57,194 --> 00:26:00,174
When you make a change, that's where the liability sends, right?

351
00:26:00,174 --> 00:26:05,249
You know, if everything's working today, likely it's going to keep working unless you
fundamentally make a change.

352
00:26:05,249 --> 00:26:09,181
Or uh or it's an outside input into the system, right?

353
00:26:09,181 --> 00:26:13,677
You put a snowman in a in an input box and it it you know blows up the system.

354
00:26:13,677 --> 00:26:17,888
You know, it's it's human input outside that, which you know, it causes a lot of the
problems.

355
00:26:17,888 --> 00:26:24,164
Like I think a a lot of mistakes in say the healthcare world didn't did didn don't come
from the system saying something.

356
00:26:24,164 --> 00:26:26,812
It's the humans are dealing with it, right?

357
00:26:26,902 --> 00:26:27,222
Yeah.

358
00:26:27,222 --> 00:26:28,884
And the the other side of it is that way.

359
00:26:28,884 --> 00:26:39,797
Like um you and I have been talking about correctness, but there could also be tests in
kind of the unit space and things like that that are focusing on performance, security,

360
00:26:39,797 --> 00:26:40,888
things like that, right?

361
00:26:40,888 --> 00:26:45,122
Um and if you keep adding to your point about maintenance,

362
00:26:45,174 --> 00:26:45,834
It's a lot.

363
00:26:45,834 --> 00:26:53,006
I'm sure most people nowadays who are kind of using other lambs for their work get to the
point where they're like, I need to restart my computer.

364
00:26:53,006 --> 00:26:55,987
I need to delete this because like everything's working in work trees.

365
00:26:55,987 --> 00:27:03,999
You run out of hard drive space very quickly, even on the smallest amount of um like
smallest repositories, right?

366
00:27:04,009 --> 00:27:05,840
'cause you're like, I can move faster now.

367
00:27:05,840 --> 00:27:08,140
I can hopefully not break things, right?

368
00:27:08,140 --> 00:27:14,182
Like, but you know, you're moving as fast as possible and then you're just like, man, this
is really painful.

369
00:27:14,214 --> 00:27:17,038
Uh, I've gotta slow down, I've gotta do this, I've gotta clean up, right?

370
00:27:17,038 --> 00:27:26,114
Like nowadays I think I'm I'm deleting build caches um at least once a month because I
just run out of hard drive space in the work that I'm doing.

371
00:27:26,114 --> 00:27:29,702
Yeah, yeah, and I I don't see that getting better any time.

372
00:27:29,702 --> 00:27:36,108
No, especially with the cost of hard drives uh growing exponentially, so it's gonna get
worse.

373
00:27:36,108 --> 00:27:37,549
I've do you know what?

374
00:27:37,549 --> 00:27:43,298
When I started working at Mozilla, I had that feeling of like literally everyone can see
my code.

375
00:27:43,298 --> 00:27:48,352
The thing that I I learned very quickly and I had really good mentors was that all code is
mutable.

376
00:27:48,352 --> 00:27:51,625
Like you can make mistakes and you can go back and fix it.

377
00:27:51,625 --> 00:27:54,798
And we shouldn't like chastise people for making mistakes.

378
00:27:54,798 --> 00:27:56,289
I think that's the the key thing.

379
00:27:56,289 --> 00:27:56,619
Yeah.

380
00:27:56,619 --> 00:28:08,589
One thing I've learned from having spent a lot of time in open source and then kind of
moved into uh kind of a private company now is the amount of the good practices I did

381
00:28:08,589 --> 00:28:11,381
learn from being in open source.

382
00:28:11,381 --> 00:28:12,202
So like

383
00:28:12,202 --> 00:28:18,427
You could never put secrets or anything like that into a repository because like all hell
would break loose, right?

384
00:28:18,427 --> 00:28:21,980
And that's actually like really good practice that we can do.

385
00:28:21,980 --> 00:28:25,062
So I when I moved from private into open source, I was like, I want to do it.

386
00:28:25,062 --> 00:28:26,327
And there's like, no, I can't do it.

387
00:28:26,327 --> 00:28:27,594
Everyone can see it.

388
00:28:27,615 --> 00:28:30,767
And it's like, I think that kind of solidified it.

389
00:28:30,767 --> 00:28:34,860
But also to me, I think I didn't have any of the really bad things.

390
00:28:34,860 --> 00:28:35,781
I probably have.

391
00:28:35,781 --> 00:28:38,473
Like there's probably awful code that I've written.

392
00:28:38,473 --> 00:28:42,178
And uh, you know, I should be ashamed of it.

393
00:28:42,178 --> 00:28:44,421
But at the same time I can always go back and fix it.

394
00:28:44,421 --> 00:28:56,276
Um and I think that's the far more valuable side of things with um that I think most
people should get used to is that like uh making mistakes is bad, but you know, we're

395
00:28:56,276 --> 00:28:57,752
human, we're fallible, right?

396
00:28:57,752 --> 00:29:04,875
going from private or companies with open source to closed source strategies uh and the
other way around, vice versa.

397
00:29:04,875 --> 00:29:08,166
I I feel like there's there are learnings you can take in either direction.

398
00:29:08,247 --> 00:29:17,931
And also, of course, mistakes that you you should have dropped, but you know are are still
there because you haven't sort of reevaluated where the the highest ROI is.

399
00:29:17,931 --> 00:29:25,098
Really from that perspective, it's all about learning from whatever experience you've had
rather than not trying to think about it and just doing what's ever there.

400
00:29:25,098 --> 00:29:31,681
Nine times out of ten, walking into any meeting, I felt like the dumbest person in the
room all the time.

401
00:29:31,681 --> 00:29:35,463
So when I was at Mozilla, when I first joined, Brendan Eich was there.

402
00:29:35,463 --> 00:29:36,563
He was the CTO.

403
00:29:36,563 --> 00:29:38,364
He's the guy who created JavaScript.

404
00:29:38,364 --> 00:29:42,446
There were John Risick, the guy who created jQuery.

405
00:29:42,446 --> 00:29:43,686
He was working there.

406
00:29:43,686 --> 00:29:48,058
There is huge amounts of complexity that I never fully understood.

407
00:29:48,058 --> 00:29:50,990
Uh and I learned a lot of it as I was going through it.

408
00:29:50,990 --> 00:29:52,022
So kind of how

409
00:29:52,022 --> 00:29:56,085
CSS is rendered because I had to learn it to understand how to do automation.

410
00:29:56,085 --> 00:29:57,986
And it's super complex.

411
00:29:57,986 --> 00:30:03,450
And half the time the the what the browser just goes is like, hey, operating system, this
is our best effort.

412
00:30:03,450 --> 00:30:04,611
Can you clean it up, please?

413
00:30:04,611 --> 00:30:06,522
And the operating system goes, Yeah, sure.

414
00:30:06,522 --> 00:30:07,794
Let me do that, right?

415
00:30:07,794 --> 00:30:14,198
But then there's the JavaScript, but JavaScript has to be uh compiled down to bytecode to
be able to be run properly.

416
00:30:14,198 --> 00:30:19,778
And I looked at that and I was like, there's a reason why I didn't do uh the compiler
class in my

417
00:30:19,778 --> 00:30:20,818
computer science degree.

418
00:30:20,818 --> 00:30:30,544
I feel like I was working on like the the nice, easy parts where other people like their
job was just security and performance while doing their like normal job.

419
00:30:30,544 --> 00:30:40,629
JavaScript as an example, you gotta make sure it's secure so no um like people can't hack
it by doing memory management when you shouldn't be able to do memory management and like

420
00:30:40,629 --> 00:30:41,360
things like that.

421
00:30:41,360 --> 00:30:44,614
And it's like, yeah, I I'll I'll just go work over there.

422
00:30:44,614 --> 00:30:49,504
It's the like it's kind of complex but it's nowhere near as complex as that part.

423
00:30:49,504 --> 00:30:54,806
I like the the architecture design above everything and I gravitated towards the security
area.

424
00:30:54,806 --> 00:31:05,139
I honestly feel like it's so much easier to r understand and write secure code or deal
with algorithms than it is to do some memory dumps to figure out issues with with caching

425
00:31:05,139 --> 00:31:10,101
or, you know, performance or the hacks that you have to put in to make things more
performant.

426
00:31:10,101 --> 00:31:14,282
It's just I don't know, that for me is just too disgusting to ever want to work in that
area.

427
00:31:14,282 --> 00:31:14,889
Yeah.

428
00:31:14,889 --> 00:31:16,371
no, I I was the same, right?

429
00:31:16,371 --> 00:31:19,856
Like no one really cares about performance of test automation framework.

430
00:31:19,856 --> 00:31:22,039
Like they just want it to run as fast as possible.

431
00:31:22,039 --> 00:31:22,600
Yeah.

432
00:31:22,600 --> 00:31:27,978
But you're relying on like how JavaScript or those things work and it's like, Well, yeah,
it's a bit that test is a bit slow.

433
00:31:27,978 --> 00:31:30,201
I don't think it's my code, it must be other people's

434
00:31:30,963 --> 00:31:31,424
Okay.

435
00:31:31,424 --> 00:31:36,758
Well then, you know, maybe it's worth diving into a little bit your experience at at
Browser Stack.

436
00:31:36,758 --> 00:31:45,106
Yeah, so fortunately, like a lot of what Browser Stack does is built off a lot of
technology in the open source space.

437
00:31:45,106 --> 00:31:47,899
People doing their tests in browsers and things like that.

438
00:31:47,899 --> 00:31:50,291
A lot of it's like Selenium or Playwright.

439
00:31:50,291 --> 00:31:55,076
And so we were able to commercialize it, make it more secure, things like that.

440
00:31:55,076 --> 00:31:56,256
And it builds all that out.

441
00:31:56,256 --> 00:32:02,358
And then uh we have interesting kind of uh proprietary systems in our data centers.

442
00:32:02,358 --> 00:32:07,283
around how we can have mobile phones because we use real devices for everything.

443
00:32:07,283 --> 00:32:16,172
Mobile phones in data centers, which then have other problems that like 'cause they've all
got antennae and they're all trying to speak to the outside world and you're like trying

444
00:32:16,172 --> 00:32:18,735
to limit who it can speak to and things like that.

445
00:32:18,735 --> 00:32:25,716
So we have had to create interesting Faraday cages around certain things so that we can
keep things on the right track.

446
00:32:25,716 --> 00:32:29,399
And again, you don't want like stuff to bleed through or have other problems.

447
00:32:29,399 --> 00:32:35,034
And the interesting stuff is not necessarily the software, it becomes more the hardware
when it comes to that space.

448
00:32:35,034 --> 00:32:41,038
Uh mobile phones, when they're always on, have a problem that like the batteries don't
like to be charging all the time.

449
00:32:41,038 --> 00:32:42,600
As an example, right?

450
00:32:42,600 --> 00:32:47,083
We have data center engineers who sit in each of uh our data centers.

451
00:32:47,083 --> 00:32:50,990
And what we've had to build out is infrastructure equipment.

452
00:32:50,990 --> 00:33:02,229
Equivalent to like if you go to a sports venue, you've got these like thousands of people
all trying to use the same Wi-Fi or or cell tower and things like that.

453
00:33:02,229 --> 00:33:07,843
And we know that like in football stadiums, they have their own infrastructure that kind
of help spread the load.

454
00:33:07,843 --> 00:33:13,427
So we've had to build out a lot of tooling to kind of handle those situations.

455
00:33:13,427 --> 00:33:17,262
We have Mac minis for running Safari because there are

456
00:33:17,262 --> 00:33:22,561
rules and terms from Apple that Safari can only run on Apple hardware.

457
00:33:22,561 --> 00:33:26,570
And so we don't want to be breaking any rules around that because we can be sued on to

458
00:33:26,570 --> 00:33:33,852
you have a lot of physical automation in play to be able to control the state of the
physical devices like remotely through some sort of interface?

459
00:33:33,852 --> 00:33:35,142
yeah, as much as possible.

460
00:33:35,142 --> 00:33:43,085
I think the last time I saw we had something like thirty thousand mobile devices uh spread
across all our data centers across the world.

461
00:33:43,085 --> 00:33:52,728
The reason why we started noticing certain flakes um happening on a certain machine is
because memory started dying on that machine and that created a flaky test.

462
00:33:52,728 --> 00:34:01,058
And so we have similar problems on mobile now where we like have to build this out and
then buy kind of the devices and then

463
00:34:01,058 --> 00:34:06,904
buy extra devices for when situations like that happen where we can go, Well, you know, we
can't repair this one.

464
00:34:06,904 --> 00:34:07,946
It's it's dead now.

465
00:34:07,946 --> 00:34:11,710
Let's swap it in for a o a new one and do kind of a hot swap on those.

466
00:34:11,710 --> 00:34:12,002
The

467
00:34:12,002 --> 00:34:14,713
The testing on mobile devices has always been really confusing for me.

468
00:34:14,713 --> 00:34:18,355
Like I don't understand why the experience is so uh like atrocious.

469
00:34:18,355 --> 00:34:26,080
I mean, obviously there's like money involved and some large organizations or companies
that are make a lot of money from this.

470
00:34:26,080 --> 00:34:28,581
So uh controlling that makes sense.

471
00:34:28,581 --> 00:34:39,597
But I just don't understand why they don't come out with like full emulators offered from
a proprietary standpoint that allow you to do this effectively, like you know, steal your

472
00:34:39,597 --> 00:34:40,738
customers basically.

473
00:34:40,738 --> 00:34:41,738
Like and

474
00:34:41,738 --> 00:34:42,359
Then they don't.

475
00:34:42,359 --> 00:34:51,166
It's like instead you see them have contracts with the hype other hyperscalers, and then
you see the hyperscalers create ridiculous boxes or any company that does large-scale um

476
00:34:51,166 --> 00:34:52,457
mobile device validation.

477
00:34:52,457 --> 00:35:02,455
And not necessarily for testing applications on the device, but sometimes using them to
interact with the actual whatever it is, so any sort of robotic process automation.

478
00:35:02,455 --> 00:35:03,926
And they're they're quite complex.

479
00:35:03,926 --> 00:35:11,062
It's like a server rack size sort of thing where the mobile device sits in there and then
the rest of it is sort of uh pneumatic

480
00:35:11,308 --> 00:35:16,298
you know, gadgets interact with it because you can't interact with the device from a
software standpoint in any way.

481
00:35:16,298 --> 00:35:21,109
Uh and now I think Google is taking away ADB on Android for non rooted devices.

482
00:35:21,109 --> 00:35:23,311
And you're like, well, that's it.

483
00:35:23,311 --> 00:35:29,313
Like everything is I need like a you know a pressure to actually uh activate, you know,
mobile parts of the device.

484
00:35:29,313 --> 00:35:37,345
And then you're not just dealing with the the issues of the physical device that you're
working with, but now the physical device that you've built to interact with that.

485
00:35:37,366 --> 00:35:39,244
I don't envy this situation.

486
00:35:39,244 --> 00:35:41,916
No, I and it it's really fascinating.

487
00:35:41,916 --> 00:35:47,881
So like I've been in conversations with both people from Apple and Google going, like we
need to solve this.

488
00:35:47,881 --> 00:35:54,046
We need a simple way, 'cause like in browsers it's a solved it's a relatively solved
problem, right?

489
00:35:54,046 --> 00:36:05,335
Like we've got the W three C doing um the web driver specifications and the W three C like
have these rules of like you need two implementations of anything to become a standard.

490
00:36:05,335 --> 00:36:08,928
To get two implementations uh in kind of

491
00:36:08,928 --> 00:36:15,391
In the mobile space, um, other than browsers, you would have to have Google and Apple both
agree.

492
00:36:15,391 --> 00:36:19,633
Um, and they've both gone, Well, this is a moneymaker for us.

493
00:36:19,633 --> 00:36:21,784
We have our ways, that's your problem.

494
00:36:21,784 --> 00:36:30,348
And that's why like the people who work on Appium, the open source project, they are
literally doing God's work to try solve all of these kind of problems.

495
00:36:30,348 --> 00:36:31,928
Um, but it's never gonna happen.

496
00:36:31,928 --> 00:36:36,264
So the other side of it is there's um the guy who created Selenium.

497
00:36:36,264 --> 00:36:37,515
his name's Jason Huggins.

498
00:36:37,515 --> 00:36:40,157
He created another company called Tapster.

499
00:36:40,518 --> 00:36:50,847
and Tapster is doing what you said, which is testing mobile devices, but he's built like
got three D printers and Arduinos to figure out where certain things should go and then

500
00:36:50,847 --> 00:36:54,966
he's got a pen driving the mobile device for testing.

501
00:36:54,966 --> 00:37:03,431
bit of a sobering story because like no matter how bad you think writing tests are and
having no correct way of doing it in the browser, there is another whole set of

502
00:37:03,431 --> 00:37:08,309
applications, a platform out there where, you know, good luck.

503
00:37:08,354 --> 00:37:09,114
Yeah.

504
00:37:09,114 --> 00:37:17,278
if people want to see other horror stories, um I highly recommend looking at the
differences between screen readers between operating systems.

505
00:37:17,278 --> 00:37:19,778
That's another nightmare in itself.

506
00:37:19,778 --> 00:37:28,292
So Linux will do one way, Windows will do another way, uh Mac OS will do another way, iOS
will do another way, Android will do another way.

507
00:37:28,292 --> 00:37:28,750
Uh

508
00:37:28,750 --> 00:37:31,781
I'm very thankful for anyone who does work in the accessibility space.

509
00:37:31,781 --> 00:37:40,788
I as someone who doesn't super need it myself, but does maintain some open source
software, when we have users come to us and be like, Hey, this is broken, uh, I'll I can

510
00:37:40,788 --> 00:37:42,579
make a pull request to fix it.

511
00:37:42,579 --> 00:37:47,693
And then I learn about a a part of the web that I had no desire to and it's horrific.

512
00:37:47,693 --> 00:37:51,145
Like it is not it is not nice stuff to overcome these things.

513
00:37:51,145 --> 00:37:58,490
And I think part of the challenge is, as you mentioned it earlier on, we just kept
stacking stuff on top of history.

514
00:37:58,661 --> 00:38:01,295
in effort not to break things, et cetera.

515
00:38:01,295 --> 00:38:11,032
And it makes me wonder, do you think that we would have ended up in a much different
position than we are now if we, you know, started to invent the tech today?

516
00:38:11,032 --> 00:38:17,768
So I wish I could say yes, uh, but I think I think we would do all the same mistakes,
right?

517
00:38:17,768 --> 00:38:20,310
Our systems are just improving exponentially.

518
00:38:20,310 --> 00:38:24,062
That's a a problem for tomorrow, from testing to performance to security.

519
00:38:24,062 --> 00:38:28,926
We now have an interesting problem of like, do we solve quantum in the next 10 years?

520
00:38:28,926 --> 00:38:31,218
How's that going to handle for security?

521
00:38:31,218 --> 00:38:32,059
Who knows?

522
00:38:32,059 --> 00:38:32,409
Right?

523
00:38:32,409 --> 00:38:36,513
Like, is it gonna break, you know, crypto to kind of banks, right?

524
00:38:36,513 --> 00:38:38,154
It's gonna be everything in between.

525
00:38:38,154 --> 00:38:40,876
Um, and so there's interesting spaces there.

526
00:38:40,876 --> 00:38:45,439
I think we've lived through a lot of the golden age of things, to be brutally honest.

527
00:38:45,439 --> 00:38:57,154
I think kind of we've got this really interesting space now where like from say 2000 to
2020, loads of people were doing open source and open source allowed us to push new ideas

528
00:38:57,154 --> 00:38:59,036
into wonderful spaces.

529
00:38:59,036 --> 00:39:04,079
And then in the last six years, that has contracted so much.

530
00:39:04,079 --> 00:39:09,632
Um, I'm not sure if we had start if we start now, would that be a case?

531
00:39:09,632 --> 00:39:13,096
If we started twenty ten, maybe we could have done something different.

532
00:39:13,096 --> 00:39:18,353
But I think kind of then I think people always go, 'Cause like humans are inherently lazy,
right?

533
00:39:18,353 --> 00:39:20,775
Like I'll I'll I'll do enough.

534
00:39:21,617 --> 00:39:22,712
And then carry on.

535
00:39:22,712 --> 00:39:31,738
You're definitely spot on there that some of the challenges that led us to the current
state, you know, unless you eliminate those, we still likely end up in a in a similarly

536
00:39:31,738 --> 00:39:32,849
challenging spot.

537
00:39:32,849 --> 00:39:39,633
Uh or maybe we don't have any of the current problems, but we just how long it was.

538
00:39:39,736 --> 00:39:40,351
Yeah.

539
00:39:40,351 --> 00:39:42,077
Yeah, definitely.

540
00:39:42,082 --> 00:39:42,922
Yeah.

541
00:39:43,183 --> 00:39:47,802
browsers are in this weird space where they're almost really great operating systems.

542
00:39:47,802 --> 00:39:50,648
They're pretty much functioning as that mechanism.

543
00:39:50,648 --> 00:39:55,392
And like on uh mobile devices, they are almost fully that.

544
00:39:55,392 --> 00:39:59,635
We have very little control over the OS there from a user standpoint.

545
00:39:59,635 --> 00:40:09,644
And so if we look at applications as like mobile experience, websites, then it's almost
entirely uh websites and the browser experience.

546
00:40:09,644 --> 00:40:10,054
I don't know.

547
00:40:10,054 --> 00:40:11,605
Like I'm of I'm of two minds here.

548
00:40:11,605 --> 00:40:20,858
I guess my perspective is I would love it if the all the sandboxing effort, everything for
interacting with websites really became a full experience.

549
00:40:21,208 --> 00:40:31,662
but then I still see what application developers are doing if I look at, you know, you
look at the chat tools we have or email clients that are available, and these are like

550
00:40:31,662 --> 00:40:33,410
literally the the worst.

551
00:40:33,410 --> 00:40:34,290
Yeah.

552
00:40:34,711 --> 00:40:40,896
No, so like I I was a big advocate uh for kind of the idea of a PWA.

553
00:40:40,896 --> 00:40:43,779
I was always a big advocate because I thought it was really interesting.

554
00:40:43,779 --> 00:40:50,704
And like part of my work through W3C, um I get to see some of the interesting bits and
pieces that are happening at the web level.

555
00:40:50,704 --> 00:40:54,808
Um Google and Microsoft had this project called Project Fugu.

556
00:40:54,808 --> 00:40:59,752
And so it's based off this idea of the sushi fugu blowfish.

557
00:40:59,752 --> 00:41:02,658
But you've got to make sure you get the best part of the blowfish.

558
00:41:02,658 --> 00:41:03,738
Otherwise you get poisoned.

559
00:41:03,738 --> 00:41:10,964
Uh and so that's why it was called Fugu, because they were trying to take some of the
concepts of mobile and put it into the web.

560
00:41:10,964 --> 00:41:14,867
So Bluetooth, address book, USB, all of that.

561
00:41:14,867 --> 00:41:17,068
Um some of that sounds awesome.

562
00:41:17,068 --> 00:41:19,606
Part of that also sounds incredibly scary, right?

563
00:41:19,606 --> 00:41:20,289
the project name.

564
00:41:20,289 --> 00:41:23,478
Like I I I you know, as soon as you call it that I I feel like you you already know where

565
00:41:23,478 --> 00:41:24,078
It's going.

566
00:41:24,078 --> 00:41:25,850
Yeah, exactly.

567
00:41:25,850 --> 00:41:30,813
We're all taught do not stick random USBs into your kind of computer.

568
00:41:30,813 --> 00:41:36,047
Uh imagine if you stuck a random USB and a your browser goes, I've got a new USB.

569
00:41:36,047 --> 00:41:38,038
Uh let me see what's inside it.

570
00:41:38,038 --> 00:41:41,571
And then suddenly like all your banking information and everything is stolen, right?

571
00:41:41,571 --> 00:41:46,316
Like it there's some because browsers try to hide all of that stuff in

572
00:41:46,516 --> 00:41:50,299
encrypted parts like the BF cache, they try encrypt a lot of that stuff.

573
00:41:50,299 --> 00:41:53,972
So no one can just randomly steal the stuff if they're on the computer.

574
00:41:53,972 --> 00:41:56,834
Um and so Project Fuga's trying to do that.

575
00:41:56,834 --> 00:41:59,866
I think there are good parts to it and there are bad parts to it.

576
00:41:59,866 --> 00:42:05,601
So there is an interesting space there that I think we should try make the web better on
mobile.

577
00:42:05,601 --> 00:42:10,704
But then we come back to the original problem that we were talking about earlier of like
how do you test on mobile devices?

578
00:42:10,704 --> 00:42:13,290
An interesting thing is that I can s

579
00:42:13,290 --> 00:42:22,678
shrink my browser on my desktop to that right size, but it doesn't mean it's going to
render that way on a mobile device because the way rendering w works on my Mac or my

580
00:42:22,678 --> 00:42:27,402
Windows machine is very different to a mobile device, um, by design, right?

581
00:42:27,402 --> 00:42:35,639
But people go, that's this is good enough and you're like, it it it is, but it's not at
least admit there's going to be differences and if you're okay with those differences,

582
00:42:35,639 --> 00:42:36,190
then fine.

583
00:42:36,190 --> 00:42:36,568
ah

584
00:42:36,568 --> 00:42:38,459
There there are some of them are not trivial.

585
00:42:38,459 --> 00:42:41,209
Like this is just a warning for anyone doing this today.

586
00:42:41,209 --> 00:42:46,781
Like, even if you change the size, even if you go to the developer tools and click the,
you know, convert to responsive mode.

587
00:42:46,781 --> 00:42:55,723
And even if you change out of responsive mode and select specific like phone sizes, uh
mobile device sizes to use, you're still not getting the gestures to work correctly.

588
00:42:55,723 --> 00:43:06,048
And the biggest ones that I've seen break are uh like media query sizes, uh or like VH or
like partial heights there, because the mobile apps will do.

589
00:43:06,048 --> 00:43:16,007
ridiculous things to the size of the viewport when you change the size the application on
mobile because you can support that or the keyboard pops up and then the height that's

590
00:43:16,007 --> 00:43:17,547
remaining changes.

591
00:43:17,668 --> 00:43:19,173
It is it is not the same.

592
00:43:19,173 --> 00:43:30,629
I am I like that there is like very it's very difficult to debug without plugging your
phone into a computer somewhere and like exporting some ADB or getting the uh inspector to

593
00:43:30,629 --> 00:43:34,472
run specifically for the application in order to actually see what's going on.

594
00:43:34,518 --> 00:43:35,929
Yeah, it's it's fascinating.

595
00:43:35,929 --> 00:43:44,765
I've been telling people that for years and most people go, No, David, like just chill,
you're okay and they're like, No, it is very different and it's nice and refreshing to

596
00:43:44,765 --> 00:43:47,266
have someone go, David knows what he's talking about

597
00:43:47,522 --> 00:43:48,062
Yeah.

598
00:43:48,062 --> 00:43:49,934
and it's not just on the display side.

599
00:43:49,934 --> 00:44:01,391
There's a lot of things with like how the security keys for I would say pass keys, but
FIDO two, web auth and stuff, will say that it works the same everywhere, but on device.

600
00:44:01,391 --> 00:44:12,128
No, like I can use my phone as the pass key, but if I'm run a mobile web s like a website
or an application and try to use the pass key like on device on it, it doesn't work.

601
00:44:12,258 --> 00:44:17,981
Uh or like on Bluetooth it works, but if you try to use a special connection where you
scan a QR code and does it doesn't go.

602
00:44:17,981 --> 00:44:27,106
And it's not it's not just the implementation of Web Auth N that some application
developer turned off because you can specify for Web Auth N like only blue Bluetooth is

603
00:44:27,106 --> 00:44:34,580
allowed, uh or only like removable like physical hardware is allowed, like a UB key or
whatever or whatever the other ones are called.

604
00:44:34,580 --> 00:44:40,273
So assuming they didn't do any of that, which could have caused a problem, there's still
additional other issues which it just don't work.

605
00:44:40,293 --> 00:44:41,632
And that's like

606
00:44:41,632 --> 00:44:43,581
a very small part of the story.

607
00:44:43,581 --> 00:44:44,363
So

608
00:44:44,588 --> 00:44:45,822
Yeah, definitely.

609
00:44:45,822 --> 00:44:47,636
Hundred percent agree with you.

610
00:44:47,970 --> 00:44:48,450
Yeah.

611
00:44:48,450 --> 00:44:50,812
So, you know, that that that's the future there.

612
00:44:50,812 --> 00:44:57,095
I I think maybe Chrome OS was this, to try to expand the browser into being the full
operating system in a way.

613
00:44:57,275 --> 00:44:59,483
obviously it took off for a a group of people there.

614
00:44:59,483 --> 00:45:03,318
I certainly always like the idea of a thin client to do the right thing.

615
00:45:03,318 --> 00:45:14,685
And I feel like one of the huge challenges with browser stuff and when we had Sam go to on
here from Google um in a previous episode, he was talking a lot about the security

616
00:45:14,685 --> 00:45:17,556
challenges, uh especially in the browser world.

617
00:45:17,734 --> 00:45:19,264
And it was that was that was interesting.

618
00:45:19,264 --> 00:45:24,176
So anyone who wants to go more in the security space, there is that there is that episode
which we'll link in the description.

619
00:45:24,176 --> 00:45:33,908
But I will say that one of the huge challenges was like sort of jumping over this gap of
the operating system, because the the browser is so limited because it doesn't have full

620
00:45:33,908 --> 00:45:34,339
control.

621
00:45:34,339 --> 00:45:37,519
It's not like a super user running on device.

622
00:45:37,619 --> 00:45:46,782
And yeah, but on the flip side, as you mentioned, there's the whole problem there of
because you bridge that gap, now the issues go back the other way.

623
00:45:46,802 --> 00:45:47,562
And

624
00:45:47,650 --> 00:45:55,586
Where the applications running in the browser would have been insulated from things that
connect to the operating system where sort of increasing that uh attack vector basically.

625
00:45:55,586 --> 00:46:05,342
And I think the easy way for kind of non security people is the wherever kind of two
systems meet, that's where the security problems will be there.

626
00:46:05,342 --> 00:46:08,066
And it's not a if they will be there, they will be there.

627
00:46:08,066 --> 00:46:08,408
And

628
00:46:08,408 --> 00:46:11,509
those are where your test break too, because it's where all your assumptions lie, right?

629
00:46:11,509 --> 00:46:14,000
Like the expectations of whatever that other interface goes.

630
00:46:14,000 --> 00:46:23,913
I feel like one of the questions that always come up is should you include in your testing
the actual third party service or library, not just their interface, but the actual

631
00:46:23,913 --> 00:46:25,183
implementation there.

632
00:46:25,183 --> 00:46:33,596
And I've been going back and forth over this on a lot of for a lot of years, like, okay,
you would want to respect the tests of that third party that you trust for the open source

633
00:46:33,596 --> 00:46:34,488
that they make.

634
00:46:34,488 --> 00:46:36,389
That it does the right thing that it expects.

635
00:46:36,389 --> 00:46:39,301
And if it changes, you can see those tests that change.

636
00:46:39,301 --> 00:46:46,726
But on the flip side, people make non-semantic version changes, call it semantic version
changes that will break your application and your usage.

637
00:46:46,917 --> 00:46:49,038
the interface doesn't match what actually the implementation is.

638
00:46:49,038 --> 00:46:51,690
Uh, here's an example that happens a lot for us.

639
00:46:51,690 --> 00:47:03,618
We found out that there's some great libraries in in Python which like to complain if the
response format from an API doesn't match the classes that it's it's defined.

640
00:47:03,618 --> 00:47:05,680
And I'm just like, don't throw on that.

641
00:47:05,680 --> 00:47:10,343
Like, there's no reason to throw an error because it doesn't match on a field that you
don't even care about.

642
00:47:10,343 --> 00:47:12,545
But a lot of libraries do this.

643
00:47:12,645 --> 00:47:16,098
And there is this fear that if you don't test it, something can change and break.

644
00:47:16,098 --> 00:47:21,432
But if you do test, you're wasting a lot of resources, potentially we talked about, but
also time and effort.

645
00:47:21,432 --> 00:47:26,296
And you can't narrow your test down just to the part of the code that you you want to
think about.

646
00:47:26,296 --> 00:47:28,948
So what's the what's the advice here?

647
00:47:30,082 --> 00:47:32,703
I think the the advice really is back to that.

648
00:47:32,703 --> 00:47:35,165
Let's write one test and then go from there.

649
00:47:35,165 --> 00:47:37,535
Right?

650
00:47:37,535 --> 00:47:38,576
Right.

651
00:47:38,576 --> 00:47:40,987
So small, medium, large tests.

652
00:47:40,987 --> 00:47:45,069
I only test what like the where I think the risk is.

653
00:47:45,069 --> 00:47:46,309
And I go from there.

654
00:47:46,309 --> 00:47:58,182
I don't I don't need to test everything because uh you know, if there's a security problem
or kind of performance or any other bugs that happen in that system, um

655
00:47:58,486 --> 00:48:02,017
The next time I update, I'll likely be fixed, right?

656
00:48:02,017 --> 00:48:05,628
And then do I want to maintain all of those tests in the first place?

657
00:48:05,628 --> 00:48:08,089
So let's just focus on one test.

658
00:48:08,089 --> 00:48:12,750
Um, and that one test is the thing I care about for that area, right?

659
00:48:12,750 --> 00:48:17,151
And it could be purely down to one API call or whatever, right?

660
00:48:17,151 --> 00:48:22,813
But I think kind of that mentality of just focusing your where the value is, right?

661
00:48:22,813 --> 00:48:26,604
You don't want to have to go and update twenty tests.

662
00:48:26,770 --> 00:48:33,554
yes, you can do it a lot faster nowadays with an LLM, but then like you've got to make
sure that it's still doing the right thing afterwards, right?

663
00:48:33,554 --> 00:48:39,037
Where if you've got one test and an LM makes a change, you can review it quickly um and
then move on.

664
00:48:39,037 --> 00:48:46,121
A lot of the problems that I've seen um are not necessarily with the third party
frameworks or whatever people are using.

665
00:48:46,121 --> 00:48:49,143
It's how people are using those frameworks.

666
00:48:49,143 --> 00:48:53,005
I was working on Browser Stack's um test management tool.

667
00:48:53,005 --> 00:48:54,442
It uses React.

668
00:48:54,442 --> 00:49:01,027
When we were developing it and we got to this one point where UI updates were super, super
slow.

669
00:49:01,027 --> 00:49:07,471
We had loads of component tests, we had loads of unit tests, we had everything, um, and we
had a bunch of end-to-end tests.

670
00:49:07,471 --> 00:49:09,693
Um, none of them were flaky.

671
00:49:09,693 --> 00:49:12,495
Um, we noticed that they were taking a long time.

672
00:49:12,495 --> 00:49:20,461
And what it was is that like whenever data would update this one component, it would cause
the entire page to refresh.

673
00:49:20,461 --> 00:49:23,022
And the tests would be like, okay, cool, cool.

674
00:49:23,136 --> 00:49:25,628
I I've handled this, so there's no failure.

675
00:49:25,628 --> 00:49:26,428
We're good.

676
00:49:26,428 --> 00:49:28,490
But that's a performance problem, right?

677
00:49:28,490 --> 00:49:30,031
And it's not a React problem.

678
00:49:30,031 --> 00:49:31,141
It's not a whatever.

679
00:49:31,141 --> 00:49:33,173
It was a like our developers.

680
00:49:33,173 --> 00:49:40,227
When we started putting all of these things together, because everyone was working on the
individual parts and then one person was like, I'll smush them all together.

681
00:49:40,227 --> 00:49:41,134
Hooray.

682
00:49:41,134 --> 00:49:49,983
I feel like uh architectures and frameworks which cause the wrong thing to happen in the
long term, which are very difficult to identify, can be the wrong thing for your business

683
00:49:49,983 --> 00:49:59,903
or your technology stack, et cetera, and take years to sort of identify by an expert who's
been in the in that area, that there is a fundamental problem with that technology that

684
00:49:59,903 --> 00:50:01,274
you'll keep running into.

685
00:50:01,274 --> 00:50:05,750
I feel like that's another episode, and I think I may need a specific guest to come on to
talk to that.

686
00:50:05,750 --> 00:50:14,726
Okay, then then you know, so there's there's one last area that I I definitely want to get
into and I haven't read the the spec or wherever the proposal is at, but um I think you

687
00:50:14,726 --> 00:50:24,373
brought up LLMs a couple of times and I try to avoid anything related to AI because it's
uh it can be a rat hole or everyone's worst nightmare, or I'm sure there's some listener

688
00:50:24,373 --> 00:50:27,065
who's jumping up for joy right now because I'm gonna bring this up.

689
00:50:27,065 --> 00:50:35,180
But it does seem like the ability to interact cross-device or cross-process in a way that
can support LLM interactions.

690
00:50:35,180 --> 00:50:38,541
Is something that's gone uh avoided for long for too long.

691
00:50:38,541 --> 00:50:45,243
I I think a lot of them when you want to interact with a particular web page or
application uh on the mobile device, it's just not possible today.

692
00:50:45,243 --> 00:50:48,094
And I have hopes in the future it will be.

693
00:50:48,094 --> 00:50:57,657
But in the browsers, I saw this thing called Web C P and when I hear that it makes me
think the right things are gonna happen.

694
00:50:57,657 --> 00:51:01,262
And I have no love for LLMs, but I I do see the the value there.

695
00:51:01,262 --> 00:51:06,427
Is an interesting potential overlap between testing and how people do it.

696
00:51:06,427 --> 00:51:13,212
So WebMCP is this set of APIs that allow you to build an MCP into a web page.

697
00:51:13,233 --> 00:51:17,997
And so MCPs are these tools that you can LLMs know how to interact with.

698
00:51:17,997 --> 00:51:28,586
um

699
00:51:28,586 --> 00:51:30,797
flights between this place and this place, right?

700
00:51:30,797 --> 00:51:35,229
And you just tell your LM, I need you to go to Kayak, do this.

701
00:51:35,229 --> 00:51:38,930
You load c it loads kayak in a browser, it goes, Oh, there's a tool.

702
00:51:38,930 --> 00:51:40,851
I don't need to try to figure out the UI.

703
00:51:40,851 --> 00:51:42,272
I can just speak to the tool.

704
00:51:42,272 --> 00:51:44,363
The tool will then figure out the UI for me.

705
00:51:44,363 --> 00:51:45,013
Right.

706
00:51:45,013 --> 00:51:54,917
And that's where um MCPs are really good is that you can have these set of tools um and
they can do very specific things really well because they've been programmed that like to

707
00:51:54,917 --> 00:51:57,058
do small succinct things.

708
00:51:57,058 --> 00:52:10,739
The thing I like about Web MCP is more of a mindset thing because you don't want to have
uh like I'm sure we've all tried to install a like say GitHub MCP and you do it and it'll

709
00:52:10,739 --> 00:52:14,272
go, There are now ten tools available, right?

710
00:52:14,693 --> 00:52:26,582
If you create a web MC like web MCP and you load it up and your LM finds it and goes,
there's an MCP, let me load the tools, there are three hundred and two.

711
00:52:26,838 --> 00:52:30,359
as a random number, the value of it disappears really quickly.

712
00:52:30,359 --> 00:52:36,062
And so to me, it's like, well, you would only have a tool for where the high value traffic
should be going.

713
00:52:36,062 --> 00:52:43,965
The the high risk, the things that you want people to do correctly, first time correct and
so you could do that.

714
00:52:43,965 --> 00:52:45,035
I've been playing with that.

715
00:52:45,035 --> 00:52:47,727
Um I built a a tool.

716
00:52:47,727 --> 00:52:56,290
It uses WebNN, so the uh web neural network APIs built into uh Chromium to then generate
an MCP.

717
00:52:56,448 --> 00:52:59,989
And so that it can go, hey, these are the high traffic areas.

718
00:52:59,989 --> 00:53:04,413
Um, it's got a whole bunch of heuristics um around what it considers valuable.

719
00:53:04,413 --> 00:53:07,636
So like if it sees a header, it'll go, that's not valuable.

720
00:53:07,636 --> 00:53:10,869
If it sees a a link, it'll go, that is valuable, right?

721
00:53:10,869 --> 00:53:12,630
And tries to balance it all out.

722
00:53:12,630 --> 00:53:17,884
On mobile, there's things like called intents, at least on Android, that allow you to
switch between applications.

723
00:53:17,884 --> 00:53:20,236
But most apps don't expose them.

724
00:53:20,236 --> 00:53:24,058
And even if they did, other apps don't actually try to utilize them.

725
00:53:24,058 --> 00:53:33,265
But I can see a lot of opportunities there with wanting to interact with the applications
or the value that your phone is providing without going through the first class uh UI.

726
00:53:33,265 --> 00:53:39,980
And the browser side, I worry what's happening is where bypassing the value of having a
first class API.

727
00:53:39,980 --> 00:53:43,692
In the first like you say the GitHub MCP, you know, wasn't great.

728
00:53:43,692 --> 00:53:49,225
And then people spin up their own MCPs on top of GitHub for a very specific value that
they want to extract from the API.

729
00:53:49,225 --> 00:53:50,956
That's both good and bad.

730
00:53:50,956 --> 00:54:00,951
Uh being able to directly define what your value is and what the use cases that you will
need it for and optimize it, you know, that's good rather than having, you know, a hundred

731
00:54:00,951 --> 00:54:03,073
thousand different tools that are exposed.

732
00:54:03,073 --> 00:54:09,602
And then you have this whole problem of how does the LLM even make sure it's picking the
right tool with the right parameters to call it in the first place.

733
00:54:09,602 --> 00:54:10,303
Yeah.

734
00:54:10,303 --> 00:54:20,731
But at least it would force GitHub to think about how to expose that data in the right way
and not add in an extra complexity of a whole user session with an expensive application

735
00:54:20,731 --> 00:54:24,895
like a browser in order to just have an LM on your machine interact with it.

736
00:54:24,895 --> 00:54:34,282
And not to mention, if you don't have an LLM on your machine because you have a GPU that
was more than one year old at this point, you're gonna be running you could now have to

737
00:54:34,282 --> 00:54:38,676
run an expensive browser process in a virtual machine somewhere and interact with the

738
00:54:38,676 --> 00:54:40,547
MCP that's created in that way.

739
00:54:40,547 --> 00:54:42,179
But on the flip side, I get it, right?

740
00:54:42,179 --> 00:54:51,037
Not every application developer understands how to build a first class API that can be
consumed and then expose it with the right documentation, et cetera.

741
00:54:51,037 --> 00:54:56,261
And if you're doing anything interesting, uh that maybe that website is just a static
page.

742
00:54:56,261 --> 00:55:06,358
And being able to effectively pull out the information anyway is going to be much cheaper
for everyone than to force it to parse the actual uh DOM tree that is.

743
00:55:06,358 --> 00:55:10,063
is available at that moment in order to skip the HTML and get out the parts.

744
00:55:10,063 --> 00:55:12,176
Because as you mentioned, it's not just the number of tokens.

745
00:55:12,176 --> 00:55:17,566
There's a lot of formatting there for human benefit that offers zero for actual value.

746
00:55:17,566 --> 00:55:26,815
The one thing to remember is Web C P is relying on the web developer of that site to
create the right tools in the right space.

747
00:55:27,417 --> 00:55:36,152
and if they like this is where I think the interesting part from the testing point of view
is like if they don't do that part correctly, they're probably not doing the testing part.

748
00:55:36,152 --> 00:55:37,123
correctly either.

749
00:55:37,123 --> 00:55:38,664
They just don't realize it.

750
00:55:38,664 --> 00:55:41,326
And I and it's and it's not a shade on anyone, right?

751
00:55:41,326 --> 00:55:44,589
Because like both of those things are incredibly hard, right?

752
00:55:44,589 --> 00:55:50,894
Like there are people who focus their whole career on user experience, but this is a
different type of user experience, right?

753
00:55:50,894 --> 00:55:53,887
Um so it's not a natural thing to do.

754
00:55:53,887 --> 00:55:54,928
It's the same with testing.

755
00:55:54,928 --> 00:55:59,992
Like lots of people focus on testing, not developing, because like that's where their
skill sets are.

756
00:55:59,992 --> 00:56:11,762
Yeah, actually in the episode we did on the Wikimedia Foundation and open source Wikipedia
tools, uh our the guest Morel actually uh really talked to us about the benefit of

757
00:56:11,762 --> 00:56:15,745
thinking about what how your users are actually utilizing your application.

758
00:56:15,745 --> 00:56:20,279
And if they want to use it through this way, they're still your user, you're still getting
the value out.

759
00:56:20,279 --> 00:56:23,201
Just the value isn't the UI structure.

760
00:56:23,201 --> 00:56:29,390
And so don't don't effectively curtail or limit a potential user because they have a
unique way of

761
00:56:29,390 --> 00:56:29,971
approaching it.

762
00:56:29,971 --> 00:56:34,035
I think it was a pretty interesting insight that I hadn't can I hadn't considered.

763
00:56:34,035 --> 00:56:39,300
And I think it's a much smarter and mature approach than saying we don't want LLMs to use
our website.

764
00:56:39,300 --> 00:56:41,782
It's you have to admit you're never going to stop that.

765
00:56:41,782 --> 00:56:44,845
Whether or not it's right or not, there are some users who want to do that.

766
00:56:44,845 --> 00:56:54,228
And the question is do you want a user that is trying to use or you can get the value out
of your your site or application or tell them to go use a competitor who is more open to

767
00:56:54,228 --> 00:56:55,920
I I could have a whole conversation about this.

768
00:56:55,920 --> 00:57:03,757
Like we ran into tons of trouble with our documentation where like links in Markdown could
be relative, but do you want the links in the LMs.txt to be absolute and like what's the

769
00:57:03,757 --> 00:57:07,871
actual domain and kind of piece those things together, especially if you have some sort of
base path.

770
00:57:07,871 --> 00:57:12,456
But I'll I'll I'll leave that for uh for another episode potentially.

771
00:57:12,456 --> 00:57:16,800
And uh maybe now is a good time to move over to picks.

772
00:57:16,800 --> 00:57:19,752
So David, what did you bring for the audience today?

773
00:57:19,798 --> 00:57:28,721
Uh so um I don't have it with me right now, but the thing that I wanted to talk about was
that um I'm a big into uh retro gaming.

774
00:57:29,721 --> 00:57:41,205
I am an elder millennial, uh so I still I still like to play kind of these older games
where uh I kind of have uh so I play a lot of Pokemon.

775
00:57:41,225 --> 00:57:42,326
That's one of my favorites.

776
00:57:42,326 --> 00:57:42,978
Um

777
00:57:42,978 --> 00:57:46,894
I didn't expect you to say that onto the list of all the retro kicks.

778
00:57:46,894 --> 00:57:49,285
Uh so like I play a lot of Pokemon.

779
00:57:49,285 --> 00:57:51,036
Um I've been a Pokemon fan.

780
00:57:51,036 --> 00:58:04,425
Um and so I have there's a company based in China called Amber Nick and they ship a like
device that's shaped like the old retro devices.

781
00:58:04,425 --> 00:58:13,630
So if you're like want a Nintendo, uh like the old Nintendo, I've got a um a Lego version
here on so I'll show that to the screen.

782
00:58:13,630 --> 00:58:16,712
it's a Lego version of the original Game Boy because I really like it.

783
00:58:16,712 --> 00:58:22,215
Um, but there's also the Nintendo SP, which was the one of the flip ones.

784
00:58:22,215 --> 00:58:28,288
Um so I I like uh collecting those and then collecting all the ROMs to uh play it.

785
00:58:28,288 --> 00:58:37,783
So kind of but I play everything from Mega Man to Pokemon to kind of full metal solid,
like all of these things.

786
00:58:37,783 --> 00:58:43,438
Cause like one of the things I liked about gaming back uh say

787
00:58:43,438 --> 00:58:48,202
20 years ago is that like there wasn't you couldn't just use the internet to download
updates.

788
00:58:48,202 --> 00:58:48,532
Right.

789
00:58:48,532 --> 00:58:52,405
So games had to be fully shaped and correct.

790
00:58:52,405 --> 00:58:54,127
And if there were bugs, there were bugs, right?

791
00:58:54,127 --> 00:58:56,198
Um and that was it, right?

792
00:58:56,198 --> 00:58:59,441
And you just kind of had these things that you could just play.

793
00:58:59,441 --> 00:59:05,446
Um and they were designed for you to like play for hours and hours or kind of very
quickly, right?

794
00:59:05,446 --> 00:59:12,141
Where I I'm not a big fan of kind of microtransactions, things like that are that are in
games nowadays.

795
00:59:12,141 --> 00:59:12,920
So

796
00:59:12,920 --> 00:59:23,466
You reminded me of this great video about the manufacturing of a of a old retro Prince of
Persia game and I there's quite some interesting lore there uh about what you have to do

797
00:59:23,466 --> 00:59:28,979
when you only have so many bits available of of memory and processing power to actually
generate things.

798
00:59:28,979 --> 00:59:34,863
And the genius that is of the the inverse fast square root uh in in Quake, right?

799
00:59:34,863 --> 00:59:40,684
No one would would bother trying to identify what that is and understand how that even
executes and maybe we'll

800
00:59:40,684 --> 00:59:45,219
And if you don't know what those things are, I'm sure I'll have to find a good YouTube
video on on on both of those.

801
00:59:45,219 --> 00:59:50,003
I had this conversation actually not too long ago about like what classifies something as
a retro game.

802
00:59:50,003 --> 00:59:56,109
And I feel like is it the eight bit shift like before after eight bit it's no longer
retro?

803
00:59:56,109 --> 01:00:01,364
Or is there like a particular year for you uh or type of game that falls into this
category?

804
01:00:01,726 --> 01:00:09,230
kind work how the but it it's kind of like anything from Nintendo advance backwards.

805
01:00:09,810 --> 01:00:11,371
Okay in that range, right?

806
01:00:11,371 --> 01:00:15,803
Uh my kids will tell you kind of anything before twenty twenty.

807
01:00:16,554 --> 01:00:17,675
it's probably retro.

808
01:00:17,675 --> 01:00:22,948
Uh we have arguments, but it's it's that type of thing.

809
01:00:22,948 --> 01:00:23,638
and it's

810
01:00:23,638 --> 01:00:24,534
They're allowed to be wrong.

811
01:00:24,534 --> 01:00:25,676
They're just kids.

812
01:00:25,676 --> 01:00:26,446
Yeah, exactly.

813
01:00:26,446 --> 01:00:35,249
Um and so like but it's for me it's like Game Boy Advance and it's those games like I say,
it's that point where you couldn't just download fixes.

814
01:00:35,249 --> 01:00:47,702
uh so kind of remember being like just spending hours uh as a teenager just like tr
grinding, where now it's like, you know, as an adult I've got responsibilities and when I

815
01:00:47,702 --> 01:00:55,052
do play I can like do these things and do it quickly, um, and still come back to it later,
uh, with not having felt like I've

816
01:00:55,052 --> 01:00:58,488
Wasted in the whole evening 'cause like I'm just old and tired.

817
01:00:58,488 --> 01:01:01,899
play Pokemon uh go by uh by Niantic.

818
01:01:01,899 --> 01:01:11,573
They had a previous iteration called Ingress, which their whole point was to capture uh
geolocation data and pictures from around the world uh before they came up with Pokemon.

819
01:01:11,573 --> 01:01:15,365
And then they just released recently that they had been doing it with Pokemon.

820
01:01:15,365 --> 01:01:26,159
So basically anyone who's playing the game has been funneling the LLM training machine in
order based off of where they go and their phone orientation and any pictures that they've

821
01:01:26,159 --> 01:01:27,958
taken, which is

822
01:01:27,958 --> 01:01:34,958
I mean I I knew they had some genius strategy, but it's quite uh may lose some of the
Pokemon fandom.

823
01:01:34,958 --> 01:01:45,737
But that that's how Google Maps got all that information 'cause um originally uh Niantic
was an internal company within Google and then they bought themselves out of Google and

824
01:01:45,737 --> 01:01:46,768
then carried on.

825
01:01:46,768 --> 01:01:49,036
But they we still had Google as a customer.

826
01:01:49,036 --> 01:02:00,321
You know, honestly, for interesting ways of collecting uh data for training LLM agents as
as it relates to sort of the whole software engineering pipeline, I'd love to get a guest

827
01:02:00,321 --> 01:02:00,831
on the show.

828
01:02:00,831 --> 01:02:08,474
So if anyone listening has someone who they think would recommend for this particular
segment, I I'd be happy to uh connect with them.

829
01:02:09,515 --> 01:02:16,002
I think I have to share my pick, which I don't know if it's gonna be as riveting, uh, but
it's this paper that was released.

830
01:02:16,002 --> 01:02:23,647
by m my university uh Cornell called the bullshit the corporate bullshit receptivity
scale.

831
01:02:23,767 --> 01:02:31,552
And it basically is a pure metric for calculating how BS a particular statement is.

832
01:02:31,953 --> 01:02:41,980
And I mean it's sort of interesting because it's what I've been able to figure out is like
a lot of the statements which are BS, uh they sound very similar to non-BS statements,

833
01:02:41,980 --> 01:02:45,742
except like instead of at the end of the statement where you like inter

834
01:02:45,998 --> 01:02:53,561
introduce the actual point or a word that sort of completes the thought, you throw in
something else that just loops back on itself.

835
01:02:53,661 --> 01:03:02,445
And it's it's the really interesting thing is that there's a people can very readily
detect it, that it's happening.

836
01:03:02,445 --> 01:03:06,156
And it causes a lot of negative sort of results in organizations.

837
01:03:06,156 --> 01:03:11,548
They people can will say that organizations aren't great or that it's negative to be in.

838
01:03:11,819 --> 01:03:15,200
however, being a exposed to it

839
01:03:15,200 --> 01:03:26,328
actually causes lower analytical thinking and insight and believing that BS was said, uh
and you believe the person to have been more intelligent in a way, for a small portion of

840
01:03:26,328 --> 01:03:27,450
the population.

841
01:03:27,450 --> 01:03:35,624
And I this sort of scares me that like some people actually like hearing BS and they will
spread it in an organization even if they know it's bad.

842
01:03:35,784 --> 01:03:44,330
And uh I think it's all it's sort of very connected to how I see the the narcissistic
language that comes out of LLMs today.

843
01:03:44,456 --> 01:03:54,294
which sort of justifies for me why so many people think that LMs are so great in in what
their what they say or or their capabilities, which is another really scary thought.

844
01:03:54,294 --> 01:03:55,674
I I'm gonna go look that up.

845
01:03:55,674 --> 01:04:02,420
I think uh yeah, I think you and I have very similar ideas around LLMs, th by the sounds
of it.

846
01:04:02,420 --> 01:04:08,784
Uh they uh I hate how much uh of a people pleaser LLMs are.

847
01:04:08,784 --> 01:04:10,646
And so I think it's it's very interesting.

848
01:04:10,646 --> 01:04:19,752
And it if you're enjoying good books, um and you like to understand how people think, you
should read the book, Thinking, Fast and Slow.

849
01:04:19,752 --> 01:04:20,633
Go read that.

850
01:04:20,633 --> 01:04:23,104
Then now like if you read it before.

851
01:04:23,188 --> 01:04:28,136
reread it and think look at how L L Ms work and it's very interesting.

852
01:04:28,174 --> 01:04:28,875
Daniel Kenneman.

853
01:04:28,875 --> 01:04:31,489
Uh yeah, no, no, I I I agree.

854
01:04:31,489 --> 01:04:34,398
it's been a while since I read that, so I'll maybe have to put that back on my reading
list.

855
01:04:34,398 --> 01:04:36,305
yeah, we'll get that in the show notes.

856
01:04:36,486 --> 01:04:38,428
Honestly, this has been a great discussion, David.

857
01:04:38,428 --> 01:04:42,824
Thank you for joining us today and talking us through the browser stuff and testing.

858
01:04:43,010 --> 01:04:44,472
Well, thank you so much for having me.

859
01:04:44,472 --> 01:04:45,919
I've really enjoyed this.

860
01:04:45,934 --> 01:04:46,766
It's been it's been great.

861
01:04:46,766 --> 01:04:52,180
And hopefully uh everyone liked this episode and they'll be back again next week.