So welcome everyone to another episode of Adventures in DevOps. Happy new Year, Warren, Happy new year. Thanks for joining me today. Thank you for having me back. Always a pleasure. Jillian, welcome back. How are you good? Good? How are you? And Happy New Year? I'm everyone. I'm pretty excited about today's episode because we have your background, AJ. So our guest today is a J Funk and his background is just so cool. So you're a software engineer for Rainforest, you are a you live in California, you're a snowboarder, and you are a previous d one and professional baseball player. Is that all correct? That is correct. I've taken an interesting career path, so yeah, definitely lots of stuff to do you here, and I'm in the Lake Tahoe area in California around the Porta in California and Nevada, So yeah, it's all true. What position did you play in baseball? I was a pitcher? A pitcher, Okay, cool. Oh that's what I know what it is. I was about to be like, I've told you what we're talking about here, but I know what the picture is. It's all about the little connections, Jillian. It really is. So cool. A j tell us a little bit about your role at Rainforest and what Rainforest does, and I think I'm interested to hear what led you to go from pitching Major League Baseball to pitching code? Yeah for sure. Yeah, so I started writing code when when I was a kid, really always as a hobby. My plan life was always to play baseball. You know, once you become an adult and you realize that's not always such a viable career path. I started realizing that this thing I do as a hobby I could do as a career. So just started kind of dabbling and realizing that I actually really enjoyed doing this, and it was a big pivot from you know, just athletics all the time. So yeah, now I ended up at Rainforest QA. I've been here for seven and a half years now. Oh wow, Yeah, it's been a long time, and it's it's been really fun. I work with really really great people, really intelligent experienced engineers, really great culture. We're distributed all over the world, so it's fun. We get to go meet up with each other every once in a while in some random place in the world, and so it's it's a really enjoyable place to work. I specialize in front and development, So I spent a lot of time with the product and design team shaping how our product works. And you know, as a quality assurance company, it's really important for us to have a really high bar of quality and reliability for a product. Obviously, if our app breaks, why are you going to trust us to make sure that your app doesn't break? It's reasonable, yeah, right, and you know, to meet that high bar, we use rainforest to test Rainforest. So being able to eat your own dog food on a daily basis is a really really good position to be in. It allows you to identify pain points in the user experience before your users do. Hopefully that's not always true, but we do our best, and it also means we spend a lot of time thinking about the best way to do QA, both from a kind of philosophical standpoint and from a practical implementation standpoint reality in other words, so this experience obviously helps guide our product roadmap, but it's also led me to develop a lot of strong opinions on how product and engineering team should shape their testing strategies, what kind of tools they should you'd be using, and how they can ship code quickly and continuously without having a sacrifice quality for sure. For me, that seems like one of those rabbit holes that it's really hard to find the balance on of like what's the minimum level of testing that you should be doing, and then like what's a what's an adequate level to get into where you're actually still getting good use for your time, you know, because obviously you can throw something at it that tests every possible combination or or path through your application that could ever be taken, and you're going to reach a point of diminishing returns there. So how do you figure out what that right spot is? Yeah? Absolutely, the myth of one hundred percent test coverage, right, Yeah, Yeah, that we we all strive for. I think they teach you very early on that, like one hundred test coverage is what we should be doing with every every time we push code there, we should make sure everything is covered. The reality is that's just not possible, right, What does that even mean? Does that mean every line of code is covered? Does that mean every possible edge case is covered? Like, you're never going to get all of those things. So the trick is finding that balance, right, We want to make sure that we have confidence in the thing that we're shipping that both the thing we're shipping works and that we're not breaking anything else. But we also want to be able to move quickly. We don't want this to get into our way, and so the main ways we go about that is really thinking about what your testing strategy is, right, What are you testing? What is your bar for quality? Because in reality sometimes we're okay if some things break, especially when we're in the early stages of prototyping something it's a beta feature, etc. And what are the layer of your testing, right, so we can we can think about our testing strategy as layers, with pictured as three layers, but more as a period. Right. So your foundation of your testing strategy is your unit tests, something we're all familiar with. We're using code to test code and small chunks. Right. What a unit is is totally up to you. We can define it as a single function or some class component whatever. The top of our pyramid is our end to end or our UI tests. That's testing our application in a kind of real world scenario. As we go up the pyramid, these tests are more comprehensive, we can rely on them more, but they're slower, they're more expensive. Right, So at the base of our pyramid, we have all of these unit tests that we could run constantly. So at Rainforest, we run these every single time you push code it. Just run them because it's cheap. We don't really care if you're lazy like me. I like to just push my code up and oh, something broke, I didn't notice, because I don't want to run the whole test suite all the time on my local machine. But as you move up that pyramid from unit tests, the middle of that pyramid would be our integration test, so basically testing multiple chunks of these units and how they interact with each other. That could be maybe one of your micro services talking to another micro service or something like that. As we get to the top of the pyramid and we start running these end to end tests, that's where I think strategy becomes much more important because, like I mentioned, they're slower, so we actually care about when we run these. We need to be more strategic about how often we run them. And they are more expensive, so we don't want to just blow a bunch of a whole bunch of money on them, right, So finding the balance between those things is the real trick. I like that you pulled out test pyramid and not one of the other newer hipster trends of the test diamond or a test climb model gone with the tried and true test pyramid. I mean, at least that's how I've always seen it. And you know, it's also really interesting that you bring up the one hundred percent test coverage is not possible at least from like anecdotal experience for me, I found it's almost like a preto distribution and follows the eighty twenty rule where if you wanted to have one hundred percent test coverage, it would actually require an infinite amount of time. Absolutely, yeah, we certainly strive to have all that test coverage, but I think the reality of one hundred percent test coverage is more along the lines of how you define it. Like your user workflows, Right, so the typical examples like your log and flow. What are the the main outcomes of the log in it's successful, log in, failed, log in? Maybe I forgot your password? And do you have test coverage and to end test coverage on that flow? If so, we could usually consider that cover. Do I have a unit test that covers every combination of things that I could type into that box. Absolutely not, but having some sort of test covered on it to make sure that it actually loads in some kind of real world scenario gives me much more confidence. With a lot of things like logins. And there's so many areas these days in developing software. You're using SaaS products as a as the mechanism for implementing that. What's your approach for. Dealing with that external dependency? Because you can you can mock it, or you can try to stimulate it, or you can actually call it. What do you what do you think about those? I feel like that the job at a me. Well, I see you, I see you're coming there. Now now I'm trying to Yeah, I got nothing came back to you. Yeah. So I think what you're asking is when we run any kind of tests, take We'll stick with unit tests for for the time being. There's a context that they run inside of, right, and if we wanted to test our app as close as possible to reality, we would test it in production on an actual machine with an actual human, which some people do. Right. There's obviously downsides to that testing and production probably don't have to get into why that's not a great idea. And the entire video game industry is wrong. And yes they are wrong. I mean, I'm not even sure that's true anymore. Like they don't even release games, right, It's just like content that you click download on and you pay for and then the game comes later or something like. I think that's what the game industry has gone towards. That does tend to be how it goes. But I still feel like they have the users doing an awful lot of acceptance testing in the video game industry and I'm like, too cheap for this nonsense. But anyways, I'm trying not to be rail the entire conversation today, so we can we can skip right on over that. No, I totally agree. It's like you do. Even when you do get a game, it's an incomplete game that's buggy, and you play it for an hour and you know, I'm never playing this again. So I'm a late adopter when it comes to these things. I wait till the Internet stops screaming about it and I started downloading. But yeah, when obviously we don't want to test our applications in production because we're smarter than that, and we have the ability to test these things in different environments. When we are running things like unit tests, we're kind of stuck inside of this artificial context. Right. If you're just running code to test code, it's inside of that specific code environment. It's given us inputs and outputs. Even as we go up the chain to some things that like to call themselves and the end tests which I kind of disagree with, which would be things like dombased testing, you're still stuck inside some kind of context. Right. So if the DOM, if you're not familiar, it's the Document Object Model, and it's essentially the application in interface that we have with the browser. So it's how our JavaScript code talks to the browser, how we manipulate things, how we read things from the browser. And so the important nuance here is that our code interacts with the DOM. A human being doesn't interact with the DOM. Right when you go click a button, you don't go talk to the DOM, interact with the user phase user interface. So we want to get our tests as close to actual end to end tests as possible, right, A human looking at the screen, a human interacting with the screen. And if we're not in that production environment, every step we take away from that gets us further from reality, right, it gives us this false sense of security. Sometimes there's a really common example with with DOM based tools, might be click this button, did it work right? Well, just because you can't interact with that button through the DOM doesn't mean your user can actually interact with that button right. There might be you know, some kind of overlay over my button. The button might be off of the screen. But when I asked the DOM, can I click the button, it says, yeah, we clicked it. It worked. Ship the code and now no one can log into your AP because no one can click the button right, And so doing ours as much as we can to get to that real world scenario, creating testing and staging environments that mirror production as much as possible, and loading these things into virtual machines with operating systems instead of a headless browser, which is, you know, basically a browser with no UI, and interacting with it in a way that human doesn't interact with it just gets us further away from reality. I mean, it's interesting you bring that up, and I'm sort of now I'm intrigued if you maybe want to roast all what we've been telling our customers. So obviously, where where we provide a third party product for our customers for a log in and access control, so they have, you know, providing them the off needs there, and I think the biggest advice that we end up giving them is like we are already testing that thing, Like, you know, don't focus on this. You're wasting your time duplicating our testing. If you felt the need to do that, it's almost like you don't trust us with our product, and then you probably should question why you're using that solution in the first place. If you get to that point, you know, that's actually a conversation more than it's a technical solution. However, we do find some customers still have a need to go a little bit further and the thing that we've done, I don't know this is the right answer, but we provide a given it is a SAS product, we provide a clone of our service as a container that can run that it is trimmed down, only has minor features, but allows the flows that you're going to test or you want to actually verify available without having to go through all the complexity that the service actually provides. Yeah, I mean, I think that's a good compromise. Right. So in this whole strategy, of fending balance between our confidence and our velocity and shipping. The reality is our testing environs are not going to match our production environments all the time, right, and a lot of times we're constrained by resources. So I think in a situation like that, that totally makes sense. You know, some kind of pared down version of your production application. I think the important thing is how you're testing it, right. If we are able to just kind of like strip down our product and test the bare bones version of it, as long as we are in an environment, right, they're clicking on it through I don't know, a web browser or whatever it might be, versus just like you know, running some script in the background. I think that's a really good balance between those two things. The key here is that we're still doing and testing right. I imagine it's you know that someone's typing in the box, a button's being clicked, there's an HTTP request or whatever it might be to an API that reads from a database, and we're checking out all of these things actually work together. So yeah, I think that's that's a good compromise. I think one of the things that actually comes up a lot is maybe just on a slight tangent is people are so focused on and testing they never stop to question should we like for that particular flow, is that where the value is for our company? Is it really where we should put a lot of resources? And do find that those that you're working with or your customers may or may not know where the highest value testing should be done, And then that's a conversation or maybe it's. Something that your tool provides. Yeah. Absolutely, And I think the trick between the trick for that is doing it early right. So, if you have a large application in a code base and you have not written any end n tests, it's hard to determine where to start, right versus when if you start early on, it kind of writes itself. Right. The first day is your login, you have some login coverage. Determining where the highest value is is certainly up to each usually the product team, Right, what do we care most about not breaking? And can we create some kind of smoke test that spans all of these right, So each one of these tests has a certain level of granularity to it. A good smoke test might be can I log into my application? Can I create a thing? Can I delete a thing? And things just like generally work. Those initially are your highest value tests because I know that my app actually loads in reality, right, regardless of what my unit tests say. After that, it's usually defined by what those user flows are. Right. So, as you're scoping something out with your product team, here's this new feature that we're building, it's really important in your planning process to include that right right tests for it. These are things that we usually as developers, kind of bake into our estimates, Right, I have the right unit tests for this. At Rainforest, we've shifted more towards baking and Rainforest tests for these things. We obviously have unit tests, But getting the coverage at the time of implementation or the time of release or whatever that might be, is usually your best bet to get that. If I have a large, large application and I don't have that coverage yet, it is certainly a balancing act figuring out what should be test first, Right, So I would certainly start with those kind of smoke tests, and then your highest used features is usually a really good place to start. The pitfall that you go into is putting too much nuance in a lot of these tests, right, What if they click into this and click out of that and then open this menu and whatever. Keeping them very coherent and legible and kind of focused on the thing that they're testing is the important piece of having efficient tests that you can maintain over time. I saw Will smarting there, and I know he's just entered into a new, glorious position at his organization, so maybe he has some unique insight that he's interested in blessing else with No. I was curious because this is an opportunity for me to throw in a buzzword that's trending, and so once the episode is transcribed, we'll just go viral on that. So does AI play a role in. Helping figure out that type of user flow? In the different like odd places you can end up. Good question? That's my knowledge. Yet, you know, as really good at some things and really bad at some things, and we haven't quite figured out how to make it give it enough context to understand how it should go about testing your app. Right, we do have some really cool AI tools that rain for us. They don't determine what your test coverage should be. Rather that's kind of left up to you and then it helps you write the test. So what we have is entering a prompt, you know, say, it could be something pretty generic. Log in and Adam add an item to the cart and check out something like that, and it will generate your reinforce steps for you. So during execution, AI is left out of it. Right, It does the initial generation and then we just execute things normally and then we have some self healing functionality. So it fails on something that we generated, we're going to try and regenerate those steps. And what's really nice about that is since Rainforest is a visual tool, we identify things on the screen based on screenshots. Right, it's possible for you to make slight visual changes, and now that image doesn't quite match up, your test might fail. You don't want to have to go back in and retake all of their screenshots. But since it's generated by AI, it could go back follow the same steps and realize this is what the button is here. It would be really cool if it could kind of add that test coverage for you, or like tell you what you should be testing. We've poked at that a few times and it's obviously just really dumb in that aspect and doesn't really give you anything useful. It's like, yeah, go go test all the things and make sure things work and it's like cool, yeah, I knew that, thank you, Yes they get smart. I feel like. Part of the answer is also the domain you're in. I know, something we haven't talked about is like really, at the top of the test pyramid is exploratory testing, whereas like add your creative human instincts to where bugs could potentially pop up while you're looking at an interface or API. And I don't think we're doing anything wrong in the creation of AI and LM models. It's removing the creativity from them, and I think that that harms us here. But there has been one area, especially within things like protocol creation or SDKs interfaces for the services, and that's I think the keyword is fuzzing. So trying an LM, any sort of AI can spam with almost a more intelligent root for strategy about what sorts of inputs tend to break your interface or your service or your product and then use that as a potential test that you can commit longer term. And again it's not for everything, like I don't think it really works so much in a UI world, but definitely depending on what your service or interface is doing. Stuff in the crypto space cryptography, not blockchain. Just to be clear, I feel like that was a dig there, Warren, What are you trying to say? You know, it's not the sort of thing I want to bring up on an episode. Well, you don't want a record. Huh yeah, definitely not on a record. We do cryptography because we're really into security and deep there, and we do not building our own crypto, but hi, we're very high users of it. Everything JAWT creation or jot creation, every single different kind of algorithm strategy we end up utilizing these, and so finding where we're not using libraries effectively is certainly an area that we've potentially looked into. Actually, according to our company by laws, we're not allowed to do anything regarding cryptocurrency, Like it's actually not allowed by the country of Switzerland for us to get involved in any way. We can't accept payments, we can't pay people in crypto. We can't even think about consulting for companies that I want to do something crypto related. That's discrimination. I work at HPC and I'm pretty sure like some of the admins will just kind of use like a little bit of the compute power from different clusters that they have to be running different crypto schemes, but I haven't. I haven't like one hundred percent caught anybody. But I'm just I'm waiting for the day. I'm waiting for it. Not to tell on them, I just want to know because I'm super nosy and like I just like knowing things like this. No, I'm going to tell on as long as they cut you in. Yeah. It's like when my you know, when my website not hacked by that Chinese jewelry store and I was like, guys, like, if you if you would just give me a cut, this would be fine. It was nice. I mean I think that's I think that really is expert advice from our resident mL expert here, because. Like just making. Yeah no, because AS just came out and said that the strategy of sharing reservations across customer a WOS accounts, Like if you're a consultant that does bundling, for instance, reservations or compute reservations, you no longer can pass along that savings to the customer. I mean, what are they going to do with all this excess capacity now other than some good old fashioned bitcoin mining. I don't know, I don't know. I mean we could be making drugs for autoimmune diseases and cancer, or or you could be making some call part cash. I don't know. It all. Such both it's not in either. Or there's plenty of compute power these guys have. You know, they're spinning up plenty of a W Westerns. They're not going to notice if that last ten is using crypto. So when this episode launches and we all get blocked from a respective AWS accounts, we can just reflect on this moment fondly. So, I mean, for the record, AWS is going to block you because the ROI on utilizing cloud resources to mind crypto is so low that you're pretty much just paying AWS. But it is a good indication that there is malicious activity happening on your account. So it is something that they will for sure investigate, and that I think is as much as a tangent on this that I want to go down for today. Right, Yeah, I think we should talk about the low code with Rainforest. I love low code stuff. How did how did this come about? And like how does it work? I just I want to know all about it. Yeah, for sure. So when I first joined Rainforest over seven years ago, our model is a bit different. We had a bunch of human testers. It was kind of the gig uber model of I have something I want to test. Here's my test cases. They're all written in plain English, and we'll provide a bunch of humans for you to go test your application, right, including some exploratory stuff like you mentioned go click all over this page and try and find problems with it. And that worked really well. It was true and the end testing, we load your app in a in a virtual machine inside of a web browser. They're actually clicking the buttons and confirming there's things on the screen. But what we found is that humans are inefficient and expensive, as we all know. That's why we have automation, right, and so we kind of shifted over to automation, but we wanted to do something a bit different from what everyone else was doing, which is these code based tools uh dom based interactions, and instead we built it all on the visual layer. So the way it works is you go in, you load your app and essentially, just like tix stree shots of things, right click on this type into this field. I can give it an ai an AI prompt and say, you know, log in and check check out in the cart and then when you execute things, it loads into the same environment. Right, you have your your staging environment. Hopefully you have some seed data with login information. You can load that all in the rainforest. It goes in and runs this whole workflow for you. The output of it is a video of the thing being tested. Results on each step, things like HTTP logs, jobscript, console logs, all the information that you need to actually debug things when something instead of it just saying you know, like in a unit test and it's like failure like one does not equal to And so by doing things at that visual level, it offers a lot of flexibility. The first thing is that we're out stuck inside of the browser. Right, we do primarily focus on web based testing, but that does not mean you're stuck inside of the browser. It means you can do things like install a Chrome extension. Right, open another tab in your browser, install a Chrome extension, interact with that extension. Because while you're still inside the browser, you're outside of the scope of that web page where you usually are interacting just through the domin you can you know, install some type of desktop application and test it through there. Because since we're working at the visual layer, it doesn't care what you're testing. It doesn't care what your tech stack is. It just cares that it loads in the machine. And it also offers a lot of like more flexible and robust in avoiding flakiness and brittleness to small changes. We have fallback methods. As much as I've been kind of hammering that testing with the DOM is not a great idea, we do offer DOM fallbacks because sometimes it makes sense. Sometimes I don't care about the visual appearance of the button, and all I care about is that there's a button there. Right. In reality, there are variables that we can't control. Right. A very common scenario is my marketing team is running experiments. Every time I load this page, the button says something different, it looks different, and so we don't want to tie the visual appearance to the past fail result of this test. So I'll use something else. I'll use a DOM selector. We also have like AI search. You could say something like the button, the log and button at the bottom of the page. And so the important point here is you don't write any code whatsoever. We have an intuitive UI that you do all of this through, which means you don't need skilled engineers to do it right. A lot of teams have QA engineers that their job is to just write tests all the time. Other teams, you are read the engineer is responsible for writing these tests, but they need very specific domain knowledge, right. I need to know about the thing that I'm testing, like from a product standpoint, what does this thing do? I need to understand the code. I need to know how to write these tests with a no code solution. Anybody can do this, right. It's up to your team who owns quality. Who owns these tests. For us, it is usually the engineer that is shipping the code we write the reinforest tests with it, or our product and design team owns it because, like I was saying, they're very tied to our user workflows, right, They're like, this is how we design this thing. Engineers are going to build it, and then any of us that have knowledge about how this userflow is supposed to work can own this test. So it makes it much easier to both write and maintain your tests over time. And then you know, if that person leaves your company and has all of that domain knowledge and how it works, you just need someone who knows how the app works and they can update your test suite. Now, I feel like one of the biggest mistakes that I keep seeing over the course of my career is as companies grow, they tend to have more, allegedly more software, more code, which may or may not end up in a giant ball of mud mass or even an extensive number of quote unquote micro services that communicate and really depend on each other. And there was always this challenge by someone who wanted to have a test that required somehow interacting with all of these components, and they never really could understand that one of the whole point so micro services was to isolate testing. But I think we live in the reality, which is there are some companies that do have a giant ball of mud that have thousands of binaries that have to be installed on running servers. Is there a strategy that I don't I don't mean to, you know, pick on your company. I don't know. If I don't think there is a strategy. I think the strategy is right micro services. But I can imagine that. You know, as a SaaS company, the last thing we want to tell our customers is, yeah, have you tried not having that problem? Is there a package manager that tends to do this solution that I see that what is everybody's favorite. Yeah, distributed monolith always works. Publish all your binaries that remotely depend on each other to a third party solution and then pull those out at run time always works. Best solution ever. Maybe Aja, you have some insight here on either something that works or something that works with Rainforest QA to deal with those situations, or maybe you just you know, it's not something that is handled today. Yeah. Sure, we do have a bunch of different micro services running, and I think I'm going to refer back to the testing pyramid. Right, is we test each one of those micro services in isolation. Absolutely, Maybe we test as we go up to the next layer of our integration test, we test some interactions between them, right, the kind of core you know, handshake interactions, whatever is the main functionality of these two microservices talking to each other. Maybe we have some tests there, But at the end of the day, these end to end tests are comprehensive. Right, If anything in that micro service architecture is failing, presumably my test is going to fail. And at the end of the day, all we really care about in theory is what the user gets when they're interacting with it. So if I'm just clicking a button, maybe there's a thousand micro services that are involved in this, and maybe I'm not directly testing each one of those, but by implementing it as an end to end test, I am very confident that they're all working because my test passed. And so being smart about how you implement each one of those layers in an efficient way. Right, lots of unit tests on each micro service and then this overarching test that just make sure everything is working together is usually a way to go about this. I mean, maybe it's a technical implementation question, like where is the environment running that the rainforest tests are actually executing us? Is this some sort of binary or CLI that's run on the client side, or are they sharing with you a set of micro services with deployment instruction so that you can run them within your own infrastructure. Sure, Usually our requirement is that you need to be able to access it via a web URL. So the kind of standard way that a Rainforests is run is we provide you a VM, and that VM has a browser on it, so I can specify Chrome on Windows eleven and the first step of your test is going to be a navigation navigates this URL. This is where my web app lives. There are some different use cases where you can absolutely go download a Binari and install it and do whatever you want with it or out of the box functionality is to primarily test those web apps, so that's where we focus. But you certainly have the flexibility to do whatever you want with those vms. No, I mean, I think I think that approaches genius. Basically, it's out of scope for setting up the environment unless you want it to be in scope of which you know. Then it's a it's a virtual machine. Go to town on how they want to deal with it there. Right, and by kind of forcing people to give it a public URL, we're nudging them towards good practices. Right for sure is set up a staging environment a QA environment and make it mirror production as much as possible, which includes being able to navigate to it in a URL. And these are small things that we see with some of our new clients are like, well, I don't have a staging environment, and sure, I guess you could load your production environment in there, but let's show you how to test this properly and not shoot yourself on the foot. I'm like, I love that you're saying that. The number one feedback I've always seen here is we can't expose our non production environment publicly, Like people can't know what we're currently working on. They will use that information maliciously against our company in some way. Yeah, like what are they going to do with it? Though? You know, like if I have seen some interesting mistakes, like you know, maybe we're cloning our production database and not sanitizing sensitive information from it or something like that. Yes, absolutely you're doing some bad things, but there are certainly ways to do this. And I'm of the opinion who cares if people are in your testing environment, Like worst case they blow up your testing environment and whatever. And in that case you figured out how they could blow up your production environment. Without losing pride exactly. Yeah, I think there's. That's, you know, like part of the undocumented learning curve of working in this industry, you know, because people who are early in their careers think things like, oh, I shouldn't expose staging until you know, they learned that that's actually probably a good thing. But like, nowhere in any. Computer science or course or or boot camp or anything do they cover these kinds of things, and so I think that's actually like a really valuable add on service that you know, you get from rain forest, or that you get from working with people who are more experienced, is just like learning that tribal knowledge that's going to help you out later in your career, so you don't have to reinvent the wheel and solve problems that we actually solved thirty years ago. I mean, that's will. I mean we've unfortunately had to append our documentation with like here explicitly are the sensitive pieces of data that are relevant to our third party application. This is sensitive. This is sensitive, Like this is not sensitive. This is like the application I need not sensitive. Like, do not try to encrypt this, Do not try to secure it, because people will try like how do I do this? I'm like, you can't stop it, Like this has to be public on your website, in your application. People have to be able to see it. You're not going to be able to get around that. And I feel like it's more than just experience. I feel like there's a whole level of pragmatism there, like weighing the cost versus the reward of actually trying to sanitize that piece of information. And having a third party testing service, as you mentioned, just reinforces that in a way, like you are going to have to expose that to be tested, must show that it's not actually sensitive information. Yeah, definitely. To me, it just reminds me of like the myth of the one hundred percent test coverage, right, It's like, can we one hundred percent encrypt everything? Absolutely not? Like people, your end users need to see this information. I see. I've seen some interesting attempts to kind of obfuscate those things, Like I've seen some libraries that prevent you from opening the davascript console, for example, and it's like, what are you hiding in there? Maybe maybe you should just not put sensitive things. Here's here's a wild thought. I just don't do that. Credentials like encoded in the HTML on your page. Maybe. I mean, I can't believe you two are joking about this, honestly, like one of the most common attacks against So I don't do. You why so I can joke about all this because none of them It's okay. Tone You'll have. Jillian, You'll have plenty of opportunity to get your models encoded with AWS access keys in secrets, and then you just ask the model, Hey, can I have an access key and secret that are valid that work for any aw's account. I did actually accidentally push my AWS credentials to get huged once, and like the amounted emails that I got from AWS was just like it was unreal. It was very It was a very bad day for me. It was a very very bad day. So I've done other stupid things, but I don't do the same stupid things, so I can sit here and be very smug about this. Like. That is like that is I've always I've often wondered about that, Like the speed that AWS and other malicious people can identify that you committed an AWS access key to a GitHub repo. It was instant. It was right then, because as soon as I did it, I was like, oh no, and tried to you know, and like try to like make the gi hub repil private, and no, it was instant. They knew, they knew it was out there. Yeah, I mean it's bad. I mean I think I saw a bunch of statistics on this that for AWS keys and on GitHub, it's about thirty seconds to two minutes having been exposed in the repository anywhere in any format, So like commit at the beginning of the repository where it was there but then got removed, so it's not in plain text anymore. You have to go back through the get history still about two minutes. Then there's exposure on like stack overflow and places like I don't know who uses Facebook in connection with their work, but that was another place, and then Instagram and read it somewhere between two and four or five days, and then there's a couple other ones where it's six and more. Some of those you have to. Thank, like GitHub for like they'll actually discover secrets there. So if you provide a third party application that has credentials, like at authors, we have our secret keys registered there, so if one of our customers exposes keys for our service on GitHub, we'll get notified, automatically revoke those keys and send them an email telling them that they did something that they probably did not want to do, multiple times if that's if necessary, because that's happened as well. I want to switch topics here real quick, AJ, because you've been with Rainforest QA for over seven years now, which is unusual in the tech industry, So I'm curious about what are the what are the things that you look for in a job, that have been fulfilled at Rainforest, that keep you there that long. Yeah, for sure. First and foremost is the people. Actually, when I was interviewing with Rainforest, the last person I talked to told me something like I say, at Rainforest because of the people, And I was like, okay, that's that's a way whateveryone said, And then I quickly drank the kool aid, I think, and I found myself saying that on interviews and I'm like, I know, this sounds like a load of crap, and I think, you know, the hiring process is super super important, right, Yeah, both finding people that are qualified for the job obviously, but are good culture fits. We have a pretty small team, so there's nowhere to hide. If you are not doing your job where you're not up to par, you're going to be exposed pretty quickly, which leads us to have a very reliable team. You know, we are distributed globally, so there's a lot of handoff. You know, I'm going to sleep, you're waking up. Here's what I did, and I trust when I wake up that you're just going to have this thing done. And if you're not one of those people, you're probably not going to fit at Rainforest. So really really qualified, experienced, smart, reliable people makes life so much easier. And then the other piece of it is, you know the mission that we're on, the technology that we're building. I think when I was first exposed to it, the first time I ship code with Rainforest, it was kind of like, Wow, how have I been shipping code before this? And the answer was I was probably breaking things all of the time and you don't notice until user catches it in production two days later or whatever. And it's something that I'm really passionate about. I think as a front and engineer, we get really caught up on the details. Right, There's all these visual layers, there's these very specific human interactions. I like building things that humans are actually interacting with, and that kind of naturally leads you to a quality assurance mindset, Right, I want everything to be perfect all of the time. How do I ensure this? And so the combination of really great people and working on something that I'm actually really passionate about and I want to see the rest of the world adopt these correct ways of testing things, in my opinion of course, just makes it makes it easy to work area. Right on. That's cool, that's cool. Is there you guys obviously do a lot of front end type testing. Is there a particular industry or vertical that you have got a lot of experience it's in, or something that has worked really well that makes a really cool story. Something I've been involved in that makes a cool story. Oh, I don't know if I have a good answer for you. Honestly, I've been, like I said, I've been at Rainforest for so long. That's all I didn't think about. I guess, right. Do you do you attract like a certain like customers with like financial apps or with like web based gaming apps, or is there a particular vertical that tends to gravitate towards your service? I think not really, And I think that's one of the things that makes it cool is it's a very generic testing tool. Right There are some some limitations, but in general, if you could load your app on a machine, you could probably test it with Rainforest, not caring about what the tech stack is those kinds of things. So there's a very wide range of users that we have from. Yeah, there's some financial some financial companies do some What I always finds interesting kind of like testing visually things like spreadsheet style apps like their tables and things like that. And then we have some really cool like visual tools like like drag and drop interfaces where you're building things like you know, lego style building, where there's probably, to my knowledge, not any other great way to test something like that, Like what you say, are all my legos on the page? Yeah they are? Are they kind of oriented this way? Like? Yeah they are? But how does it look right? What does the user see? So the real sweet spot is really visual based applications because I don't think there's other great solutions for them out there. But in general, being a kind of generic visual testing application, it really applies to anything right on. For a lot of web based front ends, it's all no JS based. Do you have a favorite no JS type tool? Are you like a React fan or next JS or view give. A personal preference? Yes, I am a Ract fanboy for sure. I started. Uh you know, we rewind all the way back to like the j Query days and stuff, right, I see that, and I have nightmares still we have some we actually have some of that floating around and our like like our admin applications and stuff where it's like a rails back in and they're like, yeah, we got jQuery in there. And then my first thought was always like, well, like how do you test that jQuery? And the answer is we don't. Uh throw a couple of wait for us tests that and call it good. And I started. I started with Angular back in the day, all right, back Angular one anyways was kind of the reverse of React. We're like, we're gonna put your JavaScript in your HTML. React took the approach ever to put your HTML and your JavaScript, you know, just smush it all together. And it's come a very very long way, I must say. So. Yeah, I find working with React very easy and intuitive, and it's very nice that the general JavaScript community has supported that and has pushed that forward because especially, you know, with with all software and technology, but especially in front of development, it's really easy to pick the wrong tool long term, Right, I picked this thing, it's great, and then we find a better way to do it, and they just abandon the project. Right. This is true with I mean anything open source, and we've run into this a lot of times, right, even with open source testing tools. And actually we had a very large Enzyme test suite on our React application, and we ran into something like this. There was a new way of testing React apps, which was the React Testing library, and Enzyme kind of said, yep, that's a better way to do it. We're going to stop supporting at after I forgot version like REX sixteen ORRAC seventeen, like, well we want to upgrade to React seventeen. It's like, well none of your enzyme tests work, so yeah, exactly, exactly too bad for us. And so now you start weighing the options of well, how do we upgrade? Right? Do we just say let's not upgrade, which is going to bite you really quickly, right, especially at the pace all these JavaScript libraries remain updated. I want that new shiny thing, I want support for that thing, and I don't want to be stuck in the past. The more you get stuck in the past, the harder it is to catch up with everything else. Right, And so our options were basically rewrite all these however many thousand enzyme tests, or we could just nuke them all, which is it reminds me of like these these memes I see about like junior engineers and the intern where they're like they're commit messages. I knuked all the tests because they were failing, and I kind of make them pass turn true and all the tests because you know, that's one of the. Past, the only reasonable way to do things. Yeah, and it sounds kind of like an overreaction, but as we started to kind of think about these testing philosophies, we're like, we have end and test coverage on all of these things, right, And a lot of the front end tests, even though they're unit tests, they they load things in a headless browser and we're kind of recreating what an end end test does. So we chose to keep all of our actual unit tests, all the kind of business logic that didn't use enzyme new call the enzyme tests, and just lean into our rainforest test because we know if the rainforest tests are passing, we don't need all of these redundant tests anymore. And instant productivity boosts Like I don't have to maintain all of these things anymore. I don't have to upgrade them. I could just get them out of my way and I can upgrade all my dependencies. And because we have really good end and testsage, we could do that confidently and know that we're not breaking things. So, yeah, choosing dependencies can be quite tricky sometimes, especially in the job script world. Did you find some places that you still wanted to reintroduce some of the React testing library for I don't know, component level testing of the UI or have you kept with like one hundred percent of the decision to not have that layer of testing anymore regarding the UI components because you focus on the full picture and the end testing for the user flow and also whatever you have with the interaction with the back end. Yeah, we still have some of it, and we drew the line at user interactions. Right. So React has this idea of hooks, which are basically just chunks of logic. It's just a function that I can use inside of a component. We stopped having any React testing library tests that were actual user interactions, no clicking on things, and instead we used it to test the functionality the logic of those hooks. So it's essentially a unit test, but it's testing a specific React thing, and it requires the testing library to do that. Everything else kind of gets hoisted up to the end to end testing level, and it's nice to just say, hey, designer, hay, product manager, like, go at this test coverage while I'm busy hacking on things, and I don't have to worry about this anymore. So that's actually a great Jillian. Oh, I was just gonna say, I'm so impressed with people who can keep up with like the UI and JavaScript plan because I've tried. I've tried like a few times and it just then everything changed. I was like, all right, I'm not doing this anymore, and we go I'm gonna go do high performance computing that hasn't changed in like thirty years. It's gonna be Yeah, you definitely start feeling like Sissyphis. You're just pushing that rock up the hill and every time you get to the top, someone tells you that you're actually on the wrong hill. I was gonna say, that seems like a really interesting approach that I hadn't thought of when we initially started talking. But like you, you can replace having to write a lot of your tests in your React app by using Rainforest right. By just focusing on what the end user experience is and testing for that, you can save yourself from having to write a lot of tests in the React standard library. So that's where the trade off is though, right, because these tests then are testing more functionality at once and so if there is a problem, you don't necessarily know like which line of code is causing the issue or which interaction there is, so you know there really is Like how valuable is that flow? I think? And that's something that, as you pointed out Agent, like you sort of have to determine upfront, like where is the value of your testing and how do you get the most value out of which pieces. You're adding and where you're validating an et cetera. And so, yeah, I mean. In your case, and tis weren't actually providing the right value in the first place, so definitely switch them over. Yeah, absolutely. And it's kind of a question of redundancy too, right, Like is redundancy good? Sometimes like I can be really really sure and I can have some extra confidence that the things isn't going to break, But most of the time it just slows us down. Right. I find that the often the best time to add more unit test coverage is when something breaks, right, because if my end to end tests are all passing, but something's broken, very often it's some kind of edge case, right, It's some either some weird user behavior, some weird input, some weird sequence of events, and those things are usually better captured in a unit test because it's it's easier to kind of implement that specific scenario, that specific line of code that is the offender here versus creating you know, a whole new end the end test and just cover some edge case. Those tests are going to just get longer and longer and just be kind of confusing. Honestly, it's like, well, why am I just like clicking in all the these random spots doing these things trying to cover these edge cases? Like just reread a unit test for it and call good. I really like the emphasis on, you know, testing for business logic and just in general having not everything controlled by the engineers, because I find for myself, you know, like I'll write something and then I'll hand it off to a user and then they immediately start using it in some way that I didn't even think of, and then, you know, and then we do like a couple of rounds of this. So being able to cut back on that person who writes thing, who does not actually use the thing, and then just immediately being able to push it off to an end user. Yeah, absolutely, Yeah, And things like testing and staging environments are great for this. We push code of those environments all the time, give them to a PM and say go run and try and try and break this thing, right. Don't always want to do that in production, right, like if the thing's not fully baked, I don't want to corrupt something in the database or whatever. And so having places to push that and have people early in the process on this, finding the major bugs, the minor bugs, the stylistic bugs is super super valuable then having one of your users find it later. Right on. So you live in Tahoe. Do you get outdoors a lot? I do. I live here with my wife and my dog. He's a lab husky mix, so he kind of thrives in the summer, thrives in the winter. There's lots of snow out right now, and so we're outside pretty much every day, you know, snowboarding, hiking, kayaking and all that kind of stuff. All Right on, how long have you lived in the Tahoe area. I've been here for about five years now. I grew up in the San Francisco Bay area and I was part of the great COVID migration out here. We always wanted to get here eventually, and I was lucky. I was still working at rainforest at the time and already remote. So the transfer up here to from remote near an office to remote it actually doesn't matter how far you are from the office, was very easy. And we're also real fortunate that we were not the only ones doing this migration. So we've made lots of friends that were like, yeah, we lived down the street from you in the city and we all live here now. So it's a very different life, but we love it and I don't think we're leaving. Ah, that's cool. Yeah, right on, Tahoe is a beautiful area. Yeah, it really is. Cool. All right. Should we move on to some picks before we do any final thoughts on. QA rainforest tips guidance that you want to leave us with aj. I think just recapping is finding the balance between confidence and velocity, right, everybody needs to set their own bar for quality, like how what is my ratio between confidence and velocity? Determining that for yourself is the most important thing here and keeping in mind that not only is it a lot, but a lot of times it's the sanity of your engineers, right, Like, we don't want to spend all of our time writing tests, So finding that balance and doing things in an efficient way is the key to success. Right on. And I think that's very use case specific too, you know, because the right answer for a financial app is going to be very different than the right answer for like. A social media app. Absolutely cool, All right, Jillian calling you out first? What'd you bring for a pick today? I am going to pick Drive by Dave Kellett. It is a sci fi graphic novel and I think it's on it's I think it's like releasing the fourth one this summer. But it's so good and it's so nice and wholesome, which is very nice because like, I really like sci fi, but I don't really like violence or gore or you know, ikey fluids or I don't like any of that. Okay, I don't like any of it. And this is just so wholesome and adorable and the main character is very cute. So that's it. That's the pick. I've got a whole bunch of copies for Christmas and I'm like making people read them and I'm gonna have like a little little intigraphic novel cult going on soon enough. Great, all right, Warren, would you bring? Yeah? So I just got back from a long hiatus being away from the show I was on vacation, and so I think this pick is a really accurate, very short book. I highly recommend Dowdishing by Laozza, which is the founder of Taoism spelt Taoism in case in case. You've seen it written but never pronounced before. And there's just so much good stuff that is in the book that can be applied to everyday life, working environment, etc. It's incredibly short. There's only like one hundred and eight principles or so, and it starts off great with the dow that can be told. Is not the eternal tao Like, you can't write down the whole truth. There is something that's never said. It's impossible to convey everything. And I know it sounds so philosophical, you know, to go down this path, but I feel like going through these really helps to put into perspective thinking outside the box with solving certain problems or interactions or the communication we have every day. Highlight right on, cool heja, what you got for us? Yeah. My reading and listening choices are kind of all over the map. But I did have an interesting one recently. It was called The Light Eaters. It's about plants and specifically this idea of plant intelligence. So obviously intelligence is a loaded word. They're not intelligent like you and I either not debating QA strategies and things like that, but they do have a lot of intelligent like behavior. You know, they communicate, they recognize their kin, they hear sounds, they transform themselves based on the visual appearance of the environment around them, And so I found it really interesting and gave me a lot to think about, especially you know, when I'm out in nature with the with the wife and dog, just kind of staring at trees and stuff. Check out that are intelligent for sure, one hundred percent totally with you. There's a there's a good one. If you are out and there's plants or grass being cut and you notice the smell of you know, freshly cut grass. What is that? It's a fear intently, a fear pheromone that's been sent off to warn other grass that that there is danger around. Like that is the sign of intelligent life. Yeah, for sure. There's lots of super interesting examples in this book, and just like plants acting like animals essentially, and it's it's kind of mind blowing experience. I read a book recently and I can't remember which one it was, but I've been studying mushrooms a lot lately, and this book showed where mushrooms actually act as a communication agent for trees in the forest, and so like a specific you know, set of insects can start attacking trees on one end of the forest, and then the mushroom, because it's the mycelium that grows underneath the entire forest floor, will relay that information to the other trees in the forest. And so by the time the insects work their way down to those trees, that those trees are producing a sind or a pheromone that actually repels the insects by the time they get there. And I thought that was super cool. That is cool. I'm in on a like a mushroom and foraging Facebook group and everybody just like takes pictures of fun mushrooms that they find when when they're out and about, and it's it's just such a nice little group because it's so chill. That's it. There's like, there's no drama, there's no nothing, it's just look at this mushroom. I found there's an app called Naturalist that I use for that you can take a picture of not just mushrooms, but anything you find that you can't identify, and then upload it to I Naturalists and it will it will try to auto detect what it is for you, but then other people will come in and confirm or tell you what that actually is. That was pretty cool. I used to do that a lot as a kid. I'd have like the field guides and go out with my field guide and try to like identify all the plants. But now we have an app for that. There's an app for that always will what's your what's your pick? My pick is There's a series on Netflix called Kunk on Earth and I thought my sense of humor was like really really dry, but this lady, she's amazing, takes it. She takes it to a whole new level. This series is just hilarious because she sits down it's you know, like a the History of Earth basically, but she'll sit down with legitimate world renow experts in their field and ask them the most off the wall questions. And that, to me, thet was the highlight of the series. Is just the looks on their faces when she would ask them these questions that had absolutely nothing to do with what they were an expert in, but super entertaining series. Definitely ten out of ten stars. Kunk on Earth on Netflix, and you know, there's actually two other things. There's Kunk on. Britain I think, and then there's like one on Christmas and Shakespeare, so you too. Oh sweet. I will have to check those out because I love her sense of humor. I've never heard that's fun. Yeah, it's very very accurate. All right. That brings us to the end of the episode. Thank you everyone for listening. Jillian Warren, thank you for joining me hosting the show. And aj thanks for coming on the show man. It's been a pleasure talking to you. Thanks so much for having me. It was a lot of fun right on. Glad to hear that, and I will see everyone next week. M