Hello everyone, and welcome back to another episode of Adventures in DevOps. Today, I have a slight bit of news though. We have an upgrade to our podcast, as will is a way for a few episodes. So I've asked Amy. Knight, are expert on reliability architecture, to step in and Amy, are you ready for today's episode? I am excited to be here. That's good. I think I put her on the spot with calling her the expert. I don't think she was prepared for that. No, not ever expert she is. She's very modest, I will say that. So today's guest is Brian POTRELLI welcome and thank you for coming. He's a founder of Fusion auth and some other products as well. It's nice to have a fellow of expert on the show. I have to say, yeah, thanks for having me us. I was interested in how this was going to go with two experts. You know, I get this question like how did you become an expert? And usually I say something like well, I started investing in learning about security a lot of years ago, and after a lot of years, then I get up on stage and someone asked me how did you become an off expert? I don't think there's like a dedicated path, Brian, how did you end up in this area? Kind of dumb? Luck. So, we we are working on a sort of a niche product, and after we realized that it wouldn't scale, we actually wanted to start building a couple of other products. And so one of the products ideas that we came up with was a forum, and so we actually built that out completely. And when we built it, we decided we weren't going to add a log in and registration component to the forum itself. Instead, we were always going to delegate that to whatever the company already had, and so that required some type of authentication system, right, And in order for us to test this, we had build our own authentication system, and so we did, right. So we built it, we integrated with our forum. Everything worked, it was magical, and then we were like, hey, this afe thing's pretty cool. Maybe we'll just use it for like some of our other apps, and so we started integrating it with everything. Well, the forum didn't work out, and we're all kind of sitting in a room and I'm like, guys, this off things really neat, Like let's do that instead. So we like basically said a one to eighty dropped the forum started focusing on off and then had to go learn a bunch of standards and read a bunch of specifications and you know learn, you know, learn a lot about security in a very short period of time. So but totally luck, No. I totally got it. Actually, we ended up in a similar spot with a previous product that, uh, there was a lot of complexity and what we were doing, and we found that our customers were actually more interested in how we solve our technical problems than the product that we were offering. At that time, this was like a for COVID, and we were trying to sell like leaders of Leadership SaaS and it turns out a lot of companies wanted to say that they cared about leaders and building their leaders up, but they didn't actually want to pay for it. Maybe that's a little bit of a shocker. Yeah, not shocking, shocking, shocking, right. The main topic of today is I think it's going to be a little bit of a controversial episode. This may be our most controversial episode yet multi tenant versus single tenant architecture. Yeah, I guess that probably a lot of our audience already has a strong opinion one way or the other. Amy any thoughts. I guess my thought is just like alarm bells start to go off in my head. But there we go back with like the reliability stuck. Like you're like, if someone says we're going to go for multi tenant architecture. Like that, hold up, yeah, exactly exactly, Like I want to note the details. At this point, yees, so you think it's like inherently dangerous to have multi tenant architecture. I was just at the point where like the I don't know if the more separation the better with that sort of thing, But prove me wrong. Well, Brian's built a company on top of it. Yes, we're building this this forum, and then we built this sort of standalone, you know, off thing. And our other product was also you know, downloadable and single tenant and standalone, and we did we built the original product, which is called clean Speak. We built that that way because it was high performance, right, so like it filters profanity and chat so wow, a couple way of saying it. And so we're talking billions, tens of billions, hundreds of billions of chat messages a month, and so in order to do that in a multi tenant way is pretty challenging, especially when you're talking about super low latency with something like chat and so what happens is like you're in a game, you send a chat message, it goes across the wire into this, you know, the chat server. Okay, so that's in a bank of servers that's sitting in AWS or you know, somewhere like that. There's no reason you should have to jump out across the open Internet again to go filter that and see if there's any issues with it and then come all the way back. You know, that could introduce one hundred milliseconds of latency. It'd be way better if it just jumps across the backplane and essentially goes right to you know, the filtering service. And we could do that in like under a millisecond. And even with you know, in internetwork latency, it's like one or two milliseconds, so we can really shave down milliseconds here. And then we took all of this sort of like common code and deployment models and bundling and all this stuff we had and we just like copy and pasted it. So the history sort of put you in this direction. It was sort of like a sign from the universe that maybe this was the path for you to take. We kind of made a bold assumption that that was a good idea. It's like, hey, let's make this downloadable, let's make it single tenant, let's make it run anywhere in the world. And then we actually started having all these companies come to us, some of them quite large, and they're like, hey, we can't use your competitors. We can't use like A zero because it's in the cloud on it, we can't use you know, like pain because they're just pushing everybody to cloud, and we can't use all these tools, and we really want something that we can run in our own data center. And I was like, well, we got that, so yeah, let's totally do this right. So that's really interesting. So taking the architecture as the value being provided by the product as sort of the competitive advantage, yep. Yeah, we continue to do that. So, like I think our one of our taglines we've been messing around with is like aughts so modern you can download it right, and it's it's really flipping everything on its head because for the last gosh fifteen years, people have been like, yep, it's got to be multi tenant, it's got to be you know, SaaS, it's got to be cloud, and that's the only way to build an effective company, profitable company, scalable company. And I'm like, no. I'll be honest too, what you're saying. It kind of shocked me. Conversations that I've been having more and more people are actually people are considering moving off cloud to save costs, which is nothing I would have ever thought of before I had this initial conversation like two years ago with someone. Yeah, if if you have let's say, like a cloud native Lamba driven, you know, lots of like io into like something like Dynamo or something like that, Like sure, when you start and you're small, can be very cost effective. When you scale, it can get so expensive these services are like outrageous. The other thing I would say too, is depending on your architecture and how long it's been around. I feel like moving to the cloud, you're sort of forced into certain boxes of what they can scale most efficiently. So if you have something that was not built to run on a certain machine type, you're going to run into issue that scale. Would that be something that you've kind of experienced. Too, Absolutely, Yeah, The the cloud providers really love to push you in the directions that they feel the most confident and comfortable, and they where they have the biggest profit. Right if you look at just you know, like Cognito as an example in our space, like that's not a cost effective solution, like by anyone's anyone's you know view. Now, when you first start, it's very cost effective because it's free, and you know, it's pretty easy to integrate. But the second that you like start scaling and start enabling any of the features, it's cost prohibitive. And so they know how to scale that and run that really well, and then they know how to monetize the crap out of it. So I love that you, you know, you tried to pull back on like, well, you know, maybe it's really easy to integrate with Cognito, although I feel like, I see, I'm in a bunch of tech communities and I just see complaints every all day long about oh I tried to make this work and it didn't happen, or you know, I looked at the documentation and start saying like amplify everywhere, like I don't want to use amplify. I'm like, well, you know, welcome to Welcome to the mess that that is Cognito in AWS. So I started I stopped saying that, Yeah, it may be cost effective, you know, a small scale, but the overhead, the total cost of ownership, yeah, definitely high with with Cognito, for sure. It was super painful. And Amazon also does this really funny thing where they force you to use like you can't just use Cognito, like you said, you have to pull in fifteen other services and you got to use platformation and you got to do this, and it's just like, guys, this is getting insane, right, Like I don't want to do this. I just I just want to lock my users. I may have cheated and listened to some of your episodes on competitor podcasts to kind of get a better understanding before this one. Is it still true also that the customization is just not there. Cognito actually made a really big update, so I have to credit they. I think they restructured their product team and then they put some dedicated engineers on Cognito. For the last like eighteen months, they kind of up their game, Like you can quasi customize their log in. You can customize like the email templates and some of the messaging templates, and I think they even like support localization stuff. So they did a pretty decent job of like leveling themselves up to your point, it's like it's still a pain in about to integrate with and it's just feature crippled. I mean, they just like have it's a very limited platform. I think the biggest challenge here is something that a lot of people dismiss when they're selecting a product out there in the world, is a the alignment with the company that you're going with, Like what is the core value that the company really focus on and cares about? Right, you were saying that the single tendency is the core value that we're you know, one of the core values that we're offering here, you know, is that aligned, Like don't just use the product based off of the features, but you know, look to see about the team that's building in and what their you know, long term direction is. And you know, I think it's really important because when we look at things like cogn it's not really clear you know, what they're really going after. And if you start comparing the products and get really deep, you can see things like they're like tendency in Cognito itself is not really there as a solution. If you have customers that are in the business space, it doesn't really fit well with that, and it's detaining your point, you know, on the refresh with Cognito. Yeah, I mean it's configurable now for sure, But there are nine different kinds of Lambda functions that you can integrate with Cognito. Like it's not a simple strategy, Like it's not just like configuration driven a lot of times, and this is one of the huge types with AWO services in general. It's that if you want to configure it, you write your own LAMB to function with your own configuration with whatever, non documented payloads and responses, and then figure out how to integrate that into your solution and deployed and whatever to the cloud to get it to work right, and if there's a problem then you know, good luck. So some people love this, Oh yeah, all of my infrastructure everything is in AWS and you know, good on you, for sure. But I don't wish Cognito on anyone. Yeah, I mean that's a good point, right, because they AWS has this sort of disease I guess you could say, where they constantly want you to use their services to do things right. So like, you know, we use like cloud front, and in order to do all the redirects that we need to do, we have to use like an edge function, and then that edge functions go loads something out of like an S three bucket in order to like download it, and then it caches it, and you know, it's like it's ridiculous. It's like all I want to do is set up redirects, and I have to jump through like fifteen hoops just to get something so simple done because of the way that Amazon has designed everything, and they're like, well just stick a lamb on it, you know that don't work, and don't document. We'll document it later, you know. I mean, I do appreciate as a company, like comparing them to the other cloud providers, And I didn't know this was the direction we were going to go with this with this episode, but I do appreciate a WS more than the other ones. I do think they suffer from a very focused mindset as sort of the two piece of teams that they have. They're really focused on delivering, you know, exactly the thing that maybe that customer wants, but that may mean over time, they're missing some other critical features. I think what you're talking about here. A good example is returning security headers on outfront responses. So you've got some data stort inn s Rebucket, and you're hosting a website or your API, and you're like, I just want to add a header that removes the x frame options, like you can embed this as a as an iframe in a website, And you'd be like, I just want to add a header, and I don't want to muck with the underlying service, which I may not even own. Right, Maybe I've deployed Fusion ofth or some other service in a container in my infrastructure and I don't even control that product, but I want to add some headers to it. Well, good luck, honestly, because up until about three years ago, you couldn't do this without throwing a huge lamb to function at it. Now they do have something called the response policy headers that you can set in clautform, But it's taken probably like almost ten years for this to roll around and even be added in this feature, while other CDNs have been providing this baseline thing for quite a long time. Yeah, no, totally agree. They there. I think their product teams are so isolated and they don't have a lot of input from you know, just standard devs like us right that are just like, hey, I just want to get this thing done. And they're they're really listening, you know, to their largest customers, which we all know. I mean, you know, they make tens of millions of dollars off some of these customers every so, so yeah, it's it's pretty painful and they takes a long time for them to get actual stuff done. But at least they have the lambdas and the things and so like you have to jump into some code and you got to do these things. But at least there are workarounds where like you said, you know, there have been times where you know, in the past where we use services and you literally just can't do it. So it's like, well, crap, Okay, I'll stand up another service to proxy requests through and then I'll manipulate the request in my proxy service and then ask it to clout right, and it's like, oh my gosh, why am I having to manage all this junk just to add ahead or business? And I totally agree with that assessment of AWS, But again, they're one of the better out there, right. So here's the other thing that's interesting AWS. I think some of their product teams are actually starting to evaluate letting people download some of their services and run them locally, right, Like we see that a little bit with like Dynamo dB. You can run a like kind of a scraped down version of it. I think there's some local lambda execution things that you can do, and they're looking at more and so you know, maybe they'll do that with Cognito at some point too. But like, one of the hiccups that I see with all the cloud providers is the ability for a developer to test these services without having to like fire up an entire org, you know, run a bunch of terraform or CloudFormation or whatever you need to get everything set up, and then you can run your tests against it and good luck having twenty developers try to run tests at the same time where you're constantly tearing things down and recreating them and so moving that stuff back local I think is actually something that the industry has been at, which means that every developer can isolate themselves at dev time from every other one and that's a huge benefit. Yeah, I think the developer experience story with aws like offline is not the best so much that there's an entire company dedicated actually out of Switzerland called local Stack to emulate the emulate it offline, and you know, it's great. And the joke is like when is Agbo's going to buy local Stack because there's local Stack in the documentation, there's local Stack replacements for a DYNAMOITYB local which doesn't work. There's great integrations for Lambda and for SAM, the servilist application Transform for Information. It's really surprising. It's like one of the things like as you mentioned, like even though our product is focused one hundred percent SaaS, we offer a shim clone of our API for companies around locally because of course, like you're a developer and you're like, you don't care about maybe the off part, but your services depend on it, and so you need to have an answer here. And it's ridiculous how SaaS providers everywhere. I figured this out, but the cloud providers haven't done it yet. Yeah, you know, it's wild. We sort of like to take that to the nth degree, right, which is like mocking is dangerous, and so we'll market and then we'll just simulate the responses we know we get from production, so like maybe they record it and then they grab some you know, some stuff out of production, and then they can start simulating, well, what happens if like a lambda changes and then the responses start changing, or like AWS modifies something and it's coming back slightly differently. We definitely see that more sophisticated customers care about it, and end testing or integration testing and having a story dedicated to that is an important aspect, especially when you're offering something that's essentially infrastructure for your your customers. I mean different when it's just like some third party service which is solving some edge case or some CRM, but when it becomes a critical piece of infrastructure for your customers, it's a question that's going to come up pretty frequently. Absolutely, I kind of had a question I was going to ask, kind of been like doing how to ask this in the most intelligent way possible? How much of your product would you say? In most customers use cases, it probably simplifies things like performance and load testing. My experience and the source of like massive bought attack and at the same time my experience performance testing in a multi tenant environment can be tricky because while you would assume that you can just go off to the races, that is a false assumption. And if you have to performance test load test at a certain scale, it requires coordinating with the company and making sure that other customers aren't doing it at the exact same time. So I guess long story short, I kind of maybe know the answer, but maybe someone to speak to. Yeah, it's a phenomenal question. So there's sort of two aspects of this, and the first one is I'm a developer and I just want to like sort of load test locally, and I want to I just want to see what sort of throughput looks like with different configuration options on my laptop. And you download, you get it set up, you integrate, and then you just literally use a hammer, hammer the crap out of it and see see what happens, and then you can reset everything and then you can try it in the cloud. When you have a multi tenant provider, they even like a lot of times say you load tests are just not allowed, right and like period hard stop. You pay them enough money, they'll probably let you load test, but they're going to have to figure out how to get that traffic off of the main servers because they don't want to impact the you know, ten thousand other customers that are on the same servers when we deploy fusion to the cloud, especially single tenant cloud, right, so like every customer gets dedicated compute, dedicated database, dedicated iout and you can load test your cloud and not affect a single other customer because your compute is again completely isolated from everyone else's and your database is also, you know, because we use rds, and so your database has a specific number of eyeops and it's you know, we're we're using Amazon, trusting Amazon to basically do that. That isolation of all those things because everything's true, you know, shared underneath the hoods. But but AWS is even really good at time boxing things and like limiting io and so that's another benefit to our customers where they don't have to call us up and say, hey, we're going to go test. We're like, go ahead and mow test it. It's your box. You're not going to affect the customers over here because they're on their own hardware, but you're going to crash your own stuff, but you know, go for it, you know, So yeah. I break it up too, because these are just things I think that sometimes people don't necessarily realize could potentially be issues that are larking. I'll be I'll be the dissenting opinion. So I like, I totally agree that you you sort of get some of these aspects for free when you change your architecture paradigm from from one to another. For instance, for us, we've had to go in a different direction because we want to be able to have a single point of reference for a lot of our architecture. And as we. Focus primarily in the SaaS version, that simplifies a lot of a support request or triaging or logging, et cetera, because everything is just rolled out into the same stack. But that means we've had to invest a lot in how do we deal with increased scale. I will say something like load test away against our service, Like we've had to put so much effort into understanding how to increase scale that no, like ten companies one hundred companies coming at us at the same moment, it's just not going to matter. I will say that that we'll hit a bunch of rate limiting stuff that we have in place, like you will start getting blocked, So make sure you have a second account ready to go, because if you start running this on your primary account, like you will probably have a production downtime when you get rate limited from doing something. But we've had to split rate limiting in a lot of different ways, like per user, per application, per individual tenant, per service client. So I mean you are making a trade off from one to another. And if you're going down one like a different path here where you're going down multi tenancy, they have to be problems that you want to solve. We were interested in solving these problems, and we were cognizant of like our team having worked in areas where there was a lot of historical challenges that they've experienced and they know their way around building large multi tenant systems. But I think it's a really great point where if you don't have that expertise, that you're going to get yourself in trouble, especially when you're providing infrastructure level products for your customers. But aren't they then just testing your rate limits? So like the counterpoint of that is that, I mean we all have rate limits right because especially at like the WAFT level and the infrastructure level, because you have to like not be flooded, but you can take those down like we can. We can basically say like okay, we're going to take we're going to isolate you and take you out of rate limits, and it's like go for it. Literally just bang on it until the servers fall over. My response is always like what do you want it to be? Like? You don't want this to be like actually no rate limits, like you want there to be something here to happen, and so yeah, for sure. I mean, if they're testing their own software, I think this is where the mistake is. They believe they have a need to sort of validate how our software is going to respond to their needs. And I think that's the fundamental flaw here is that they like, either you're paying us, so you trust us with this product and we give you assurances there and if you're not willing to trust those, like you may think about why that is, like why is it that you actually want to take these extra steps. We do get questions like oh yeah, how much can we have? And I'm like how much do you how much do you want? Because you can have that much, it's not a problem. I assure you, you're not going to find out where our service is going to fall over for you because it's it's going to scale automatically to handle whatever you throw at it, and you can for sure test. That if you want. But often I find the bigger problem is when rate limits come into play, is that they're usually at a moment where your customers are not necessarily compaired to handle the rate limiting. So even in a single tenant architecture, you know, what do you want to happen there? Do you want just one user to get kicked out of their flow? Do you want to use Like I think this is like an unsolveable problem because our customers will say we want the right thing to happen. I'm like, I don't know what the right thing is here, Like why don't you tell me what you think the right thing is? And that's probably how the service works, and then they usually get stuck because it's very difficult to correctly answer, like what is actually the thing that's supposed to happen? Yeah, and it's going to vary for each customer, right, Oh yeah, So it's semantics, So no, I totally agree. So I guess there's you know, it's it's just more of a statement. But there are benefits to both. One of which is like, if you have that level of scale and a multi tenant system, yeah, you're gonna have to really think about a lot of these constraints and how you change your rate limits and how you can get all that that much data through AWS versus if you have a single tenant system, then you're like, okay, well I'm just going to focus on singularly getting that through one little window just for that customer and not worry about everybody else because they're on their own, right, They're they're doing a different type of throughput. So I got I got something that you know, I'm sure is going to come across as part of the controversy. It's so much easier to deploy upgrades to a multi tenant system than it is to n repeated instances of a of a single tenant system. Right, That's that's just that sound dangerous. And they're like, well yeah, but off zero and every Cognito does and everyone else does, and I'm like, well sure they do, and they will break you, Like if they change something, you will be broken and you won't even know it, and like just go look at I mean, like there's just hit Reddit and say, ah zero upgrade broken, and you'll find so many people complaining about some change that off Zero or Cognito or Microsoft or somebody did that broke their entire application. And so our meth are sort of like theory on this whole thing, because you really actually don't want a multi tenant upgrade, right, you want a single tenant upgrade. You just want the ability to do it very easily and seamlessly. And so what we do is we say, we release new version of software, Please bring it back to dev, run it locally, run all your tests against it, make sure it's completely perfect, then schedule an upgrade. And then you basically we just we allow them to schedule an upgrade, click a button, we upgrade their services, and they're they're off to the races, right, And so we build processes and tools that allow this to have, you know, our customers to do this very easily, but it's very important, and we talk to every single one about it, is like, please take this back to dev and run all your tests against this new version before you move into production and make sure nothing's going to break up. Auto upgrades are just dangerous inherently in the industry, and like build tools, dependencies anytime you're automatically upgrading something in your stack without fully testing it. Beware. Yeah, I mean, I think you're absolutely right. I've been on this particular horn for quite a while about companies or software dev teams that automatically upgrade dependencies in their requirements text file or package chase on or whatever have you, with the argument of automatically getting whatever security upgrades come with us. And I think what I really see here is a responsibility model, like who is going to take full responsibility for their being a breaking change somewhere? And it sounds like, you know, with these other competitors out there in the market, they don't take responsibility for the breaking changes that they make. You've made it transparent and we promise no breaking changes. So I mean, there, it's really ridiculous that you can be in this state. I mean, I would be totally okay with Amazon taking full responsibility for no breaking changes if they are going to upgrade my RDS instance or dining with tob et cetera, other models, But it really can't be the case that you're using a managed provider and they roll out features that can break your production software like that, that's not a real that's not a real solution. In my mind, breaking changes is a very hard thing to define. Yeah. Right, So, like there's three levels of bugs in software. Right, There's a top level bug that is simply something that was inherently unfunctional and then becomes functional again. There's a semantic bug at the top level, which is I changed the semantic of something because it was not correct previously according to our docks or whatever, and now it is. And then there's nested semantic changes, which means I've called through service through service. Through service through service, the one way down at the bottom changed and it revealed a bug all the way up to the top. So I go fix the service way down at the bottom, and then the one at the top gets magically fixed. Okay, so when you have a patch release that's fixing a bug, it's still possible that someone's depending on the broken nature of that and you blow them out of the water. And this can it's literally just a dot release, right, So I could have thrown an exception and now I no longer do that. Well, the smart developer is like, well, I'm going to catch your exception and I'm just going to handle the path where like it's fine, Well, now you're returning the status code that I don't expect. Oh crap, I was expecting a five hundred. Now you're returning to four or one. Dude, like you're killing me your smalls, Like I can't. So, yes, you say you have zero incompatible changes, but that is too hard for a standard developer to reason through. So the only way to truly figure this problem out would be for the build tools, the testing tools, and the development time tools to basically certify that the entire landscape of all public things in our API, our code, whatever it might be, that people can't consume, here are the breaking changes, and here are the non breaking changes, right, because there's every release has breaking changes depending on how you're using the tool. You bring up such a good point. We're in a certain place we were discussing before. The call is famous for this. I seated at multiple places, but there was a lot. At this place. These are insanely hard problems to solve. And what's happened in the software development industry is that everyone got so excited about new languages and frameworks and building apps fast, and now there's like, you know, you know whatever Jive coding or whatever it's called I don't know what yes, and like bobob coding. So like there's all this stuff that we're just like we're just throwing it at the top end. We're like, oh, this is amazing. Look at all the stuff we can do and these cool frameworks and you know, React and all these things, and we forgot to go fundamentally solve software engineering problems at the core level, which is like, how do I certify that this version and this version are unquote compatible at the binary level, compatible at the public API level, compatible at the consumption level, compatible at the run time level. We don't have tools for that. There are literally no tools in any language that certify those things because we just as an industry forgot about them and we assumed that the developer was smart enough to do them, and they're not. Like, no one is. No one can know the entire you know, code base and certify that it's compatible. It's impossible. I made fun of some of these new terms. My son's a software engineer, and so he's like, Oh, I'm going to vibe code this thing tonight. I'm like, god, dude, vibe like an authentication just like, oh my god. There was actually a post about using Claude to generate two compatible integration and how much it failed basically even with the driver being a very experienced senior engineer in developing some like helping develop some of the standards. Like, that's how ridiculously not safe it is to do that. I do want to call out, like, this is for sure a nearly impossible problem. And I don't think you've you've even sold it enough here, so like let me, I just want to share that. It's like if you haven't if your if your service like returns an enom like a value you know, zero, one or two, and you add in the ability for it to return a four, like is that like does that break someone? And it's not a breaking change technically, but it is it is it for sure will break someone because in most software languages there is no code to say, like I expect only these results and if I get a different one, what to do in that scenario? And so you will be putting your customer in a scenario where their system will break in some unexpected way. So I think if. You're ready to go down this approach, if you if you run managed software like a cloud provider, or you've done something ridiculous like we have. You have to really understand the system thinking approach, like what based off of our current API, what did our customers write, Like what magical thing happened in their head that they wrote down that was was correct at the time, and now after this change is no longer correct. And so very often when I say no breaking changes, I mean that means like you can add fields, and even that's a little bit on the edge, but like renaming things or adding error codes, you know, we're very careful about. I think a standard one is like don't don't, like you have to be so careful not to over engineer anything, because that for sure means later you're going to be like, wow, I wish I hadn't put that in the API, because now someone could be depending on it, and it's impossible to know what fields someone's depending on with a particular get Now, there are some tricks here for anyone who actually does care and does decide to do this. You can embed the assumptions in your SDKs that you roll it to your customers, and then you can track which SDK versions they're using to understand what sort of things that they'll run into. And by making sure that the requests that coming from their service are all using an upgraded version of the SDK. You can be sure that any dependency that rolls out for that customer would not have a problem. Which does mean that in our own code base we do have feature flags for certain customers to potentially take certain dangerous upgrades. But fundamentally we do have to segregate by customer and understand the SDK. And that's still not sufficient because customers will delegate out integrations to all services to backstage and whatever other internal developer tooling or other client that you don't even control, and so getting that that integration to work correctly is just another huge thing. Like it's not always like you can get your customer on the phone and promise them a huge discount to make to make a change, or you know, threaten them with a huge increase if they are still on an old version of Kubernetes. And I mean, I'm your ofth provider in the cloud because you know, oh the cloud friders are doing that now too. So I do think that there's a whole spectrum here of problems that you're going to run into, and you have to be conscious of how you're going to tackle every side of it. Yeah, it's tricky. The software engineers have to think so hard about the architecture for their unges, their APIs. It's like, are we going to do? Are we going to version them? So like you know, slash V one, slash V two, slash v three, And then when we make a change, when do we upgrade the version number? And then is that version number tied to the SDK and the SDK only calls into this when it's this version or cause the old version? And can you make a compatibility translation between V one and V two and B three? I mean, it's like it's a lot of mental overhead just to make a simple change, Like, ah, dude, I just want to like return this extra field in the API. You're like, well, is another field dependent on that field now? Because if you have dependent fields, you can't make that change unless you version the API and then version of the SEK, and then you have to make sure you have a compatibility layer between those two versions. And their brains explode and they're like, dude, all I just wanted to make a feel change. Sorry, this is why I don't such faith. And a terrified for this specific reason. It really really does. And like at the stuff that I've been seeing playing with different products that are coming, it's learning it's hallucinations, and like, I just don't see how this stopped. Like I know there's obviously balances, and you know it's people obviously like test these before it comes out, but once it's like once it's deployed and it's learning its own hallucinations, Like how do you stop it? I don't. I love I love how you said obviously people test these because you know I. Once you haven't like deployed in your environment. It's what is it the Schroeder's cat, like if it if it's but I forget how that goes and if I've even pronounced that correctly. But Schroder's cat where it's like, if it happens in another world, does it then become reality? Like if it's hallucinating on something that's false, but now it has become reality, Like is it reality? I don't think it's a phenomenally existential question. What I was going to ask is you know how the S and MCP stands for security? Where are you on the spectrum of AI is terrible and going to ruin the world? And is it the best thing ever created by humans? And I guess we know where Amy stands on that particular point. I mean, I'm sure it's going to excel at certain things, but solving the types of things that people think it's going to solve, I just don't. I don't see how that can happen, because I mean that we're fandomically, like we're at the limb of ourselves and we fail. So how it's really hard to engineer a prompt that tells them about your entire software development life cycle and the way that you've architected your entire system, right, So, like your prop would have to be like generate me a new API and blah blah blah. But keep in mind that we version our APIs this way and these are breaking changes. We use this SLDC process to get our SDK updated. You have to link the SDK to this and that and that. There's the AI is. You just can't do that, right, So, like AI is great for helping me code complete a four loop, right, and it'll guess based on the things that's seen in the code and where I'm at and logically what I want to kind of do, and I can make a prompt. It's like, hey, make me a map reduce on this list, and okay, I got you. Great for that stuff. Generating full code bases and let alone adding to existing code bases large chunks of code freaks me out. And I always tell people. I'm like, don't let that anywhere near your security layer, Like anything that has to do with security, please, please, please, please please do not let AI generate it and just ship that like, let it help you, but review your code, test it, do a security audit, do a pen test, do a load test. Like you still have to do all the things that we're doing which require knowledgeable software engineers to do them right. You can't just let AI do that either. So like the doomsdayers are like, okay, well you know engineering is dead, you know, no, no more engineers coming out of college. No, we're stopping. We'll just take that off to college curriculum. And I'm like, we're so far away from that, so so so so far away. Like you guys, keep going to college, let go get your degrees. That's going to create more jobs. Yeah, it will, it will, It's like. It will. So I think the biggest problem here isn't that You're absolutely right. It's that people believe that it's going to take over stuff, and so it's already affecting things like universities and whatnot. I mean, there is something to be said for specifications. I find that if you have a very well written out spec using an LM to generate on a transformation so a translation or getting it written down, let's say, open API specification into something else that's programmatic, so an SDK. But at that point, why are you just not using a rules generator? But maybe the thing that generates the generator, that thing could be l M based. So you know, I see, I see the hesitation here. You know, I'm totally totally on the same side for the most part. Yeah, for sure. Yeah, it's again, it's a tool. Use it effectively, you're good. Don't use it for everything like that. Just that sounds dangerous. I like the argument that people have been using Oh but there were a lot of naysayers about the Internet when it came out, So can't you envision at some point AI also being great and the people that are jumping up and down right now like they're just on the forefront of innovation. And I say, remember the Internet was never designed to host applications. And I think you've established some converts probably today after. You know, it's interesting because from my experience it's been that a full multi tenant solution is never the right answer. And I also am not a fan of full single tenant solutions. Like we end up usually somewhere in between. We don't have millions and millions of database instances running one for every customer, but there are things like dedicated tables per customer, or dedicated certificates per customer, or dedicated you know, CDNs, et cetera. But there are also some shared components as well. And I feel like understanding where the direction the business is going and where the value is that you're providing helps to pick the right part of the spectrum and not assume it's like a pure extreme case, like it's only A or B and nothing in between. Exactly. Yeah, I mean we're you can consider us fybrid too, right, Yeah, when we run in the cloud, we I mean we're even looking at like starting to share a single database instance across like lots of small customers because the cost of scale there is is way better and it's way easier to manage and multitude of reasons, right, So we're we're already looking at starting to do some of the we would do like a database slice per customer. We would probably put them all on the same database and just like our tables or anything. But but yeah, I totally agree with you, right, So, like we we want everybody to have the best of both worlds, and it's like, how do we do that most effectively? Though? So the coolest use case that we have, which kind of is only usable and you know, downloadable, single tenant and they deploy it in their own special way, is that we work with a satellite company and they actually have pushed fusee not up to these low Earth orbit satellites, and so we can say that fusion oth runs in space. It's pretty sweet. And so we're like, we like to say we're on all all all the continents and space, although Antarctica we faked that one. We just you know, had somebody that was doing a tour down there, like fire a FUSEE near it, and then we said, okay, we're on Antarctica now. But yeah, for. Sure, amy have have we converted you any if I managed to at least pull you back a little bit away from multi tenant is always wrong and security vulnerabilities, I definitely. I mean, if you's in a perfect world, there's one scenario, but in a practical world, I do agree, like it depends it has to be crimination. Yeah, I see some uh past trauma there really starting to show. I think we've had trauma on both sides. We're likeable. It was expensive and so hard to manage, and how do I get this into production? And then it's like, well, we're using this service and it just crashed, you know. I mean I think we all have a decent amount of stories from both sides. Yeah, I mean I think my my single tenant one was definitely we were running Jira, and uh, I mean that that's that's already the beginning and the end of the whole trauma right there. Yeah. But it turns out that when you're running it and they tell you that there's a major change, you would hope that the provider gave you the capability to automatically migrate the database and all the backwards incompatible stuff, But very frequently there would be database crashes and you would lose all of your data, and it's just like that's a thing that happened, and I think the pendulum swung really hard to the other side was like, we don't ever want to deal with this ever again, because we don't trust companies to provide us the tools to. Actually do the upgrades. And now we're back going I think, really coming the other way, which is, yes, we don't trust companies to you know, still be alive or you know, running their APIs in a non backwards compatible way, because now they just release new stuff all the time and it breaks. And I think this is just a story of it. It's no matter what solution you pick, it it's wrong or bad in some way. Everything everything is very much it depends and then it changes constantly. So you know what you have one year, the decision you make you're one is going to be a very different problems the decision you make year two, three, four, hopefully. Yeah, I totally agree with all things said. Everyone's camp okay, Yeah, that's that's a good, good camp to be in. I think we've we've hit the AI topic, which everyone is obligatory. Now everyone has to hit an AI topic. And the truth is, we actually have quite a few episodes recently in the past few months that have heavily delved into ai. UH and LMS and anything on the topic. So if there you are, if anyone is interested in reviewing those, there's a there's definitely a plethora of not not a limited amount of information podcast episodes, so you can go into hours and hours where we debate one way or the other. So then let's uh, let's move on to picks. Amy is okay if I put you on the spot? Sure, I you know, since I'm like new here, I have a slew in my head. But I'm going to pick walking since that's what I'm doing today. Morning is usually when I have the most energy. So I think I've gotten I'm looking on my tunel here almost two. Miles treadmill do you have I've been like condubating getting one. I am a Peloton fan girl. That could be my second pick, but in all reality, I was looking so I have the first version of the Peloton thread. I like the slated one. However, this one I think is slightly smaller and so it works perfectly for my face. I have a desk up here. You can actually I don't have it hooked up right now, but it's the external monitor. You can actually hook up your laptop to the external monitor too, so it works really good for like a little setup. Nice, all right, I'll look into it. Brian Feelfred, I guess why I'll go. So two. I have two things. Actually one of them is just totally random because it just popped into my head. But like I like, like, you know, snakes on a plane style movies, right, So, like I watched one recently which I thought was was really hilarious and it was just like sort of like way over the top in terms of like the the way that they did it, And so that was that was one of them. Now I'm just Fight or Flight and other kind of like stakes not a plane movie anyway, if you're into that kind of like heavy gore, like crazy comedy, like ridiculousness, that one's a fun movie. But the one that I'm really interested in kind of because it's you know, kind of goes along with what we're doing here and talking about, is there's this cool tool called search Craft, and I'm like on like this huge kick with it because it's it's a completely new search engine. They've rewritten, you know, from scratch. It's not built on Lucine, it's not built you know on Elastic. It's completely different. It's built in rust. It's like super efficient, highly performance, requires like you know, seventy percent less compute. You can run it locally, you can run it in their cloud, and you can run it in your cloud. It's just a really really cool evolution of something that you know a lot of apps in need, which is just a simple search service. And and I think they really have something that's gonna change the way that we think about search. So anyway, that's that's the thing that I've really been gone how about lately. So anybody wants to check it out. It's just I think it's searchcraft dot. I owe searchcraft dot I owe. I think is it Well, we'll get the link and put it in the in the show notes. Is this just like every company I should be using it? Or is like if I'm just developing something myself, it's there's a great opportunity to also pull it in. At some point every company should be using it. It's because like you know, you know Elastic, right, it requires like seven servers and they all have to have like full pi ops and they have to have ten gigs a RAM each and it's just a it's a hawk, and is it impossible to run and manage. Search Craft is not quite there yet. Their clustering is coming along, and they're really they're really heavy in development right now. But like, just as a nice search search is for an app that you're building, you should totally be using it. Don't even think about open search, elastic search, just skip that junk and go straight to search Craft. Yeah, I'm I'm totally with you in general here. I find that, first of all, i'm really suspicious of any application that's built in Java. That's that's that's my first thing. But the second one is is that even the promised managed services in a WS with the other cloud providers, they're they're not really fully managed. It's just pretty much like you don't get access to the easy two machines, but you still have to pretty much manage it yourself. So there's a there's a huge allure of. Just not having to understand the complexity of index management UH and document management and just using an API that runs with low compute or memory impact. So it's huge and it's hybrid again, so you can not locally run in the cloud. Okay, thank you, Okay, So my pick today is going to be uh these uh Scarpus shoes. I actually really liked them. I found them randomly one day when I was walking in in Zurich. They're wide enough if you if you want wide shoes, and they're must in Switzerland if you just hop on some walking trails. They have vibrum soles. I just really like them, and I never heard of them before. Apparently they were made in Italy, and I guess today I'm just one of the lucky ten thousand. Scarpa correct me if I'm wrong. But they started making mount rock cling shoes right so then. Oh yeah, so in Switzerland it's it's been a huge path down here. So really realistically these are just regular walking shoes. But you can get all. There's a six ratings for hiking in Switzerland T one two, T six and T one's like flat ground. You don't there's no danger. All the way to T six you're totally exposed, risk of death everything. And they off they have dedicated shoes for every single type of hike you could possibly go on. Ones that support crampons, dedicated hiking shoes, you know all they go all the way up to your like above your ankles for support dedicated like climbing shoes, so you know, the Will band in case you're you're into rock climbing and whatnot. They're really nice, Like I think they really high quality, really soft h They're some of the best ones that I've seen. They don't work for me for hiking, unfortunately, but for walking around everywhere. I absolutely love them so far. They're great. That's awesome. What was the model on those? I think they're the Planet Mohido Suede. Okay, because they've got like a bunch of different because I used to do rock climbing and stuff, and they've got like their whole climbing thing. They've got trailing, They've got like a bunch of different styles. So I'll have to go check those out. Yeah, I mean these these actually don't even have the virum soul, but these are. They're they're pretty. They're pretty nice, thick thick soles that don't get worn out. I have a lot of traction. But yeah, I mean, if the scarp is fit, I definitely recommend them to to anyone. Uh, it's actually really interesting. Like I've been I've really gotten to shoes lately because I've just realized I have no idea what. I've been buying my whole life. Binding Appropriately fitting shoes actually a real challenge. Don't just put anything random on your feet. And so like I've been watching like what do people wear out there in the world, and a lot of people are just wearing crap, honestly, Like you go to one of these regular shoe stores and just buy stuff, and there's such a huge difference from buying shoes like this versus just picking up the one shoe that will you'll your foot will go through the sole after a year or two years, and something like this that just keeps on lasting. Honestly, Yeah, yeah, I totally agree. So I was like at a conference and I just my shoes was shot. So I jumped into like an H and M and I'm like, I'll just grab anything that's in there. And I grabbed them and put them on my feet, and I walked like another mile that day, and I'm like, oh my god, my feet hurt so bad, and like by the end of the show, like the heel was worn out, the bottom is falling apart. I'm like, oh, well, that's why they when they cost like forty dollars, you know, thirty five bucks or whatever. It was crazy. Yeah, I mean that's pretty cheap, even for a shoe. Okay, well, I guess that's a good point, probably at the episode before we get into into too much into into shoes and hiking. So thanks Amy as our temporary guest expert here and reliability, And thank you Brian so much for coming on the show. I hope we see you again. Yeah, this is awesome. Happy to come back anytime. Thanks for everyone listening, and we'll see you all next time.