And we're live for anyone who doesn't know. We also live stream this on LinkedIn, YouTube and twitch. At what time is it? It's uh what ten thirty Eastern time on Tuesday morning? It's two thirty pm UTC. If you're not in one of those time zones, you're gonna have to do the math yourself. But I love you for listening and watching the live stream. Meanwhile, what's up born? Hey, thanks for turning that up. And I always get the time zones, you know, totally wrong living in Switzerland where at plus two UTC at Yeah, I mean some of those special ones in North America are quite challenging. Yeah yeah. On my team currently, I'm in GMT minus six. I have a person in GMT mine three, one in GMT plus one, and one in GMT plus two. So you know, I only is thickering here because we built a product a stand up bought called the stand Up and Prosper and it was part of an old product that we had built a long time ago, and very quickly we realized, you know, we may actually be one of those companies that now has to understand really lots of intricacies about how time zones work. Because like we have to send we'd send messages to people when it's eight pm or eight am in their time zone, and we try to rely on like the Slack time zone for that and it's not always valuable. And then we also have to display it an RUIS and also get it right on Slack display and trying to schedule in a different time zone is always such a challenge there. Yeah, I worked at a startup. One time. We did medical imaging for trauma hospitals on their trauma patients and they decided to rewrite the core application in Java. We're not going to go down that path, but that's what they did. Right the very next time that daylight savings time went into effect for the US, we had about two thousand trauma patients disappear from our system because they didn't account for daylight savings time. So that was a good time. Yeah, I mean, there are definitely some things where we allow events in the future and then we try to hide them being displayed, just because prepopulation of data is so important for production scale applications because you don't want to wait until you know something actually happens before having those events in your database. So I can totally imagine that they're omitted in the future, so don't display them in any of the UIs to users because that would be confusing. Right. But speaking time, it's that time of year that we're coming up on the end of year, the holiday season, which can have a lot of it can mean a lot of things depending on what industry you're in, how your company handles closing the books at your end, and so that's what this episode is all about, is just kind of I think it's kind of just us talking out loud as reminder to ourselves as well as a reminder to the listeners of our show that those things are coming and to start thinking about them, because we're going to be executing those tasks here before you know it. You say that, and the first thing that comes to mind is when it hits to be cyber Monday, I guess, or you know, three weeks ahead of that, because realistically, that's when every year it just gets soon and sooner. And I remember working at so many companies where someone was telling me, you know, something's going to be different now, But as an engineer, I'm just like, I don't. I don't know what that means. I don't I don't know how to what do want to do that information right? For me, the hardest part has always been getting that information out of marketing because they spend a lot of time and effort doing campaigns, marketing campaigns where this's ads, email blasts or whatever to build up these big sales events, And in doing this for three decades, I have never been successful in convincing someone from marketing that, hey, if you're gonna funnel an extra two million people to the site, did you ever think about giving someone an engineering like a heads up? Yeah? I mean that's always the most ridiculous thing. I think it leads the conversation sometimes in an disappointing direction, though, because it goes from like, Okay, well we're going to sign out two million emails, but how many? How many of those are going to be opened? And then you realize by many marketing departments, the answer is we have no idea, and it's like, is that two million users coming to our site? Or is it two hundred thousand, is it twenty thousand? And so a lot of times that stuff happens and there's zero impact to the business, And the most effective thing you can do is, you know, just stop performing those experiments in that way unless you actually know what the click through rate is, what the conversions are going to be. And that's interesting you say that because when we build authors a while back, one of the most common concerns we heard from our customers that we're using our competitors is how they charge by monthly active users. And that means if you sign in, you get immediately dinged for a user count that then is amateurized through the whole month, which means you're basically paying for thirty days worth of your advertisement, which is a huge cost, especially when these marketing campaigns go out. But yeah, I mean yeah, engineering very rarely knows what to do there, and so we went to a pure charge by usage, so you know, if they only log in once, there's not a problem. But I still think most of those marketing organizations they have no idea how many users are going to actually come to your site. Yeah. I think the big takeaway from that is that I chose the wrong career profession. I chose one where my performance can be measured. That's my fault. Oh you say that, and now I really wonder how are you? How are you being measured? Because performance measurement in selfware engineering, I think, like anything is, it's much more difficult than widget production, which is, you know, highly repetitive in which you can actually measure the throughput and you know, quality metrics of the individual machine that's that's manufacturing those widgets, whether it's human or automated. Yeah, from my perspective, since I'm mostly infrastructure based, a good bit of my performance evaluation is based on uptime and ease of deployment and error rate and things like that. But I would I don't have any data to back this up, but I would say that a bigger part of my performance evaluation comes from perception of the teams that I support. Do they feel like they're being Do they feel like they're getting the level of support that they need? And that just comes down to customer service. Recently had a situation on that exact same topic where someone felt like they were not getting the level of support that they needed, And the only thing I changed was at the end of every meeting asking them, Hey, is there anything that you are blocked on because you're waiting on us to do something, and just doing that, just calling it out to attention, you know, was enough to change the perception of the performance. Yeah, I mean, I think that was one of the biggest learnings I had from when I started professionally, is I thought that my deliverables as I coded them, as I architected them, was worth something. But I knew that there was this thing of No, you're judged by politics is the word I would use to describe it, which I really think is a misnomer honestly, because you aren't fairly or equality judged on what your outputs. You know, it is really about the perception. And talking with whoever is in charge of promoting you is really the first important step that everyone really needs to take on day one when they get a job. How am I actually going to be evaluated and over time whether the things you're working on are actually relevant to being promoted or even just staying in your job. Yeah, for sure, that's the whole like social side of your career. Like your technical skills are only part of it. I mean, if you can get by working like not writing any code every single day and just you know, saying like spitting a good game to your managers, you know, you can get really really far, so much so you laugh, but so much so that there's actually a name for this, and it's called effective managing up And you actually see a lot of not helpful, probably not good managers doing this and that's how they got to those positions that they're really effective. It commits other people that they're doing things. Yeah. Yeah, it's which goes back to your earlier comment that is very political in the US, that's the basis of our entire government. Without turning this into a political comment, let's not go there. We shouldn't do. Yeah. But so back on topic here, you're in what does what does your end look like for you or office? Honestly, we don't really pay too much attention to it because we don't go through the marketing cycles. Uh, there is usually some amount of uptick that actually starts at the beginning of the year, especially for for two reasons. One for our products that deal more around the ASIC nature. So I mentioned we have a like number one or two slack bot for handling stand ups. And at the beginning towards the end of the year. Beginning of the year, there's a lot of vacations. You know, people aren't all in office. It's more remote work, and so figuring out how to manage your stand ups in an acinc way is usually valuable for authors, which you know, obviously is our main product. Budget comes in at the beginning of the year, and so that's when a lot of teams start thinking about more what's going on there, and towards the end of the year they're dealing with all of the tech debt, all of the mistakes they've made in the previous six to nine to eleven months before they get to that point. And I think one of the biggest ones is the marketing spam, you know, having actual users come on board. And I actually remember that earlier on in my career, I was working at a company that was had billions of dollars in revenue coming in every year at the cyber Monday, Black Friday, you know, huge deal days, right for getting in lots of users to the site, like not even having to ship out orders. I mean, the biggest problem is actually the manufacturing world having to then turn around and manufacture and then ship that stuff out when it's not just sitting on your shelves. So much so that I remember that frequently management quote unquote, I don't know who these people were, you know, you hear it through the grapevine, like it's coming. You know, there's going to be a massive wave. You need to prepare, prepare for that, so only really important software should be done at the end of the year. I feel like it's the even more ridiculous version of don't merge code on Friday afternoons. This is don't don't even think about code between now and after Christmas. And also there aren't any people there in the office, so maybe also you know, push that out to like, you know, January eighth, so from now until January eighth, everyone's on vacation. And that just always struck me as so utterly ridiculous that a software engineers could anyone really, but I'm gonna blame software engineers could determine the difference between like a critical thing that needs to get the production and like feature and something that's not because you're often driven by that perception of this needs to get done, and you're often not in the situation of being able to even make that decision. Yeah, you know, up until recently, I had never participated in the year end code freeze strategy. But at Polygon we actually have certain platforms where we do a year in code freeze, and it's frustrating to me because I'm like, no, just just just know. This is the part of the show where I could wish, where I wish. It could cuss. But but the truth is is where we have some very very early stage products and some we've built some things that, for all practical purposes, have never been done before. So when it comes to monitoring and alerting, we have room to improve on that because like, if you've never done this thing, how do you know when the thing is doing the thing that it's supposed to do and how are you going to alert on that? And there's there's certain stages where we don't We just kind of watch it until we get a like get the pulse of it, and then we can put monitoring on that. And so that's the I think that's a big reason why we're reluctant to do. Your end releases. And then also, you know, combined with that, like you mentioned, a lot of people are out taking vacation and then we're completely globally distributed to so we you know, async communications are very very async. Yeah, I mean there's a platform thing there, right, Like if you write games as your company's product, there's a validation process, same with I mean similar to if you're in healthcare some other apps, if you're deploying a mobile app for iOS or Android, there's a turnaround time there. And so even if it's even if your company is absolutely perfect, we do live in a more just in time world, a more agile world where you are depending on third party partners. So even if you do the work, you know realistically it's not going to be actually able to get to production. So rather than having it stack up where you're creating a lot of waste there that someone still like, you don't want tons of merger requests sitting open waiting for someone to come and test and review your code. You want to get those production as soon as possible. There's no reason to start that work because you know it's going to be blocked somewhere. So I do totally get that, especially if you're depending on clients, right, you actually don't want to do any work unto your client already says why is it not delivered already? Right, Because if they're not already waiting on you for a chance, you're going to do the work and then have to wait on your client and then you're like, Wow, those three weeks to three months are longer. Unfortunately in some concern cases that I'm sure you've got well, that you could have been doing something more valuable in that time. Yeah, for sure. That makes me think of doing mobile development and getting your app approved to go into the Apple App Store? Is that three days? Is that three months? Who knows? Is it consistent? Nope? It could be one right after the other every single time, and that makes it really challenging to roll out new features. And then compounding that, just because you release a new version in the App Store doesn't mean people are going to update. I remember when I was active, we had a very very small group of people, I mean less than ten that hadn't updated their app in two years. They were running on a two year old version of the app and perfectly happy, but it was causing us some painful technical debt. And we finally had our marketing team. They did a marketing and customer service. They jumped through all kinds of hoops to actually figure out who these people were, and then I think they were able to contact them and ask them and send them a gift card or whatever to upgrade their app. I got two great stories on that point. Actually, the first one is having built a very technical API. People talk about deprecating old stuff, but realistically, if you write it, you should just pretend it's never going to get removed. It's so much like the amount of churn challenge, complexity of trying to convince your customers, as you described, to make a change there where there's zero, like, there's negative value in making the change. Right, it's all work, it's working today. Why would they why would they realistically change? It's just not worth thinking about in most cases. And there are so many companies I've worked with who had this mindset of, oh, we'll get them to change, and so they then have like five or six versions, like one every couple of years. Every time there was a new engineering lead team you know, owner of that product roll out something new. And so we started having version numbers in our endpoints and things like that, and so you see all of them available in production. Still some of those companies I worked with, you know, they still have those oldest versions still there because they don't have that amount of pull within their organization with their customers. So actually, one of the things we did when we wrote Authoris is that we just completely discarded that notion, like there are no versions, it's will be like this forever likely. It's it's so totally unrealistic. Now, we do make some changes where we then like update our documentation, we may remove reference to fields that we still support, but we don't remove them because the amount of overhead it goes in it is really high. The other thing I'll say is I am totally that person that has old apps on their phone, like from multiple and I think there's sort of like a two sided extreme here where both of them are not great. It's there are tons of apps out there that every single time there's a new version, it's like you must update to use the app, and I hate it. And I'm gonna call out Signal in Google because they both do this. They tell me all the time, Hey, there's a new version Google. At least you can skip it. But the pop up always happens at the wrong time where I'm like, can you interact with me at the end of my workflow because then I'll accept the upgrade, and Signal just says no, I'm sorry, you can't look at your messages until you upgrade. And that's just an incredible pain and I totally get the other side where people are using the app multiple years later. I'm just so afraid of it breaking something or draining my battery life because I can't trust these apps and there's no way to stop them from interacting with the Google Play integrations to get battery hell or send messages, notifications, et cetera. Sure that's the I'm the same way, Like as a developer, I'm like, why are these people not updating? But as a user, I'm like, I'm not updating that it's working just fine. There's no way I'm going to risk this thing breaking. I'll tell you a secret. And maybe this is letting the cat out of the bag here. I upgrade apps when something looks like it doesn't work, and I won't know why it doesn't work. So you could, for sure if you had some way of feature flagging, like a couple buttons just breaking in a weird way, right, Like you know, I use Plex and if you hit the like you know, if I hit the play button and it like pauses it or you know, starts playing and it like does it weird jump in the video and it keeps like it just starts happening, I'll be like, oh man, I got an upgrade. So that's the secret for convincing your users to upgrade is really, you know, have something where they don't think is really your fault, that you would have done intentionally to get the benefit. And now if you productize that as a service, you've got you next huge startup idea. Yeah, graceful like non graceful degradation. And yeah, we like our core metric for our ICP would be like the number of broken apps we have in production at that moment, right our customers are currently destroyed the user experience of one million users that could. Be that's funny, write that down right. But meanwhile, back to your end, Yeah, because that was our chosen topic till we wandered off, I. Think it's really challenging. Yeah, So in my mind, there's a couple of different categories of things that you have that come up at your end. One of which you brought up that I hadn't remembered until you brought it up is budgeting for the next year. So you got to start campaigning for whatever it is you want to do and building a use case so that it makes it into the budget. And then like we already mentioned potentially depending on your industry, increase in traffic for your end sales and special and you know, for a lot of companies that's just largely a scaling issue, Like, yeah, I've got auto scaling, it's fine, but I always like to go ahead and just kind of cross check everything before we get to that point, because we've probably made a lot of changes throughout the years since we last tested this really hard, and so I just want to make sure that we didn't inadvertently turn something off that we were counting on or something like that. And then their category for me is like ye're in closing of the books, you know, which in my mind that's that's a financial thing, but I also think there's an infrastructure side of it too. It's a good chance to. Check all your backups and your data retention policy and you know, restore from backups into a dev network and validate those and just make sure that all of that stuff is working as well. Yeah, I mean, you don't want to be that Australian pension fund company where they lost all of all of their one hundred billion dollars worth of financial bookings for all of the citizens of the country because they were using. One cloud platform that. Decided to be extra helpful in setting up their environment one day. So I don't know if you know the story here, but apparently yeah, so apparently there was an internal So the first the first knit is that they were using VMware and on prem which I totally get, I've been there. But then they wanted to do a lift and shift, so they did it and were hoping that the cloud of choice we're going to offer them VMware as a service, and they did so they shifted to GCP and the problem was the configuration that GCP provided for VMware was not sufficient to meet their needs memory ram zpu, I don't know what it was, and so they filed the support ticket to ask for them to basically create a sphere cluster for them manually. And if you've ever built a technical product that has a high requirement on reliability or quality, you'll know that the thing that you really want to do is make sure you go through the public interfaces. And if the public interfaces don't support that, then you know that someone's going to have a bad day sometime future. And realistically, it's a it's a good reminder where requests from one team to another one like the support team to the ops team or product engineering team should be quick to resolve either know we're never going to do that, or support it or yeah, of course we'll get that in in this case updating whatever public API. They had to be able to utilize it to actually generate the cluster in a needed way. But instead of going through that in the good old fashion and we can do it ourselves strategy. Uh, this is the team responsible for solving this task for the pension fund. UH ran some scripts to dynamically generate a cluster using a tool that was built for non production environments to automatically self implode after a period of time. Uh that you the default was one year and uh, you know, probably to stop probably to stop development environments from consuming a lot of resources after they lost their value. And it was used for this, and so a year later after creation of their environment, it disappeared, just as decided by the engineers of this internal tool. Nice. Yeah, so you know, if you may need to look all the way back to what happened last year at this time and use it as a metric for what could possibly be going wrong. I remember one year at one of the companies I was working with. We had a production outage soon after holiday, and that's who we called it. And it was related to the fact we had a table called calendar table, and I kid you not, the calendar table ran out of days and so it crashed production for all of our one thousand windows services that we had running. And the calendar table, you know, may sound like it's a pretty clever thing, but I what it actually was is a day count number as the index, followed by the day of the week that that day was, and that was the whole table. And so if it didn't know what today was, it wouldn't know what the day a week it was, so it wouldn't be able to do planning to figure out how much actual planning, both on the manufacturing side but also planning how many people that needed to be in the office manufacturing plan in order to run the plan effectively. And so the fix, of course was to insert more dacent the table. And of course the fix was someone runs a sequel script that just inserts, you know, one hundred thousand more rows into the table and no one thinks about this problem. Ever. Again, this sounds like mayan calendar as a service. I guess I guess. You know, maybe the lesson learned here is that when it comes to the end of the year, things are going to break it away. Yeah, because there's always always things like that where someone you know hard coded a year or didn't you know they're doing like math between months and then so whenever the month number decreases, it screws something up. And that's a really good point. Those are things too. I don't know that you can effectively look and identify those ahead of time without just a complete audit of every piece of code that you have, but it's something to keep in mind after the at the end of the year, when you go into the new year, when something starts acting really weird, don't rule those kinds of things out. I wonder if there's a framework that could actually be used and executed here, like like the thing to do on January second is control shift F through your entire code base or grock or whatever you're using and search for like WTF or slash slash to do and see what actually comes up. That's a great idea. Just schedule a reminder, like every team could potentially do this. I mean, I hate these comments. They don't there's so completely worthless in a lot of ways because it's like there's something wrong with the code that I'm looking at, but I don't know, I'm too lazy to do something about it, or there's just a time bomb here waiting to go off. At least explain what the problem is, right, like what am I looking at? Why is this weird? Why is it confusing? And maybe twods are okay, but realistically I feel like either it's something you want to do, in which case you know, file a ticket, or be like this is intended behavior. And actually one of the things we do is we add an explicit log statement, so we don't actually implement those cases, but we do log whenever there's a weird thing that gets hit, so it doesn't instead of WTF, it's like WTF, like log track this message? Why is this even happening? And someone should investigate so that if we do hit that use case, we know to do something, and if we never hit it, no one. Has to care. Yeah, for sure, I tend to. You know, I can't say I've never created those, but I have gotten better over time at in the comments putting heywill or if you're reading this, so that it kind of stands out and then try to set the stage so that future me knows why past me screwed me over like. This, Yeah, yeah, I totally hear you. I'm definitely Like there's lots of paragraphs related of content and comments in front of things that just aren't super obvious if you look at them, like why are we comparing this to the number seven, for instance, and like you don't want the comments like we compare this to the day of the week, Like that's not helpful, Like it's helpful when you explain why from a business standpoint or a product standpoint, this whole code section is split this way. Like if you're doing a like a merged sort or any sort of sort algorithm, or you're trying to find a maximum value or a minimum value or some arbitrary P value, Like don't say, oh, we try to find the P value of ninety nine, Say we want to find the P value of ninet ninety because and then fill that out so that you're actually explaining why the code is the way it is, because often that's not going to be included in the ticket. Maybe it is that generated, but usually not. And obviously you don't want it to jump to that or look at the code comment that's that's related there. Yeah. I think it was Don Felker that had to tweet a really great tweet that said that comments should explain why you're doing something, not. What you did. The code itself tells you what you did, but the comment should tell you why you chose to do that. Yeah, for sure, one hundred percent agree. But I don't think anyone's going to explain to my previous senior engineer who is working on a different team in a previous company at the end of the year, who at my request to not run production software on their own machine, decided to run production software for on their own machine for a critical piece of software for the entire company, And during a long vacation break in the company, the cleaning staff unplugged the machine and crushed the service. I kid you not. And the really ridiculous part about this story is it was actually right around a holiday. Like it was not not a good story. The ridiculous part is we were migrating to AWS at that time, and this particular engineer, who I actually think was a like a equivalent of a staff or senior staff right now, said I don't understand why we're going to aws. We have working software on prem in all of our data centers. That's sufficient. We have you know, x or it's running these data centers. And well, I think that statement sort of stands for itself. Yeah, I'm sure you've seen the meme of like the laptop where the lids partially closed with the sticky note on it that. Says, this is a production server. Please don't close lid. It's not a joke. Like I have seen that in my career like four or five times now, Like somewhere I am. That is a thing, And it's always in places that have a really experienced engineering division, and they have engineers in not experienced engineering divisions or staff that aren't it focused who were forced to install something and run that application themselves. So if you're on an OPS team and you tell one of your customers, hey, we're not going to do that for you because that's not our job, this is what they go and do. They go and install that production requirement that the marketing software that ends out all the emails to your customers on a single laptop somewhere and put a sticker on it and then actually find out about a problem. Like I would be sitting somewhere there'll be a laptop there and someone will go over and unplug it and I'll be like, you just turn that off? I have no idea why it was like that. It's weird, Like you should know that it was weird. Why did you do that? And then like two hours later, someone will randomly show and be like, hey, this laptop. Did you see what happened here? Did someone like turn it off? Like what happened? This has to stay on? And we're like, I don't know. Someone just came over and unplugged it and then laughed, right. And my favorite part is when that happens is someone gets really mad that you had the audacity to turn off that laptop, like how dare you turn off a laptop? It's like, wait, no, why are you running production software on a laptop? That's the core issue here. It's not the fact that I turned it off, it's the fact that you're running production software on your laptop. Yeah, you know. I try to think of it from a multiple perspectives. If there's a way to some how put myself in their shoes of they just don't have an understanding of what the criticalness is or the interim, the lack of flexibility, the likelihood of this causing a problem of the actions they're taking. I think human beings by nature are so myopically focused on short term and not deal it thinking about the long term. But also, I mean, we're the ones in the position of thinking about reliability and the operations of the software that's running and how to keep it reliable, and you've got to imagine that other people are not. Yeah. Absolutely, it's so hard to look at things from other perspectives. Oh for sure, because like, whenever I'm doing infrastructure work, you know, I'm pretty tightly focused on you know, high availability, fault tolerance, redundancy, all of that kind of stuff. But then. Like the next day, I can be writing code and you can look at the code that I'm writing and be like, has this guy ever heard of high availability? Because you just get so focused on solving that problem and fail to step back and think, wait, what other what other pieces do I need to make this? That sounds like there was a story there. Oh, there's probably a lot of stories there. And it's those that like at the time, you're just like, I'm all read in the face I'm like, I can't believe this is happening right now. I have to totally chill out. And they're the ones that make for the best laughs later. I mean, maybe it's a little cliche, but those really terrible experiences that you're you're gonna have between now and the end of the year, Uh, they're great stories. Like I got another one. We were the face. I was running a team that was the face for our customers U I and a bunch of back end services, and we didn't get any public praise. Is all about the perception like our stuff just ran no problem, And when it came time for the end of the year and increased load, we would often say, we don't need to do anything. We've looked at last year and the year before, and also what's happened in the last couple of months, and it looks like we have tons of capacity. What we're going to do, anyway, is just allocate twice as much capacity for no reason whatsoever and just run like that, because who cares. It's our infrastructure. We've done a lot of cost auto optimization, so we don't have to worry about if a couple of months or even a week or so we just bite the bullet there and it was always a good idea, but totally unnecessary in almost every situation. That's one team now we dependent on. The reason we were so we were so critical of the organization is because we depended on a lot of services from other teams who did not write such reliable software, who were not as concerned about soft were services that would never be interacted with directed by the customers. So we had a lot of things in our code like automatic retries, acing processing, pre caching, loading from cash like. We would call lots of services and they would fail, and then we would have really complex workfloths to figure out, like predict what the value was going to be in order to utilize it in most cases where we could. I mean, obviously you can't do it in every case, otherwise you wouldn't need the service. However, there was this other team who had problems. Every single week, there was something. It was always firefighting one hundred percent, and because it seemed like they were constantly under pressure, that means there were also constantly solving problems, which gave them a positive perception, and when it came to a huge load, their services would always get pounded into. So the moral of that story is actually two more teams had to be spun up just to support the load of when something went wrong. So a support to teams in a support organization to talk to customers and deal with integrations, because those software issues of having a service that's not reliable have real reaching impacts to customers, and they got rewarded for that, which has just always been not a thing I've looked kindly back in my career over but they would say the most ridiculous things ever where, Like on the surface may sound really intelligent, but if you dig down into it, it's like, so, here's one. We know that there should be high load all the time that we often can't handle, so when there's no load, in order to make up for it, we know that load is going to come later. So whenever we experience a drop in volume, we scale up all of our services as much as we can to prepare for that waterfall that's coming next. And on the surface, it's like, well, that's actually kind of clever. You know, if you expect one hundred requests per second and you don't get any or ten seconds, well where did those thousand requests go? Someone was maybe hoarding them for that period of time. I mean, that's at least the thought. However, you also realize, well, like that's not really a good sustainable scaling strategy. You might as well just figure out how to deal with the burst load when it comes in, you know, figure out use asynchronous software to use services that will actually allow you to support pulling or pushing that load off and so you can actually normalize the curve over time. Don't rely on superstition or tradition to determine what your scaling policy should be. Seems reasonable, you know. That type of scenario makes me think, and I think this is specific to the topic of today, of your end preparation, where you have a lot of different pieces coming together. There can be certain scenarios where it's useful to create a temporary team and pull in people from different disciplines, different teams that are just focused on that temporary problem. You know, like say, okay, we know that we're going to have a massive increase in traffic because of the holiday sales this year, and that's going to impact that's going to be driven by marketing, but also it's going to affect the mobile app team and the back end API team and the infrastructure team bears the brunt of that. So rather than try to manage all of those separately, it's often a good good time to just pull key people from those teams into a temporary team and set their scope. Your scope is to manage this incoming load and make sure that we're successful at handling it from our customer's perspective. And then after that event's passed, the team disbands and goes back to their normal job. Yeah. And I think a product a project team in that regard, in that situation is like the one time where it actually makes sense. And as long as they're not they're not building anything. They're actually trying to prevent problems by using only what's already available. And because there's no long term output really from that process, you don't have to worry about disbanding. And at the end, like that's a really good idea. Yeah, And I think one of the reasons it's successful is because you clearly define the scope. And when I talk about defining the scope, I'm talking about not only saying, hey, this is what you're going to be focused on, but you have full authority to say no to everything. That is not that thing, which is really important. I've seen that a lot throughout my career. We say yes to things almost with like this hidden assumption that there's always more bandwidth available, So we'll do everything that we were doing plus this new thing. And at some point you reach critical mass where you in order to say one more yes, you have to say one more no to something else that you're not going to be doing. I actually put it stronger. Everyone should be at that point already right now, whether you think you are or not. Like if you like, even if you think you have three pre capacity, you probably don't. I know it may feel like that, but realistically, I know as individuals we're very bad at evaluating our own free capacity and be able to do things. And if you actually had free capacity to do extra things, do the thing you're doing more like faster, like why is it just not already getting done? There's no reason to put something else in that spot. So I think realistically, everything you do something next like I would always use this, especially for inexperienced engineers at the beginning of their career. You know, we should solve this, we should fix this problem, we should refactor this code, we should switch to. Now it's Rust, which I can finally get behind. But it used to be you know, have assembly, or we should switch to whatever fancy new version of React that's out there. I'm so happy we don't use React office for so many reasons. There are so many better alternatives. And I'd always say the same thing, like, you know, great, we can absolutely do that. Which thing that you're working on right now? Do you want to stop working on so that you can work on that instead? And really having that priority first conversation, because well, most of the time of the year may actually be practiced, it doesn't matter if they pick up this extra thing. It really helps to think about the priority order. So when huge important events come up, like the end of the year closing or whenever you're end of the year closing in, or if you have a big marketing event, whenever that happens, whether you can know or not, it really is the production version of can I prioritize effectively? Yeah? For sure. So that's a. That's a it's sort of along the same topic here. You talked about, you know, Rust, and a lot of times as we talk about end of year and we're looking at projects for the coming year, and there a lot of times there is a topic, you know, like oh we should we should rewrite this in rust, or we should rewrite this in you know whatever. You know, you can pick whatever. How do you go about making the case for that, Like, if you're campaigning for time and budget for the coming year, how are you going to structure that campaign? I mean I get to make those decisions, So I guess I should. I'll I'll put this maybe in the flip side of like what I'm looking for when someone comes and excellent point to create this whatever that change is. I mean, I don't feel like I go out and hire engineers, Like that's sort of like a skill capability you have. Maybe you're systems thinker, like you're able to understand how doing one thing impacts doing another thing. Yeah, for sure, you can write in some a couple of different languages, But realistically, the teams exist to solve some business problem. That's why they're there. And so looking at the business problem short, medium, and long term is what I care about. So you know, tell me a story that solves short, medium and long term problems. So maybe there's a problem today and it needs to get resolved, and so you know, you want to deliver quick little improvements today, you know, what does that look like to solve that? And then medium you want to do you know, a bigger project to make sure that we never have to worry about that again. And long term, like why are we even discussing this problem? Like this problem is too small scale for us. It should get automatically solved by what we have, by our infrastructure, by a single new feature ticket. And so think about conveying those things when discussing potential ideas of what to work on. So the change to rust, talk to me about error budget, Talk to me about speed of delivery for new features. Talk to me about maybe they're all ability of what we have, the quality of what we have, the ability to deliver new things based on the frameworks that are available, etc. Etc. Right, the actual impact of doing that and nothing comes for free? Right? You know, I want to know the cost of if we make this switch, how long are we going to pay paying the cost? Short term, medium term, long term? And I got to see some benefit there. Now. I don't expect a perfect financially planned strategy for every single part of the process, but some thought has to have gone into it. Right, how many engineers know about this programming languages? How many of them want to know hiring in the market is. A challenge there? Right? Maybe maybe not? Do we have big projects coming up where we already started with something we have and now we need to potentially put that on hold in order to do the switch, or we'd have to write it twice. Right, So they's throwaway work, and an understanding of that tells me that you've thought enough about this problem that I could let you go and run with it, can delegate it to you effectively. Otherwise you're basically saying I want someone else to solve this problem. Right when you come up with a problem and you share with someone else, Hey, we should do this thing, try to figure out what you're doing. Right. Are you complaining about it I don't like X, or are you saying I want you to fix why? Or are you saying I want to do this? This is what it will look like. This is the impact to the team and the organization, and then I'll say great, sounds good, or if it's a bigger impact of the business and some way, it'll probably get into our initiatives for the next quarter and our quarterly planning and they will just show up there and someone who is most relevant to be working on that at that moment may or may not be you. Right, you know, just because you came up with a good idea doesn't mean that it makes sense for your team to work on it. We'll take over and start running with it. Yeah, for me, the starting point of all of those conversations has to be one of two things. How does this increase revenue for the company or how does it decrease operating costs for the company? I think you're maybe being a little bit too unfair there. I mean, I totally agree that should be the actual impact at the end of the day, Like, it has to be one of those two things realistically. I mean there's also social capital as well, right, Like, maybe you're scaling up longer term and hiring additional people is important. So you know, maybe you've got PHP or Ruby you're using and you want to sell it's difficult for us to get engineers. It's difficult for us, you know, two or three years down the line, we're going to have to make this switch at some point. It has nothing to do with the operating cost today or making future making revenue right now, but longer term there's a lot of trade offs. I it needs to be thought about, but like you don't have to come to the table with like this is the number of dollars that we're going to say now maybe, I mean, will your boss apparently, I mean it's much better, right, you know, if you could actually do some sort of analysis, you know, feel free to take that. But it's sort of the thing where like a pull request on architecture, you don't need to come with the code perfect all written for the whole feature. You know, I have some idea. Should I even start thinking about what the next step is here? Or should I forget about it? Like, you know, give me some feedback because this is a good idea to switching to RUSS maybe a good idea. And I will say, could be maybe you know, tell me about it. And then you say, I actually don't know, let me go do some investigation. I'll be like, okay, sounds good. Yeah. Now I'm still stuck on those two I think. You know, for a long time I was as well. The thing that broke me was companies that focus on some sort of social good which is neither necessarily cost reduction or revenue generation. So if there's a tragedy of a common situation where you want to get out from under it, you know, take the climate crisis, right, we have a thing that's neither revenue nor cost. I mean, of course, you can model the the criticalness of our direness of our human society at this moment, it's a huge problem. You could measure the rate of temperature asmospheric increase or oceanic temperature increase as your metric they're going after. So, I mean, whatever it is for your business, right, you know, if you're making money, then for sure, I'll say profit, not necessarily revenue. Both of them are bad metrics for different reasons, depending on how you're abusing them, right, And you're right startups care about revenue, not profit, and real companies care about profit and not revenue. But you could just be making a lot of money a return on investment that really isn't worth it. So you know, what is the important thing for your company? I totally agree with. You, will, yeah, for sure. And that's probably something that happens in my mind that I don't express enough is whenever I talk about currency, it's not always dollars. It's the product itself. Like if you're an open source company, you don't sell your software, but you still have a currency. The currency is the number of people who download and use your software and build on top of it. So when you talk about increasing when I talk about increasing revenue, it may not be US dollars or euros, it's whatever currency measures the success of your product. So Solona or Ethereum. Yeah, or Polygon, you have your you have your own toke gun we do yeah, Matic okay, yeah, yeah. So yeah, that one, that's that's that's the most. Important currency to to actually own, so to measure everything in madic and then you'll know whether your idea is a good one. But that's that's a really good point because as we're an open source company. Yeah, and so for us, the currency is getting people to build on the Polygon network. Matic. The token itself is actually a byproduct of that. That's just a an exchange ticket that all of these people who build on top of our network use used for transactions. The success of Polygon itself is building a network that enables people to do that. So for us, revenue is the number of people adopting the Polygon network as their blockchain framework or blockchain. I mean, I mean it is money in a way. Each transaction that happens is you're getting the revenue out of that that the number of transactions can be your revenue, right. Right, Yeah, And the actual ematic monetary mattic associated with that goes to the validators who Polygon is some of those right now, but mostly is other people. So we're we're even only partially affected by that, with the longer term goal of not having anything to do with that. I mean, arguably it's not even necessarily relevant for you. I mean, that's happening outside. You're a contributor in some way, and so whatever that is is more important than what's necessarily happening in the network. I mean, well, the network is super critical and important for success, it may not be something you can directly affect, right. Cool, So all right, closing thoughts on your end. You know, I never met an engineering team where if I said, like, you know, today, you've got to do everything differently because it's a particular set of numbers on the calendar, and then actually went and made some fundamental changes in their process. I don't think it really works that way. Unfortunately, like you it's not like you change how you're doing code reviews. You're not all of a sudden more secure. You don't automatically change what you're doing, or maybe you add extra reviewers. You don't change the values of your team, the identity, or your processes just like that. I think preparing for this is a cultural chef. It's what happened throughout the whole year that got you, that makes you prepared for situations like this. As well said testing your backups, other out of band processes that you want to validate, or potentially building specific teams that have an identity that matches the execution you want. But yeah, telling software developers to you know, don't commit as much code it doesn't doesn't whole lot of water. Yeah, I would agree with that. I think for me, the big takeaway is it's a good time of year to pause, think about it, figure out how it applies to your situation, and then take action away from that may have an impact, it may not have an impact, but I think it's worth taking a moment to just stop and think about it because we get, you know, so focused in the day to day, like today, I'm focused on these tickets and here's the thing, And so I think the big takeawy for me is just. The call out. Today's ticket is to take a step back and look at things from an entire organizational perspective. I mean, I think that's a really good point that is lost. I was reading a paper about improving based on training and the tickets. The standard work is sort of your production mode. You're doing the work, you're delivering it. In order to become better or even sustain yourself, you need a practice mode. You need to go where there's it's safer to learn new things even and when do you actually deliberately go and do that. If you don't, then you're going to just you're burning your ability to deliver effectively. And could be a good reminder that at the end of the year it's time to start doing some things deliberately that's different than your standard strategy. Actually thinking about how you do that work or what it is, and even a simple thing like validating some of your processes huge benefit. Yeah, for sure, even if you're Yeah, maybe especially if you're junior in your career. It's a good training exercise because as you advance in your career, these things are going to be more relevant to your role. So it's a good chance to think about that and then have that conversation with your manager or your boss and start gaining some perspective that way to help yourself. Later on in your career. Yeah, for sure. Cool. Should we do some picks? Yeah? All right? Which good? So I've been playing a lot of one particular game since the summer Steams sale. I bought a couple, and I mean, I am I just like keeping old apps on my phone. I'm playing games from years ago because they're cheaper, and I don't mind playing out of date content. It's like thirty more years and I'll probably play Breath to the Wild. Okay, So my my pick this time is actually a game called frost Punk. It's the best way I can describe it is it's a rts against the cold weather climate, and you, instead of trying to defeat an alien race or some other civilizations that are out to attack you, you need to deal with the temperature rapidly decreasing in a ridiculous scenario, and so as a resource collection and et cetera. Except but you know, civilization building, it's not not that you like you link get to like six hundred people or something so good number And it's not technically an RTS, but I think RTS against the weather as how I'll categorize it right on cool, it's good, it's really stressful. I'll say, like I I will go at like actually during the day because I played it into the night, you know, And I'm not signed kind of person that gets stressed over things. But if you ever, like worked, and you said something that you felt like was stupid to your manager, and or you didn't you didn't think it was stupid, but then your manager said something it made you feel like what you said was stupid, and then you think about it all the way into the night and while you're sleeping on it. Like, that's what happens to me with this game. Gotcha, I am. I don't hardly play games anymore, and I do miss it because I used to love playing games. But the few times I've gone back to play games, like, my skills are so rusty. I just suck so bad that it's not even fun. The exception to that, I'll have to actually make this my pick. Did you ever play the half Life series? I did not. I loved the half Life series. That was just such a great game for me, and I've got the Oculus VR headset, and so there's a new relatively new this game is several years old. Now. There's a new addition to. The series called Alex that's in VR, and the game mechanics of it are just so well done. When you put on the headset, whenever you pick something up, you know, you get the little vibration slap in there, and it's very intuitive, you know, you put the headset on and within like thirty seconds top, you're completely immersed in this world, having tuned out the real world that it's just that well done. So, wow, that was a cool game. So no Half Life three, but there's a subset for VR. I mean, that's gotta be the biggest joke of them all. Yeah, yeah, no half Life three, but there's four add on expansions for Half Life two and the side quest half Life Alex. Wow, that's that's a very ridiculous. You know. I used to play all sorts of games, many different kinds, and I never really figured out what time of gamer I was, And over time I realized there are certain things I just don't want to feel like work, and so like a lot of games I realized just became work like like RPGs that are the JRPG style just felt like like boss grinding, just so much work. Lots of games just ended become like resource management games, just like a lot of extra Like I just spent all day thinking about architectures and software and then I go home and I'm playing basically a game where I have to do something. It's just a huge nightmare. And so like trying to really narrow down the types of games I play. I did play fps IS for quite a long period of my life. Though. Yeah, I hear what you're saying on that. I'm the same way, Like I don't want to be challenged in the game. I'm not here to level up. I'm looking for more of like an interactive movie. No interesting, that's my gaming style. I want to be entertained. If there's no plot to the game, that's like immediately I don't know why I'm playing this as well be playing with a fidget toy. So that's for sure number one requirement. The second one is it's gotta like I almost don't like combat, honestly, like it just it feels so as like an extra chore. Puzzles, I like, I like puzzles in a game, so like those two things super important for me. But most of the other things, like if I got to do the same thing over and over again, I'm just it's not for me. So it sort of eliminates a lot of things, like I don't know how people do the rogue like games like The Diabolos, which you just go into the dungeon and murder a bunch of things over and over again. For hours time Slaughterfest. I mean, I've done it. You know, it's sort of interesting the first couple of times, but after that, it's like I've seen the story, I played the game, is there? I mean, maybe this is a second ending somewhere. You know, it's got to be something at the end that you get. I used to be achievements driven, but after I realized, yeah, it's pretty much just extra work. I don't get anything for it. Then. I mean, if a game pulled me, if I get through all the achieved, they would give me like like a tenth of my money back. I would totally do that. I would play the game longer. I would definitely play all the game through for sure. Yeah, would you do it? For like social validation achievements like you get this, you get this special icon on LinkedIn or whatever. I did the Xbox achievements way back, a long time ago. I did. I tried to do all of them. I used to be a completionist, and the game that broke me was Assassin's Creed. I absolutely love the game. I know everyone's going to telp me. If you've ever played, oh, two and three or three and four, whatever, you know, the ones with scot artillery are so much better than then what was his name, Desmond playing al Tayer, But I'll say that much better. But there were a thousand of these flags in the game. We had to go around and find all of them, and there's no indication of where they are or which ones you've gone. So even if you know where all the flags are, you have to go to every single location in this giant open world in order to actually find them. And I just like, after the third time of going through literally the whole game, in every single place where every flag is, and I still couldn't find some, I'm just like, that's it. I'm done. And I realized, like, who is looking at these achievements like they're a sort of for me, like, I don't have like tons of friends who are like, oh wow, Warren has all the achievements for all of these games like social validation. I mean, what is that so. Supermodel sliding into your DMS? Hey, I saw your achievement there. I mean, if that happened, I think I'd be a different person today. But so I guess it's safe to say that never happened. I mean I don't know, you know, I guess I wish I had lived your life will where you know, that was something to look forward too. That keeps the voices in my head entertained. So that's what's important, right, Well, there are all there are all all the voices. In your head. Sure, yeah, go with that, all right, cool, Warren, thank you Free Time. It's great with you today, and thank you for listening to this episode. And we'll be back next week. I don't actually know what next. Week's topic is. I have no idea, but uh, you will all be surprised. Yeah, all right, thanks everyone, We'll see y'all.