speaker-0 (00:07) Welcome back to Adventures in DevOps. Every episode is a deep dive into a specific topic with an expert guest. And I'm hoping that this is finally the year of IPv6 for the desktop. I mean, for our cloud providers. The expert we brought in a veteran in the cloud infrastructure space, previous senior staff engineer at EF and now CTO at MoVEXA, Don Bolaghe. Welcome to the show. speaker-1 (00:29) Thank you. Thank you for having me on. speaker-0 (00:30) You know, I think it's been almost two years that I've been trying to convince you to finally show up now. And I think maybe that it's ⁓ a little bit of a serendipitous that at this moment, the cloud providers have chosen that IPv6 may finally be a thing that we can start using. speaker-1 (00:46) don't know about Google Cloud and Azure, but definitely on AWS. It's finally happening after I made the life of our account manager slightly more difficult than it should have been. speaker-0 (00:58) I feel like if you're signing up as the account manager ⁓ coming on board for a cloud provider, you've got to be open to the possibility that your customers are going to ask for things that you're just not prepared to have available. speaker-1 (01:10) Yeah, the answer on AWS that I've always gotten was, well, if you want to have better routing and a less complex network, just take VPC lattice. speaker-0 (01:20) So why is IPv6 even important in the cloud? Isn't everything abstracted away in such a fashion that it doesn't really matter what the underlying infrastructure is? Why go through so much pain and suffering trying to convince technical account managers, the cloud providers to really start offering support for it? speaker-1 (01:36) Well, if you've ever seen the traditional diagram that AWS recommends and everybody also uses for their networking setup with public private subnets, taking on IP spaces, pairing them sometimes with on-prem and having to deal with IP collisions, you just don't have any of that with IPv6. You get a clean network. You don't need to do public private subnets. You can, but... You don't really need to. You don't need to pay for the biggest scam on AWS, which is NAT gateways, because there's no NAT. And you want to peer different accounts, sure. You just share the IPv6, SIDR, prefix, put it in your VPC group and congratulations, you are peered. No need to care about collisions of IPs or anything. And it's just the amount of... difficulty that IPv4 brings along with it. It's just ridiculous. At AWS re.invent 2023, I went to this talk and quick side note, this was a level 500 talk. These are the best talks at re.invent and they're typically empty for whatever reason, because this is the ones where you are not gonna sit into a marketing presentation for an hour. This is where the actual technical bits are. Anyways, this talk was from Netflix. who internally has gotten fully IPv6 on their Kubernetes cluster. And it took them a couple of years, but they showed the diagram of their networking setup before and after. And it looks like some piece of satire of how much simpler it was. speaker-0 (03:19) So in the current world, ⁓ it sounds like just move to it, but I feel like... even all of the services in AWS didn't support IPv6 for such a long time. And I don't know what the current status is of the service. feel like there are some limitations that just were always getting in the way of actually migrating. If you have all of your stuff running purely on say virtual machines, you may be afforded some additional flexibility in choosing your network topology. But if you're using some of the more complex or first-class services that support serverless or Fargate, et cetera, I feel like there was always some particular issue you'd run into where it, should just tell you that it's not supported to utilize those services with IPv6, even though there are some fundamental limitations with the IPv4 infrastructure. speaker-1 (04:06) Well, it took them long enough to support single stack IPv6 on just VPC and EC2 to begin with. If I remember correctly, this is a thing from the last two years where they made this happen. Before that, it was dual stack and you could attach IPv6 to it and you could do that for... And every year, I used to run a platform team and every year I would go out and just... try to set up a core piece of our infrastructure with an IPv6 native infrastructure. And every time we would run into like these fun quirks where like, yes, this should work, but only once in a blue moon. And if you sacrifice your firstborn and then you go complain to AWS support and They say, we don't know what's going on. And then you end up talking to a PO and the PO is also like, yeah, let me, let's get back to you and see what's going on there. Because they are also all puzzled and all I'm doing is trying to follow the guide I was given. Now this has become significantly better over the past few years. For example, my company, Mevexa, we are running fully IPv6 native everything on Fargate, on ECS. with RDS. However, RDS specifically requires a dual stack setup. So I've got one VLAN specifically for the database, but everything else is running in a VLAN. Sorry, subnet. For what specifically? speaker-0 (05:35) What's the reasoning for that? For RDS needing dual stack setup and not just being able to rely on pure IPv6 as the network configuration. And this is the thing is that AWS had a release, I think it was in the last year, saying that RDS fully supports IPv6 only as an option, but then you go and you look at the configuration that can be there and when you actually try to select a single stack mode, it... speaker-1 (05:45) I wish I knew. speaker-0 (06:00) If it works, end up with an issue either pulling images or whatnot, or more likely is it just doesn't even work out of the box. speaker-1 (06:08) Well, last I checked that option didn't even exist. IPv4 single stack or dual stack? speaker-0 (06:11) Amazing. And that's something I sort of want to get back to on IPv6, is it seems like it helps from a technical standpoint, but I'm curious if you have any opinions on how or if there's a customer perspective. Like, didn't Mavexa go with the direction of having IPv6-only infrastructure? Was it purely technical or are there other considerations? speaker-1 (06:35) Well, there are several technical considerations. One of them is, of course, being much easier to set up and also much cheaper because I don't have to pay for an app gateways and IPv4 addresses, which, by the way, I want to thank AWS for. This is like the best charge they ever introduced because now people are are starting to care ⁓ because they have to pay for it. speaker-0 (07:00) I see. Maybe that gateway has been like a very long play, right, from decades ago to convince the world to move to IBV6 because without that, maybe the NAT gateway charge, you know, if it didn't exist, they would be, they would have lost a critical tool in actually convincing people. speaker-1 (07:19) Well, I've spoken to many people on this subject and the feeling I get is that no one really cares. They are just paying this because, that's what it costs, right? But then people suddenly did start caring when in 2024, AWS started charging for IPv4 addresses, even if they were in use, because previously it didn't matter. You would only be paying for the ones that were sitting around. And at the time at the company I was working for, ⁓ I did see a jump in our bill the moment they started charging and that was quite noticeable despite our spending, which was quite significant. speaker-0 (08:02) So the IPv4 switch to charged IP addresses, like that was actually a motivating factor. speaker-1 (08:08) Well, my previous company, wasn't because there were more important things to do and we're just going to absorb the pain. But if you're starting green, just fresh, I would totally see this as a motivating factor to at least explore IPv6 and finally adopt the technology from 9095. speaker-0 (08:28) I see. I noticed that one of the concerns that we had initially with ⁓ especially VPC architecture with IPv4 was ⁓ being able to spin up quickly ENIs, which used to be a huge pain for serverless architecture. I don't know enough about the network to stack here, but I have to imagine that AWS must have made some improvements over... ⁓ what was in place historically for the network architecture for IPv4 only to circumvent the limitations that were in place. speaker-1 (09:01) Probably. ⁓ I did have to deal once with ⁓ our VPC being exhausted with IPs because we were spending up too many serverless functions in one go. You get traffic and you can only assign 200 IPs in your subnet. speaker-0 (09:11) How does that happen? ⁓ There must wait, is some not really limited to only 200 IP addresses? speaker-1 (09:22) Well, it's 255, ⁓ 253 really, cost of your gateway and your net mask. speaker-0 (09:28) Okay, so if you're running a serverless function inside the VPC, of course it's going to be in one or two subnets. You're going to be limited by 250, less than a thousand concurrent invocations. speaker-1 (09:42) Well, of course, it depends on how you set up your network. In our particular case, we had a slash 24 block for this specific network. Of course, you should probably be taking something like a slash 22 block, which gives you quite a few more IP addresses. speaker-0 (09:54) I can imagine with an architecture that's not serverless, if you're using Kubernetes or even some sort of swarm setup, poorly defining, like not per, I would say not even poorly, not perfectly defining your usage of the IP address space would lead to exhaustion. Like that problem just completely goes away by switching to IPv6. speaker-1 (10:10) Yes. Yes, because you get more IP address than we have known stars in the universe or sent sent corns on this planet multiple times. speaker-0 (10:25) How does this work realistically in practice? Are individual nodes each getting assigned an IPv6 network address from AWS like global proper, is there some sort of, I don't know, magical network appliance that's, I mean, there's no network address translation that's happening. Like there is no like LAN that's in existence for the IPv6 infrastructure. So I'm not actually familiar with what must be happening here in order to provide the level of sort of performance or growth from getting requests in. speaker-1 (10:57) So let me quickly break down IPv6. A IPv6 address is 128 bits. The first 46 bits of that is a writable address, also known as the prefix. And the last bit is the identifier, the last 46 bits. In the original RFC, which was RFC 1886, I believe, 1883, it was quite simple. You had your first 46 bits as your gateway address and all the devices below that would use the order 64 bits to basically translate their MAC address. So the first 64 bits would be what in IPv4 we would refer to as your external WAN address and the last 64 bits would be known as the internal IP. Of course, this is IPv6. Everything is globally routable. And there were some nightmares about privacy there. So... in 2017 if I'm not mistaken. They introduced the privacy extension where the last 64 bits are no longer based on your MAC address. Before you could already do that ⁓ using static routing or DHCPv6 which is also not needed. speaker-0 (12:14) Couldn't you spoof your MAC address too? speaker-1 (12:17) You can, of course. Most mobile phones already do that out of the box. And Mac OS as well, I don't know about Windows or Linux out of the box. Doesn't if you use networking manager. But that wouldn't necessarily ⁓ relieve your problem because you'd still have a static address effectively. How it works nowadays is you just get a random identifier when you request an IP and... your gateway will know how to write this and at any given time depending on your hardware for example this computer i'm recording this on currently has about 12 ipv6 addresses assigned to it is that a lot? that's quite a lot but also i didn't do anything to change this this is out of the box configuration speaker-0 (12:56) that will. How is the assignment happening? is it per, I don't know, network process or by process that's actually requesting these? speaker-1 (13:10) Per process? Well, I've got a little order bit to go into after this, but no, it's your networking interface. Funny that you should say per process. ⁓ I was at ⁓ FOSDEM and there was this interesting talk from this older gentleman who had the idea of the internet of threats, where you would assign each threat on your operating system an IPv6 address. And because of how mind-bogglingly large the IPv6 space is, you can... totally do this without any concerns. It did raise a couple of questions because what I hear from people that don't know about IPv6 or typically have excuses to not look into it and these misconceptions is that, no, IPv6 is a privacy nightmare. But realistically, the only, these first 64 bits are the important bit, ⁓ which is the same as your IPv4 public LAN address essentially. And if everything Every thread gets its own IP address. That problem is just completely gone. Also, generally, the problem is completely gone with the privacy extension. But with the Internet of Threads, every single browser tab could have its own IP address, which is quite exciting to think about. I don't think this is going to happen. speaker-0 (14:30) I was going say, I think you and I have different definitions of exciting. But I get the benefit of not having to reuse connection pool in some way or uniquely identify. I mean, I can also imagine some of the challenges that come along with providing that, from a, I'll say, if you are trying to monitor or validate traffic that's coming, the fact that each one of these things has its own unique IP address could potentially be problematic. speaker-1 (14:54) It could be. I spoke to somebody from Fastmail. They launched ⁓ this back in the day, in 2012. And there's an article from their CEO on their blog. And he talks about how they had enabled it and everything was amazing. And then there is a blog post slightly later, like, disabled it again, with no specific reasoning behind that. But after talking to a bunch of email people at Fostem, it really seems like One of the big problems here is spam filtering because most companies still rely on IP reputation lists. And because IPv6 is such a mind-bogglingly large address space, these lists are easy to circumvent because I can just change the last bits of my prefix, sorry, the last bits of my writable address and keep the prefix the same. So I essentially have a 64-bit number space to play around with. Realistically, you should just be set those prefixes on to, sorry, the IP reputation lists on the prefix, really. But anyway, those lists don't really exist and email providers, unfortunately, rely on this, which is why hosting your own email server is quite the challenge. speaker-0 (16:14) Email providers are behind on technology that doesn't surprise me in one bit. do like that ingress and egress blocking, has historically been based at the IP address level, ⁓ basically feels dead with the migration to IPv6. The wonder why I like that is because all of these companies with security parameters claiming to do security and I have that in quotes. like actually now have to join us in the 21st century and shift to what everyone else has been doing, which is some form of zero trust. speaker-1 (16:46) Yes, and actually do some security rather than security to obscurity. So one of the common myths about IPv6 is that it doesn't use NAT and NAT is a security feature. yes. And this is very easy to break down quite quickly because NAT itself, yes, it translates your IP address and you can't go to ⁓ a computer directly. speaker-0 (17:02) amazing. speaker-1 (17:16) behind the NAT. And this is true. But most NATs come with a built-in stateful firewall. And that's the magic bit. They're stateful firewalls, the same stateful firewall that you should be using with IPv6. If you go disable the stateful firewall in your NAT, your NAT is not going to be as secure anymore. Because the moment you go visit a host or run any program, they essentially can just directly talk to the end machine. And you want that firewall to be there. Now, and this gets even worse because IPv4 is such a limited address space. You can, and there's plenty of people who do and plenty of companies who do, scan the entire range and port scan everything and just do like knocking on the doors of NAT and do some NAT hole punching. Now with IPv6, it's actually more secure because you can't feasibly do that. It's just not realistic to scan a 128 bit number. speaker-0 (18:13) today, right? speaker-1 (18:14) Well, I don't think ever. speaker-0 (18:17) This is a question of whether there's some sort of quantum computing process which would help in this way, but I think given that you have to explicitly enumerate them, I don't think any sort of quantum annealing would even help expose relevant information. I think I won't spoil my pick here, but part of it I think was the ability to potentially use the IP address space and the ICMP packet to make a hard drive. speaker-1 (18:44) I want to know more about that one speaker-0 (18:46) So I'll say more about that at the end of the episode. ⁓ so the thing here is, and maybe it's worth spelling out exactly what Nat is doing. So in a home consumer network, like I have a router that's attached to the ISP where it gets an IP address. And then I have all my devices connected to the network, which can see out through the internet, but requests that come back from the internet don't understand about my specific machine. And the router is routing those packets. to the individual machine. is a little bit different, right? Where it's sort of rewriting the packet and faking what's in the network. And I think it is a security feature where if everyone was using a cloud provider, cloud providers prevent sort of hijacking and making fake NATs and lying about the IP address. But you can attach any, your own version of a NAT to the internet and lie about your IP address all day long. And it will just go, now ISPs are supposed to be blocking that. But there's no guarantee that they're doing it, especially if those packets are coming from some ⁓ renegade Wild West place ⁓ in the world. And so I think it's always been a lie that IP addresses ⁓ provide any sort of security, that Mac tracking provides any sort of security feature, and that, as you pointed out, that NAT is using NAT as a replacement for security with the built-in firewall. speaker-1 (20:03) Yes, notably that built-in firewall. ⁓ All NAT really is doing, as you said, is rewriting packets, and that's it. And ⁓ I would like to mention mostly TCP packets, because a UDP cannot do NAT. If you want to do UDP, you need to do NAT hole punching and just establish an end-to-end connection. This brings me to another fun feature of IPv6. Well, not necessarily a feature, but a big benefit. speaker-0 (20:24) interesting. speaker-1 (20:32) Because UDP was designed with an end-to-end connection in mind. If you want to have a UDP connection be reliable, you're gonna need to do NAT hole punching. And for NAT hole punching, you're gonna have to run a state machine. And these state machines can get quite complex, especially if there is this other thing, terrible hack that is known as a carrier-grade NAT, where you have your own NAT at home, but then your ISP has its own NAT as well. So you're sharing the same external IP address with 500 order customers of your ISP. This is quite commonly seen in mobile networks, but also thanks to the exhaustion of IPv4, a lot more in residential connections these days. speaker-0 (21:17) There's a common aspect called multiplexing when you have to send data along an internet pipe. There's different ways of breaking up giving the bandwidth to individual subscribers. You can break it up by time delays where the first millisecond of the bandwidth is for the first customer and the second millisecond of the bandwidth is for the second customer. Then there's frequency multiplexing where you send multiple different digital packets along, they're analog, along the network cable at different frequencies and different customers are different frequency. You know, this is an interesting standpoint of like how do you expose this data to the outside world when it's analog? You sort of have to make these determinations yourself It sounds like what you're saying actually is we've been trusting ISPs to provide us a level of security with IPv4s And I don't remember the last time I'm like, you know my ISP. I absolutely love it speaker-1 (22:05) Well, I'm fortunate enough that I can say that my ISP in it 7 here in Switzerland is probably the best ISP in the world. I love you guys. But yeah, on the average ISP, I definitely wouldn't. I want to quickly go back to UDP. For UDP to work again and to add connection and IPv4 does not provide that. You need to have this. You need to have something in between like NAT and NAT rewrite your packages, but UDP packages can't really be rewritten. So. At the end, you need to have the state machine ready. this is problematic because this is all complexity. It's not very reliable. And if you want to, for example, have a conversation like this, recording this podcast, we're in two different spaces and these tools often under the hood use UDP for streaming. And right now, if this were to go over IPv4, there would be a complex state machine in between doing that hole punching and keeping track of what's going on. In IPv6, you just have a direct connection. You don't need any of that. You just have a straight line from A to B. I used to play World of Warcraft for a very long time. World of Warcraft, since it's very early days, had this beautiful toggle in its settings menu that would enable the preference for IPv6 over 4. And if you would enable this, you would actually get a more reliable connection because you would get an end-to-end connection. There's another one that I would like to highlight, which HTTP 3, the new standard of HTTP that the internet is slowly moving towards. HTTP 3 is not going over TCP. It runs on this new not quiet layer 4 thing. It's more like layer 4.5. It's called QUIC. Yeah, it's QUIC, which is built on top of IPV, sorry, on top of UDP. speaker-0 (23:51) quick yeah I guess the problem is UDP doesn't benefit from the net packet rewriting, so it's less reliable over IPv4, right? Is that where this is going? So how does that actually work then? speaker-1 (24:08) Yes, exactly. you just get an end-to-end connection for A to B because all your devices have a globally routable address. speaker-0 (24:17) Okay, so TCP is the one that has the handshake. You need to verify every packet that comes back or the server connection has to basically resend it to verify. UDP is fire and forget, but in a way, that's why it's used for streaming stuff. However, I will share that because of the world of streaming gets more complex every day, the way that recording actually works for the podcast, maybe this is interesting to someone, is ⁓ my stream gets recorded locally and so does yours. and that's what we actually stitched together to make the episode. But while we're recording the episode, we care about lossy real-time packets rather than the lossless recording. So you can be talking, and I may not even know what you're saying because the stream is being dropped, but when we stitch the episode together at the end, we have the full context of what everyone was saying. And what's happening in the browser is we don't have a direct connection because I'm on a private behind-the-net connection and you're on a private network behind, you know... that or you're using IPv6 and I'm not connected with that, but it's using WebRTC or WebSockets and it would be using WebSockets, which would be better, but the problem is that WebSockets have historically been very slow and the problem with WebRTC is that most of the time you're blocked from the direct connection. So I'm wondering, first of all, where is QUIC going and does this help solve the real time streaming from, you know, person to person? speaker-1 (25:36) Well, I love that you bring up WebRTC and WebSockets. In HTTP 3, there's this new feature that is slowly being rolled out. Right now, only Chromium-based browsers support this. But thanks to Interop 2026, this is also happening in Firefox and Safari now, it's a feature called WebTransport, which is like basically the same idea as WebSockets, in that you can have a bi-directional connection between the server and the client. Except WebSockets come with some ⁓ limitations. speaker-0 (26:14) But it's per job. It's like a per domain or something, right? speaker-1 (26:17) It's 8 per domain, yeah, something like that. Or 8 per browser tab, I'm not sure. speaker-0 (26:22) I think I remember it was domain because if you have the same tab open nine times, then if you have a WebSocket implementation, it starts to fail because you have too many tabs open, which is just amazing. speaker-1 (26:35) beautiful. Okay, so it's nine per domain then, eight per domain, which is even worse. But WebTransport solves this problem because the way that QUIC has been designed is basically UDP streams, ⁓ multiplexed UDP streams, where you just open one connection and WebTransport utilizes that very same connection. So effectively what you can do is you can open a thousand WebTransport ⁓ sockets and you can just chats with whatever with the server. And it's faster and also quite a bit more reliable because of HTTP3's design. And I'm super excited for this to finally launch and so I can use it. I really don't like web sockets. speaker-0 (27:23) It sounds like... You know, we've had ideas on how to make the connections of really the internet more efficient and more performant for a long time, legacy companies have been getting in the way. And I feel like we're a bit beholden to the cloud providers now, which are quote unquote said to be running most of the internet. ⁓ At least it seems that way when one of them is having an incident, it's very clear that there's a problem are now responsible for doing this. It's like another one is the ⁓ the new verb for query that I've been waiting for for decades, basically. ⁓ just to throw this out there, there's the rest verbs, right? The ones that the browsers can use, get, post, patch, delete, you know, whatever. And what I was really missing was the ability to send a payload and have it be cacheable. And the only ones that accept payloads aren't cacheable. You can't catch a post request, right? It's meant to be represented of a new request every single time. But if you have a search query, right, you're looking for some items from an e-commerce store and whatnot. It can be incredibly complex to stick the whole query in the query parameters of a URL because you don't get a body sent with a get. Now, technically you can send a body with a get request, but all the cloud providers, they vomit when you try to do this. And the solution is to add a query. And I feel like we're beholden to them to actually support this new verb that comes out, just like we're beholden to having them actually support IPv6 in order to start utilizing it and therefore the improvements with the other technologies. speaker-1 (28:50) Yes, I totally agree with that. I don't know. I'm going to paraphrase this because I can't remember the exact quote, but Mr. Fint-Serv, one of the creators of TCP-IP, said at some point that IPv4 was just like a proof of concept, a thing that should not have been widely adopted and that TCP-IPv6 was going to be the real thing that should have been adopted. Now, back then they also had an IPv5 which essentially became ⁓ voice over IP and they had experience with IPv7, 8 and 9 but IPv6 is the one that got stuck. IPv4 should never have been the thing that got deployed. speaker-0 (29:29) This is true of a bunch of different technologies as well. I think the same thing is said for JavaScript in the browser. Like, yeah, I just needed to enable it so that I could do one specific thing and now only, well, then there was a lot of years of the... cross-site requests being executed before we actually got to ⁓ fetch implementation for real API access. But it was also set as a thing that, yeah, probably won't stick. It's not even really that critical. And the implementation was quite naive at the time. speaker-1 (30:01) Yeah, I just want to come back to AWS for a second. Yeah. Specifically at AWS reInvent 2023, I had a brief chat with the person who designed VPC lattice. Now, for those that don't know what VPC lattice is, it's basically an overlay network on AWS where you can do fancy stuff like IAM policies for routing and skip a whole lot of the networking complexity that Really, IPv4 brings and with IPv6, the only benefit of VPC Lattice would be routing policies, because you can have IAM rules for the routing policies rather than IP addresses. speaker-0 (30:41) It sounds like someone over engineered their network stack and a large enterprise said, you know what, we have very specific requirements where we want permission controlled access based off of the actual network packets that are being sent over the network and want to control it with IAM. And I feel like all of those things is something no one should ever want. speaker-1 (31:00) Well, for certain environments, can see this being a thing. that's a very small, small subset of that. I think if you just have a well-defined IPv6 base, that's just going to be so much easier because you don't need any of that complexity. basically, this guy that designed this said that one of the primary use cases was just to stop having to deal with IPv4 complexity. because as I mentioned at the start of this episode, the traditional way people have been architecting at AWS has been suggesting your architect, your VPC setup is you have a public subnet, one for each availability zone. So in most regions you will have three of those. You ⁓ put an app gateway on that and then you create more, ⁓ one more subnet for each. availability zone that you call a private subnet that has routing rules to go through the public subnet to the internet, but that's the only way they can reach the internet. And if you want to attach multiple accounts to that and you want to have routing policies between accounts, for example, you want to have one big database cluster because you have a distributed monolith. But you pretend that you don't have one of those. So you've got your central database account that all of your microservices talk to. that are deployed by individual teams. ⁓ speaker-0 (32:28) I feel like I'm going to have a stroke. No one has what AWS account that they call their production database account. Please tell me that doesn't exist. speaker-1 (32:35) It totally does and I happen to deal with one of those. Anyway, so that's a thing. Well, this historically grew because there used to be an actual monolith. Sure, it was a 15 year old monolith. It was terribly designed because it was built during the area where it was popular to go and outsource to India to the cheapest bidder. And in this particular instance, those cheapest bidders, the cheapest bidder would go even cheaper internally because they were also doing that internally. And at some point this went five layers deep and the code that was written was just absolutely atrocious because I got what you're paying for. This entire project was quite a mess. However, ⁓ over the year it was tried to be salvaged and at what was finally being deprecated or being shut down was actually... Yeah, it was a terrible code base, but you could work in it. ⁓ You could make your changes. This is your typical legacy code base. There were decisions that were questionable. anyway, the idea was to move out of this. And that was supposed to be a two year migration. That took nine years. speaker-0 (33:46) So right on schedule. speaker-1 (33:48) Exactly, right on schedule. Very much written budget. And this migration went to somewhat microservices, but there were so many dependencies on this particular database still that there was quite a few apps still talking to it. But they were all ⁓ running in separate AWS accounts because, well, you want to do microservices now, so each team is going to get one of those. Well, first of all, there was an Excel sheet with IPv4 ranges that were being used because this was also being paired to the to the no longer active ⁓ on-prem beta center. The IP address had never been released. speaker-0 (34:23) So the missing part here that's also important, I think, is that if you want to peer between multiple AWS accounts, they need to have non-overlapping IPv4 ranges, which I don't actually understand why, because when you're connecting them either direct peer or over ⁓ transit gateway, which is like a multi-account peering strategy for IPv4, it could do the translation itself, but it doesn't. ⁓ It requires all of the VPC subnet ⁓ cedar blocks to be mutually exclusive. speaker-1 (34:53) Yes, however, this has been fixed by VPC endpoints ⁓ where it will do some fancy translation and this is very useful, however... costs a lot more money, Which again, IPv6 saves your money. Where was I at? ⁓ speaker-0 (35:01) Just pay more money. Problem solved. I cannot tell you now. speaker-1 (35:13) That doesn't gonna be cut out, just rants. ⁓ speaker-0 (35:16) man, ⁓ speaker-1 (35:18) ⁓ Right, ⁓ this shared database, everybody has been moving to their native-based account, been writing their microservices, but there was still this dependency on the central database. Now, there were multiple ways that this was being solved. One of them was being a massive Kafka cluster that everybody would subscribe to. However, this was using AWS managed Kafka, ⁓ MSK, which was IPv4 only, so you would have largely half the same problem again. Now, that was being... problem was being solved by throwing a graphql on top. Actually no, so what they did was they had all of the topics right into a postgres database and that postgres database had graphql on top. And this was a bespoke layer that was written on top of it because it also had access controls that were built in house to just deal with this problem. While looking at this from the outside you'd go what the fuck. And this was actually a quite reasonable solution to the constraints. Just don't think of that GraphQL like an API. wasn't. It was just stored procedures as a service. ⁓ speaker-0 (36:26) There is something that comes up a lot of times here, because I've seen this too. I was working at an aerospace company and there was one team that owned the data. They weren't even the DBAs, it was just one team that were tasked with the ownership of the critical manufacturing data for actually putting together the vehicles. There were other applications owned by other teams that would need access to that fundamental data. The solution was, of course, that the team that owns the data would create a REST interface and expose the data to, who are we kidding? No, that wasn't the solution. The solution was one engineer got access to the database that was outside of the team that owned it and they created a REST API and that REST API had one endpoint called post which literally took in a SQL and just ran it against this database to return the result for whatever anyone wanted. speaker-1 (37:22) Did it at least use basic ouf for the ⁓ database user login? speaker-0 (37:27) ⁓ you know, I don't think I don't think I'm legally allowed to share and if I do the penalty is like like up to 10 million dollars in 10 years in prison or something if I if I do speaker-1 (37:38) yeah, I won't pressure any further on that story. ⁓ speaker-0 (37:41) what it was called though it was called the warp transfer function speaker-1 (37:45) I know what company you're talking about, so that makes sense. speaker-0 (37:47) The other really clever thing you could do instead of exposing the data, and I've seen this done multiple times actually, is that you try to implement a real-time database replication, like read and write across every AWS account at the same time so that everyone can just integrate with a database that's stored in their account. And then the magic part is somehow all these databases are kept in sync, which is actually just not a solvable problem. But I've seen it tried. Yeah. speaker-1 (38:16) Beautiful. ⁓ speaker-0 (38:17) Is the migration over at this point? Did they ever finish? Is it going in the right direction? Is the database going to be removed or is that database going to be turned into its own first class service? speaker-1 (38:30) Within my last year, I am very happy to say this database has been shut down and the migration was considered finished. Now the database is called Salesforce. speaker-0 (38:37) or if that pat on the back that we completed our migration to Salesforce. speaker-1 (38:42) I don't know which one is worse, honestly. Well, because at least your own database, no matter how terrible your routing setup may be, ⁓ at least allows you to query it and get data from it. Salesforce, the moment you start developing outside of it, it becomes very difficult to do that because you get hit by limits, platform limits is what they're called, that can be quite annoying to deal with and ⁓ just stupid things. Like for example, in Salesforce, And I might be wrong, but this is what I remember. Whoever's listening might want to fact check me on this. ⁓ If you create a table on Salesforce, you can never add more columns or change the column definition. So what you would get is you'd have a well-defined table, but there would always be like 50 more columns. was an additional column underscore one, additional column underscore two, et cetera, be sitting there for additional fields that might arise from future business cases. And if you look at queries where you would be querying like select ID, comma, additional field 43, comma, additional field 2 from table 1 and then join on where additional field 43 from this database equals table 2, additional field 50. It'd be very nice to read. ⁓ speaker-0 (40:14) ridiculous. yes. It doesn't really matter whether or not it can be different. The fact is that they thought that this is what had to be done and this was the right way to go about it. speaker-1 (40:25) yeah, this is absolutely awful. One way to get around this is just to migrate your entire data to a new table and solve this problem so you'd have like, ⁓ customers underscore 54 point old dot back, ⁓ which would be several terabytes large because you're a big boy company and you many customers. speaker-0 (40:31) payable It's one thing to say migrate and I'm totally with you. my, thing is that realistically in my entire engineering career, was always migrations are expensive because it's difficult to write. But like at this point in my career, I believe the app, the opposite is actually true. Migrations are incredibly cheap to write. Like writing a migration and switching to a new table, that part's cheap. Actually getting the data transferred from one physical location to another one. That's the thing is like, Oh, you know, we're trying to copy the data over and it says it's going to take us 31 days and that's. a normal amount of time for this to happen with the fastest hard drives that ever existed. Like, it's just absolutely ridiculous. And yet, you know, that's actually where the problem is. speaker-1 (41:26) Yes, 31 days with the fastest hard drives available means that you have a very good problem probably because you have a lot of customers. speaker-0 (41:34) Or, or, or, ⁓ I one-up you one there, and that's we have so much critical data that ⁓ is very important for the business that we can't afford to delete. I assure you, was in these databases was essentially like ⁓ a bunch of IoT sensors that were collecting information for multiple years. And one of them may have something really valuable that we just can't let go of. speaker-1 (42:02) Of course, of course. That's how it works. It's business critical information. speaker-0 (42:08) Okay, this may be a good opportunity to switch over to. Okay, yeah, so what did you bring for us, Don? speaker-1 (42:10) It's time for pics. So I'm going to bring something that's completely unrelated to ⁓ IPv6. ⁓ But also comes from the bit of there are simpler ways to do things and for some reason they're not being adopted. ⁓ And my pick for today is going to be SolidJS. So I bet my company on SolidJS rather than React, mostly because... speaker-0 (42:31) For summers. speaker-1 (42:42) I've been writing React for six, seven years. I can balance it. Especially when React hooks were introduced, I believe this was 2018, they came with the rules of hooks. And they add quite a lot of mental overhead or cognitive overhead I found in my development cycle. Now, if you write React all day, you might get numb to this. And you might only, every now and then, shoot yourself in the foot, which is why React Direct Team introduced Direct Compiler in React 19 to alleviate this problem. But to me, this is just like there must be something fundamentally wrong with this approach. I agree. And I'm a big fan of JSX and there's other frameworks out there like Vue, like Svelte, like Angular. They're also all great frameworks, but they don't do JSX and I really like JSX. But there's this one other framework out there called SolidJS. And it's been around for many years at this point. And they do two things very different from React. ⁓ One is, while they're in the hooks, they do two things very different from React. One of them is the reactivity model, where in React, ⁓ when you have a component, every time something changes, it will just re-execute your function over and over again. In Solid, that's not the case. It will render. It will execute your component once and your state is managed by signals rather than these magical hooks. Now, the syntax for signals is the exact same as in React, except the thing the output is an array with a function that returns a value and a function to change the value rather than just the value. So if you want to use this, what you do is you go into your return statement And when you want to use the value, you just execute the function rather than just putting it straight in there. And this approach gives you very, very fine grained control over reactivity. And because of that difference, I found SolidJS to be so much more intuitive to write despite it looking exactly and feeling exactly like React. speaker-0 (45:01) I always found that the concept of having to explain what a functional component was, was always messed up to me. And Hooks just added this extra complexity on top. It's one of the reasons why I personally was always a fan of Vue and the transition of Vue 3 also added this sort of, it doesn't matter where you are, it doesn't matter if you're generating ⁓ components that are used in your UI or doing something functional. You can rely on the state management. Just the same, there's no difference there. Everything's, you know. is very similar, irrelevant of the context. And I think that's one of the biggest problems that you mentioned that realistically, if you're using a framework that adds workarounds and wrappers to solve a problem that it created itself, it should tell you like, don't use this thing. And actually, I think this may go into, I think if anyone has their bingo card ready, I'm gonna mention AI for the first time in this episode. If there are examples of doing something out there, then LLMs are more likely to provide you with those examples when you need to solve a problem. But statistically, if those examples are more likely to be wrong than right, then likelihood is you're going to end up with more likely to have wrong code in your production environment. So you actually want it. There's a trade off of switching to frameworks and languages that are more likely to be objectively correct in what they're doing. OK. Yeah. I like your pick. speaker-1 (46:20) ⁓ I would like to highlight as well that Solid.js is so easy to adopt. recently had a developer start on my team. They had never looked at Solid.js before. They have had many years of React experience, but they were creating actual functional pull requests within a day. speaker-0 (46:38) Cool. I like your pick. So I guess I'll share mine. So I mentioned this earlier. I actually switched because of something you said. So there's this YouTube video out there ⁓ where the content creator goes and creates hard drives, basically the idea of a hard drive from different technologies that exist. And it's called Harder Drive. And there'll be a link in the description ⁓ below the episode. And he goes through basically what actually a hard drive is, like conceptually, which is basically storing data. and how can you store data via other mechanisms? And if we think about hard drives, they're not permanent, right? They can degrade over time. The metal can wear. If you're using a old style HDD, then the platter can degrade or can get scraped. So it's not perfect. And in the video, one of the harder drives that he makes is actually using ⁓ ping with ICMP packets. So basically you can actually add some additional data after the header in a ping packet that will come back to you. So you can verify the packet is authenticity. And so you can actually store arbitrary data in there and then send a ping to an IP address. And part of that is sending the packet back. So that time, that dwell time that the packets in the internet basically is stored. Your data is stored in the internet. And obviously there's a failure mode here where the data may not come back if the IP address isn't being used or is down or doesn't respond to ICMP because they're being blocked. in a lot of cloud providers, their IP addresses, don't. your production servers, right, applications, you don't return pings, you may only be open on port 80 for the IP address, basically narrows down, scans the whole range, figures out which IP addresses are actually responding to pings, and then goes to store some data there, and then does some additional calculations. And there's a couple other mechanisms too, for storing data that I find as a good thought experiment, including like how many chainsaws can you juggle in the air? Very clever, I think, as an episode, and very entertaining. Okay, so thank you Don for coming on and chatting with us about IPv6 and other futuristic improvements to the network topology and internet to come in the near future. And thank you. And thanks to all the viewers for checking out this episode and hopefully everyone will be back again next week. speaker-1 (48:47) Thank you very much for having me. ⁓