Sam Newman 谈微服务、分布式系统三定律与 AI 时代的工程韧性

Sam Newman: I've described microservices as being an architecture of last resort.

Sam Newman: Really, yeah. Absolutely.

Sam Newman: What a lot of people get wrong, I think, they say adopt microservices, and they think sort of magic dust going to happen.

Host: You talk about the three rules of distributed systems. What are these three rules?

Sam Newman: The first is that you can't send information instantaneously from point A to point B. The number two is sometimes network calls fail when you talk to things there. And the third rule is that resource pools are not infinite. The vast majority of the time that you have a system outage comes down to that: some resource has been totally saturated somewhere.

Host: What are the things that we know that we do not know?

Sam Newman: I think the tech world in general is quite naive about AI and fundamentally misunderstands it.

Host: Sam Newman is the author who wrote probably the most widely read book on microservices. And yet, as you heard, he said that microservices are an architecture of last resort. But why? Sam was in the room when the term microservices was coined. I find him to be one of the most thoughtful engineers on this topic, as well as on resilient systems. And his latest book, Building Resilient Distributed Systems, is out now.

Today, we cover specs versus code as a source of truth, and why Sam says that production is truth; idempotency keys versus fingerprints, and why you should avoid idempotency keys when possible; the four dimensions of resilience: robustness, rebound, graceful extensibility, and sustained adaptability; cognitive depth and cognitive surrender, and Sam's advice on how to better use AI to avoid these and many more. If you want to understand what it takes to build resilient systems from an expert who has been knee-deep in this topic for 10-plus years, this episode is for you.

Host: This episode was presented by TurboBuffer, the AI retrieval layer built on object storage. It's fast, cheap, and extremely scalable, and Notion and many others are happy customers.

Before we start, let me mention Entire, our season sponsor and a product and team I am bullish on. You probably already know Git hosting has a problem: AI agents. Git is increasingly becoming a bottleneck for modern agentic software development. Engineers like you are creating more code with agents, those agents push more code, and most of us are running parallel agents which push way more code. GitHub is clearly struggling to keep up and racking up outages.

So what's the solution? Entire was founded by GitHub's former CEO, Thomas Dohmke, who, by the way, was a guest on our podcast when he was still at GitHub. Thomas rebuilt Git hosting for the agentic era from scratch. Entire was built to be very fast, and to have your repos regionally close to you to reduce latency. This allows fleets of agents to push in parallel, and Entire's performance is next level. Their repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market. When GitHub is down, you can still keep working, and you don't even need to migrate away from GitHub; you just sign into Entire and the platform mirrors your repo.

And here's another neat thing: have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself. Oftentimes, Entire captures all the prompt history with your agent right inside of the repo, easy to check back on, and it's got a pretty innovative UI to show all of this. If you're looking for Git hosting that works even when GitHub is down, head to entire.io/pragmatic, install the CLI, and mirror a repo with a click. I've already done it, and I love it. Oh, and did I mention that it works with any agent and it's open source?

Sam, it's great to have you on the podcast.

Sam Newman: Thank you so much for having me. I'm happy to be here, very happy to be here, big fan of the show. As I say, I'm not going to lie: I've become a bit of a millennial, and I think I watch a lot of Shorts rather than watching the whole things now. I'm becoming my son.

Host: Yeah, millennial is the new Boomer apparently, I learned.

Sam Newman: And so I was happy with Gen Y, now I'm whatever I am now.

Host: Well, so let's go back to the beginning. You've been known for a lot of your books, your takes on resilient systems, microservices; before that you worked at Thoughtworks. But let's rewind to the very beginning. How did you get interested in tech and computers?

Sam Newman: I'm quite fortunate living in the UK, we had access to computers quite early on. We had an Acorn Electron at home, which was sort of one of the generations of the BBC computers that came after the BBC Micro, and then I got a Spectrum 128 +3, which is the Spectrum that had the built-in tape drive.

Host: This is way before the Commodores, is that right?

Sam Newman: I think some of this was... I'm pretty sure that the Acorn Electron was pre-Commodore, but the Spectrum would have been around the same era. The UK was a bit more of a Spectrum and BBC Micro market, whereas the US, the Commodores and the Ataris and things were a bit bigger. So that sort of got me a bit interested, and then I always stayed interested in computers.

I did the very boring thing: I went to uni, I did a software engineering degree at De Montfort University, which the running joke in my family is it's in the top five universities in the Leicester area—there are only two universities in Leicester. I mean, it was... I loved it. It was a great class base in my syllabus. And back then we did these things called sandwich courses.

Host: Ah, sandwich course, yeah.

Sam Newman: And so the basic idea was you do two years of study, you do a year in industry that was paid, but you effectively go in as a student, and then you do your final year.

Host: Nice.

Sam Newman: This was actually fantastic because I kind of was drifting a bit. And when I did my year in industry, so many things clicked; it meant I was much more focused going in.

And I got to do work for a company called GEC Alstom. And we did software for the European Space Agency and gas turbine control systems, which was really cool. I joined there the first time for that internship, and then I went back after graduation. And when I went back, it was around the time of the first dot-com boom.

And so everybody left. So I went from being the person that was doing like nothing to the person who did everything.

I remember my first week there, I was given a Fortran 77 codebase. Fortran 77 was released in the year I was born, and every single loop had a break condition, because we were basically doing analysis on these Unix machines, and machine time was expensive, and we were worried about accidental infinite loops. This was us basically trying to solve the halting problem. So every loop across something like 8,000 files had a break condition of, I think, 9,999 loops, and if you went over that, the program basically broke and aborted. But the model had got bigger, the computers got more powerful, and someone had to go in and change all of them by hand.

Host: Back then by hand?

Sam Newman: Well, this is one of the nicest things anyone has ever done in my professional career. Because Fortran 77 was based on punch card limitations to an extent, your lines couldn't go very long. So you couldn't just insert an extra digit easily. My boss said, "I reckon it will take you about two weeks if you do it by hand." And then he slid a book across to me and said, "You can read this." He gave me a copy of the Sed and Awk book.

And so I spent two days learning how to do regular expressions. And I wrote a refactoring using regular expressions, accounting for line lengths and everything else. And I think that's the closest I've ever come to feeling like a god. I did it, and I thought: that's the thrill.

Host: That's the absolute thrill. I'm not sure if it's a joke or a story, but you know: two builders need to build a wall, and one of them starts to build the wall one by one, and the other one is drawing and making plans. They say, "I don't know, it will take us three days to build the wall." The first two days, the planner has built nothing while the first builder was doing really good. And then on day three, this other builder puts together a machine that builds the whole wall for them.

Sam Newman: And then, but sometimes the whole thing falls down the moment you switch it on! But I love that story, and that story was true. It's one of the early times I could remember learning something and having such a visceral reaction. And I remember that distinctly. And then after that was Thoughtworks...

Speaker 0: sensible graatin. and they've ked tell some stories about like what what happened in presh career. i told this story, i said, i added the night made the models speaker. and then one of the one of the grads said, why didn't you just replace it all the constant? so you you you change number ononin any time in the future was like, that's so i'm learning once again.

Speaker 0: yeah. so it's i supposed ble. i love it. it's like you, you get humble all the time is industry. you get to learn stuff. yes, as reinventure that we are going on. but you we're inventing better whalels whole time.

Speaker 1: so and i think it's just nice around us every now that i'm sure everyone who's built cold and programs, like as somebody have the realization that you do feel like a god that you can do things with these things, which is just so difficult to do by hand, right? like there's. i think we forget about that when let's kind a day today, and we're just like doing our stuff or even like right now, we using l ams, which you go also do te incredible magical stuff that would have been impossible. but like it's kind of call that we are in charge. we can do amazing stuff.

Speaker 0: you get that little phyzy dopamine here. and i think that swe're dropping with a bit now is that people are getting further away from the code that trying to work out like, how do i still get that want to doopen me? yeah, and it's same way. um it's not as much an issue for mikes bit. it ter ter, you know, have been doing someof thirty years now, and i've already been mostly working at the aof of architecture for and so architecture infrastructure for a while. so i feel a little bit more removed, but it's uh, yeah talking to. my sound is a programme is good into it around an interesting time right now,

Speaker 1: um and then you eventually made your worge to thought works, where you spent a bunch of time, what kinds things did you do at thought works? and what are what are things that you learn? there did anything?

Speaker 0: i mean, it was i joined when they were probably just under four hundred people left with us about four thousand, and i worked and know gogot to go away away place. i joined it. good time because a lot of the i mean, we were very much an XP shop and nothing that the cooexthing broken yeah. and in the call, the business still kind of primarily is and a speed to people, they, of course,

Speaker 1: but it exexpelike. what was that inffencby by cancanbcks XP?

Speaker 0: yeah, that extreme programming is very much. it's it's it's can extreme programming. there was a bit of a london flavor. um so the kind of london school of techgiand development and extreme extreme cheessea clumb XDC around here. so what was the london flavor of of XD sort of like if you wanna know you pretty to regrowing object orange software diided by tests, one of the worst titled books and existence written by a couple of friends of mine, stay stay freement in that price. and i think that was a lot. there's a lot of people in the community around he. they knew that as a rich meets have seen. and so i kind of is good good juice of people through there.

Speaker 0: and i mean, and i jow, i was same time was offered job by somebody else is a bank. the bank could tell me exactly what i was going to be be ing ing thought works. basically, the recruit to the time is now i of my very good ds ds or my oldest friends. now she said to me of, i don't me know what you be doing. and i like that sounds of the job for me, II it. it's like one. and this is that time when, like, you know, know, we talk continuue integration, but but the tooling wasn't there and talks about TDD and the tooling wasn't there. so it was in if we know it was around that sort time that we started. i ststted building the bill pipeline staff in about two thousand and four, and then i worked day day files was first first telead.

Speaker 0: uh, what sounds like day fly? yeah, yeah. he's my first tecular, though it works. i hight, i was on the people interinterfuge jaz humble to backfield for me a projeject melway. so so you project IOL of all places, and i think we use it back that it looks like, like, like, like, see, i'm so of the half. is this side me in half every every so II see myself. so we we needed someone to take over my job. and so i leent to you, jazz um and all around, what a love of interesting things were happening. so as fortunate enough to be there, while we were sort of kicking those ideas around. and obviously, people at daviand jazz, those those ideas together really, really well. you was t's surrounded by everyone talking thinking about these ideas. i think you got that i got the idea very early er that this starts all lot for grabs. these things ren't certain stone that we can push.

Speaker 1: and when you're saying these things like you vention the continuous delivery, you mention,

Speaker 0: what what put out the chechesed automation, um h ill, ill pipe, my optimization, uh infrastructural automation. i'm ennever to the US in two thousand and seven as she to work in a big, but she worked to google for it a year, even a half four, a thought of employee wow in do right? where is we were doing? we were had ading a project. uh, re's all been told about publicly is a great right up by mike bland all about our experiences. i was may is a few of me, a few, my toorist colleagues, some people from industrial logic and some people from a pivotal, the old pivotal preact preanquis there there.

Speaker 1: there were no for their really excellence and during culict when it was just brought these company when else.

Speaker 0: and we were there to teach googlers primarily how to write automated test and how to write code those testiball. and so i got out there, and i'm talking to people out the valanced stuff is ying. and we were going in these enterprise companies to intest automation using stuff like sylennium, and you go round talking to people. and so people didn't II mean conversations of people like yahoo and even a big chance of google who did not think you could do functional automative testing of like websites and stuff.

Speaker 1: i guess ss cause you just never that it right?

Speaker 0: yeah, absolutely. it's for me that i hate it. whenever i haven't pushed myself or by found a better way of doing something. and i was lucky enough that early days and thought like that was is the constant turd of ideas and reinvention. i mean, goodness knows how many mocking frameworks and vivional control things old II built two different replacements, accrews control myself right? and so, but like II know know, well, it was that that's how my great inkle DB deploy my friends. niccashley and grand tacky was was like the first dates basue frfrching ing tall, normally source thing succeed, it exist, he needed them. and it was like, if you, if you want to do to do it, that was in mind said. and so i think it was always this idea working at thought works was always i what's next, and i was quite fortunate that because of it was growing so much. it was always things needed to be done, and my devi was knowledge to say, yes to everything. so i got travell all over the world was fantastic. met my wife. they thought works. most of my best friends will come from thought works now still al to day. so although has not been there for a long time, it was kind of very formative. i think in terms of,

Speaker 1: and it is really interesting because laff works has uh alone of my pipeline. people who used to work there who are now really well known the industry for their ideas for for thing differently. some some of the educate people, others uh have created businesses to to build software in better ways in certain dimensions. and it's just interesting because like usually, i wouldn't think of how i'll just like put it the box. i you know, yeet product companies, like like google, like meet a like they have their builder on products and and you have like different tears of these companies, some can afford the best people. and they have an next ontimes you in culture. and google was known for this matters to smix stand. and there some companies, maybe who are like law untier, and then you have the social consultanes, who are you know, they don't built their own suffer. they no do too much a bit. they come in and they help companies build, or you have a problem. you hire a group of experts. and thoughtwork technically is in this group. and usually, you would not think that there's too much innovation going on in these places, but without thouthoughorks proves to me, yourself and other people, actually, you can have like when you are the best in what you do globally, conactually have a lot of innovation that happening. there is all right.

Speaker 0: yeah, i was innovation with the difference set of constraints. i mean, always say there about that,

Sam Newman: Many people have come into Thoughtworks that were really great startup product people. I mean, a lot of, you know, a lot of Thoughtworks people make it scale up. People, all together, actually very few real entrepreneurs came out of Thoughtworks. I know a few of them, don't know many of them. But I think there was always the opportunity to innovate even within a constrained enterprise environment. So much about working conditions as a consultancy who motivated co-sourced delivery at that time. We were doing kind of co-sourced delivery, but enablement-free delivery, your working conditions are often based on what the client does, unless you're explicitly there to change how the client built software.

Sam Newman: So what you got is what you work with, but that was self-perpetuating because we were already used to challenging and pushing internally and doing new things. That spirit carried on. And so I think we grew quite aggressively, but that aggressive growth was rotating people through with that spirit. Did that carry on? I mean, that we had to start most engagements with convincing people why testing was a good idea, why pairing was a good idea, let alone TDD. Yeah.

Speaker 1: I imagine how the conversation went, like a manager going like, "Well, what would you do here? I have like eight engineers. Now you're making it four, which sounds like a terrible thing to do."

Sam Newman: Which, and this is, I mean, my good fortune, because of the people at Thoughtworks. I got to know Martin Fowler quite well. You know, he supported me a lot, right? My first book. I still remember when he said, "Programming is not typing." That fundamentally made it click for me, as I was now having those conversations with clients. If the client thinks of programming as typing, and that's why they don't want you to pair program, then you've got a fundamental problem. Once we deal with that, then it's not a conversation about pairing. The conversation is: what do we think programming actually is?

Speaker 1: But let's bring this a little bit forward, because right now we're having the same conversation, right? For a lot of CEOs, even founders who have been building tech companies, they all look at what is the most time-consuming part of what we do in building software: 50% of the time has historically been writing code. So a lot of people think programming is code, and LLMs are really good at churning out code. So what is programming?

Sam Newman: Okay, what is programming? I think of Peter Naur's article — the guy behind Backus-Naur Form. He wrote that paper years ago, he was a Turing Award winner, and he sort of describes what is a program: a program is being a theory that lives in the minds of the developers, that is actually us working together collectively who build a program. The fact that we happen to make that program work through code — the code is really a side effect of how we work together. And I think product development is just that extended, right? What we are building collectively, we are building a system together, and the code is one of the mechanisms we use to deliver the system, it's not the only thing.

Sam Newman: So I think holistically, that's what it's all about. We focus on the code because that's historically been where we sort of spend the time. But I think this is also a classic Theory of Constraints stuff, right? If you speed one part up, you get constraints elsewhere and it exposes a friction point elsewhere. I would also argue that a lot of the time the coding was not the problem, not the bottleneck; it was how the coding was done, delayed feedback cycles and things. But I do come back to that piece again, like modern stories referenced recently: building software is a collective endeavor, it's people working together who build software. Kent Beck put it in his very first book that software design is an exercise in human relationships, right? AI doesn't change that. And I think all that's really happened over, say, the last 15 years in my experience is that we've got a bit better understanding that we need better working relationships — product folks, for example. The programming and creating is the collective of operating software, and it's not just developers who do that, never has been.

Speaker 1: And so after Thoughtworks, you've gone independent, right?

Sam Newman: I did three startups, none of them worked out for different reasons, so don't ask me for advice on how to succeed at startups. And then I went independent. Yeah, I was living in Australia at the time and moved back from Australia. And yeah, been independent ever since, doing kind of a mix of stuff. Ever since I've written the first book on microservices, which I think I wrote in my spare time at Thoughtworks. And then that book came out. And I'd been at Thoughtworks a long time, I realized that I was becoming institutionalized. You get to a point with a company, I think: these are things this company needs to change, and I've tried changing those things, and they don't want me to change them. So either I'm bad at changing them, which is probably true, or they don't think it needs to be changed, which is also probably true. It's like, I could hang around and get bitter, or I just go now. So I was going now. I thought I'd be an independent.

Sam Newman: It's been really interesting ever since, because on one level the shackles are off. The problem of working for yourself, though, is that the boss is frequently an asshole. So yeah, current Sam frequently is cursing past Sam for the commitments he's made.

Speaker 1: Haha, that's so true. Yeah, yeah.

Sam Newman: Yeah, yeah. I mean, as I say, I'm almost 50, and I'm still not learning the lessons, right? And yeah, made those mistakes enough times now.

Speaker 1: So I want to talk about microservices. You wrote the book Building Microservices in 2015, you started to write in 2014. And as reference, Uber started to talk about how they have thousands of microservices in 2015. So you were kind of writing this book right as the concept of microservices was starting to go mainstream across startups. Do you remember how and when microservices took off? And where was it even coined? Was it Martin Fowler and James Lewis?

Sam Newman: So going back in the day, you've got to remember again, it's like something in the water. Like before we have a bushfire, there was a lot built up before the bushfire kicked off. Say this would be 2011 to 2012. James was talking to a bunch of different companies who were building what would have been classed back then as service-oriented architectures. Earlier there had been this movement we refer to as the guerrilla SOA movement, which was trying to simplify service-oriented architecture, shifting toward REST Web services done by people like...

Speaker 1: Yeah, that's the original service-oriented architecture, had like this...

Sam Newman: XML formats, and WSDL, and Simple Object Access Protocol — never was a word "simple" so poorly used. So you had this whole heavyweight thing. And Jim Webber had been pushing against centralized managers, like "make the endpoints smart, make the pipes dumb" to keep things simple. This was all the background. We were also, at this time, doing a lot of work around continuous delivery. I was running continuous delivery practices for Europe at the time. And so a lot of engagements were like optimizing the final mile: you've written the code, how do I get it into production as quickly and effectively as possible, basically taking the ideas of continuous delivery. I was around when Jez Humble did those workflows back in 2006, having those conversations.

Sam Newman: James was talking to companies that were creating service-oriented architectures where they could replace the services quite quickly. You'd look at it instead as a fine-grained form of services. That was all happening. So when we had this architectural symposium up in the Lake District, James was calling them "micro apps" at the time in this room. And somebody else in the room said, "Microservices?" I don't remember who said it, we had a television on in the room and most of us were hungover. And then James and Martin write the first blog post. The first paper on Martin's website was written by them. So James is the original discoverer of that pattern they called microservices. At the same time, I was doing a lot of stuff around infrastructure automation.

Speaker 0: and so i was coming in it from a different lens. so i think james was coming it more from an architectural lens. and i was coming into it from a continuous delivery lens. i spent at this point, almost ten years would out be my focus. yeah. and then i was starting to see the architectural impediment. so if i want to change things frequently, but the things i want to change are integrated with other things...

**Speaker 1]: like monolith, yeah, although not now that i would describe it exactly right.

Speaker 0: i mean, i did a talk on designing for rapid release in 2012, which was touching it. so basically James had thought this through and put a name to this stuff and put some patterns, and at Thoughtworks the stuff i was talking about over here seemed quite sensible. and i always wanted to write a book. i mean, honestly, i wanted to write an animal book. i wanted to write an O'Reilly book, because as they said, the O'Reilly book is the book. yeah. and so i've just started writing. i'm quite lucky that i find writing enjoyable and quite easy. so i just sat down to write the book. but for me, it was an extension of continuous delivery, and it was a style of architecture that was only possible because of continuous delivery and because of DevOps. we'd had these ideas before—i mean Erlang is a representation of these ideas—but making it a real, practical, and tractable thing... really fundamentally, the whole thing's been an exercise in smuggling through concepts from the 1970s, which is fine by me. right, i get it: it was attention to information hiding, and coupling and cohesion.

Speaker 1: that's alright. but then as you said, like this was just a spark before the bushfire. and then the bushfire starts: like suddenly, again, i think Uber was a very notable company starting to talk in public about having thousands of microservices on conferences like InfoQ, and that got people's attention. and then it kind of seemed that overnight, a lot of startups started to say, like, "you know what, let's do microservices." it was a lot of rebadging going on.

Speaker 0: i mean, i was lucky to write the first book. Adrian Cockcroft and folks from the YOW! conference—Adrian was looking after performance and reliability at Netflix. and i also knew people at REA Group in Australia. and these are companies that were very much pushing the envelope in different ways. and so i had two very good real-world cases to draw upon in my book, but most people weren't talking as publicly around that. and also before Netflix referred to their architecture as microservices, they just referred to it as fine-grained SOA. and to an extent, you know, what happened at Uber and elsewhere: we didn't have a different word for it. then, of course, what happened was that everyone just called their SOA architecture microservices, which is sort of semantic diffusion, but that is to be expected. so i just kept fighting the good fight: is it independent deployability? and i positioned it as: it's an opinionated style of SOA. you may not agree with the opinions, that's fine, but that's what was repeated.

Speaker 1: so then what is the difference between microservices and what was SOA services? so in the architecture service orientation, what are the differences in your view?

Speaker 0: very nice. not to be clear, right, i have my definition, you might have your own definition, but i'm the one who writes the books on it, so like my definition... write your own damn book! um i always began it as: there are two kind of very strong opinions i think we hold with microservices. the first, and it's most important, is that your microservice needs to be independently deployable. so the unit of deployment, the unit of change, needs to be a single microservice boundary, a single service boundary, rather than deploying everything together.

Speaker 1: that's like deploying three services at the same time.

Speaker 0: yeah, minimum, right? and this is a single service. and the second one, which i think there can be exceptions to, is that the default way for defining your service boundaries is around your business domain. this is taking the concepts of bounded context that you have to do, rather than a lot of SOA architecture which was dominated to an extent by technical decomposition, traditional n-tier architectures, layered architectures, where you've picked your decomposition based on the nature of tech, rather thinking about the business functionality. that's a bit softer; independent deployability is the main thing though.

Speaker 1: okay, well, thinking back at Uber: i was at Uber at the time when your book was starting to become really popular, from 2016. and we had independently deployable services. we didn't really think too much about the business or domain. they had all these interfaces, they had contracts that they offered, increasingly strongly typed, and we used better ways to do it. but it was just more like—as you can imagine, there were devs often deciding based on their experience: "let's build a new service, here's the interface, then anyone can use it." these were developers, even not very experienced devs, spinning up a microservice and others started to use it. and suddenly, years later, there would be these exercises to figure out the critical path and label services. and then this loan service was on the path of something critical and needed to be reclassified as a really high-tier service because a lot of people were using it.

Speaker 0: i mean, that was always... also in some environments, hearing from SoundCloud, they had similar things. and this is also a lot about a corporate culture: what was acceptable in that kind of environment to go and do stuff.

Speaker 1: it's absolutely fine, and i think that's why Uber, for example, went there. i just talked with the CTO at the time about what was the real deal about microservices. and he said, like, he never really wanted that many services, but it was just more of a constraint because Uber had a monolith. the story was—which he told on a podcast, we'll link in the show notes below—that Uber had this big monolith, and it was really slow. it was slowing delivery down, and they needed to move fast because all the other competitors were out there. so they said, right, they wanted to get rid of the monolith into particular small services, but they knew unless they had a strong mandate that you cannot add new stuff to it, this will never happen. so he said the mandate was: everything new needs to be your own service, you take care of it, we don't care. and suddenly so many independent services were born, which was not the goal. and years after the fact, when they decomposed the monolith, they're now looking at maybe it makes sense to merge based on business domain, etc. but this story wasn't really told at the time; it was just all looking like, "look, Uber has thousands of microservices, almost one or two per engineer, right?" and this is what the world saw.

Speaker 0: i mean, it's also though... but this is the reality of microservices adoption: it's just a tool in the toolbox. i mean, i've described microservices as being an architecture of last resort. really, yeah, absolutely, because it pushes you quite naturally to a more aggressive distribution of your state and everything else. and so you have a good reason: one of the best reasons to use microservices is if what you actually want is an organization with a high degree of autonomy. if what you want is teams who are able to act independently from each other, the microservices architecture is a fantastically good fit. but then really it's a means to an end, right? so that's often a lot of people get wrong. i think you say "adopt microservices" and they think some magic dust is going to happen. and no: if you want organizational autonomy, that's what you're working towards, and this is just one of the mechanisms to get you there. and most of enterprise organizations, the vast majority of ones i've worked with in this space, that's what they're after. they've got huge numbers of developers, lots of middle management, they do not get much work done, and they are thinking: what do we do? organizational autonomy becomes the solution to all problems. again, i don't want to get into saying when that is a good idea, but i think that's where microservices can be especially useful, though there are other exceptions as well.

Speaker 0: and so the organiztions u. you know, uber IIIIII know netflix as well. things like mamaing exsignment talked about netflix, but the teams are already faadly. well scope around. it was approach already no. and you had those natural high level boundaries around the business dominins. so those things flow naturally. and so that for me, was a logical kind of stepping stone. you also have a difference. so this is i i've rerezzed quiely on when you go and do the work in the dot coms. this is stand prizoranization.

Speaker 1: when you say dog comes, you mean, the digital companies.

Speaker 0: yeah yeah, the companies say companies i'm old enough. i do talk about dog comes, but the tech companies, i mean, fundamentally algwith, you know, the make from the the main concept that inccredibly simple. the domains are very, very simple. compared enterprise ororananiztions by large, the mainins. they are much much more complicated because it based on business processes that have existed for ten twenty thirty, forty fifty fifty years. yes.

Speaker 1: yes. so so what you mean in a lot of of companies of product is pretty matasure re's re's product. here's hepth makes money, and that's kind of it. and we have another product in uber, and that's a uber like a simple thing as there's riits, which it makes money like this. and then there's uber eats, which is a separate business overtime by the way, does get more compleicpbut. but when ststart, it's really, really, really simple ah and it's really separate and it's clear like where one ends for just where the other begins. and then you do some hacks when in these launch uberriies with it quickly, and he built it on the right infrastructure. and the uberberies order has like two trips attached to it because but anyway, but but but as you say like early on, it starts very simple.

Speaker 0: but then there also you knows environments your business processes and the digital mentation as business processes is evolving together. a lot of enterprise organizations didn't have that mean their first generation of digital products were codifications of paper based processes also also youare talking to these companies as before digital.

Speaker 1: let's say, and adidst or a less issue company virtue,

Speaker 0: all banks,

Speaker 1: viviral, al banand. and now they're like we have all these processes. and now we need to build a software that mmics what we do, which, of course,

Speaker 0: that's that you start off up in. that's, but the whole imagzine book came at two thousand, and it was a magine zone came at two thousand four, and it was a reflection of the fact the vast majority of software development was the process of us trying to model existing non digital processes. and how you do that.

Speaker 1: i wonder it if we will have a different or but similar problem of the same thing. now that you know, we know AI tools are here. we know that, for example, the're greaagenererating code, and we have two sebs companies. we have a company, a digital company that has this is prea. i there used to be a writing, protocohehement document, deploying developers, product managers, designer. yeah QA. they all have the roles and it it works like a really well old machine. and now you have these so called AI native startups who just again, they have a few people they're just at from the start. they're using agents for everything. they have a very different postdent at someway. on this whole thing me will mature, and we will have best practices will have the existing companies who, like the previous on, they will try to retrorofch. you know, this, this tlling to how they're working today.

Speaker 0: it's already happening. i mean, then anything is enterprise companies will really be successful and doing those first genam things. because ultimately the corporate DNA would just crush that stuff from then right? and it'll take a while, but i think the whole industry getting is grrapping with the idea of, like, are we now moving to a new level of abstraction, right? so in socp, as all about abstractions, right is, but you know, we went from doing machine code doing somebody he went from the sebrgital purpose, programing languages. and you coming to idea of the programm being the mental model would evolve that mental model of the code one. if you that was a most effective place was us working at job. you know, java naxi shop whatever is rather than working at assembly level, right? and i okable is actually what we're talking about really, which dipping that level up. we still actually building a product together. we're still thinking about what the software is. we still building that mental model, maybe now more collabortively involving, you know more product people, for example, in that conversation or we writing aspects. and so the question for me is, are we ready to make that step? and the answer is no, not everywhere.

Speaker 1: this is a really good point because if if you think about it, like you know, like one of the the montrtras inside it facebook gies to you that i think some something along the land, that code is code st truth or something like that. but in generally a lot of tech companies, if you want to know like how things work, like where the true lives is that in documents? or is it in the code business logic? it's in the code in the business logic? and if if it's well maintain code often well comon to are structured, you can i should see the business logic in the rules for, i don't know, for example,

Speaker 0: say II feel like i've got to channel my friend charity at this point and say, the truooth is production that as thing elis is, is the lies we tell us. our production is truth. um but i think in terms of like a definition of what the system should be doing there. is this ongoing debate about where is that definition? is it the code with the spect brigeto brobroketed, a nice article on martin photos website recently about spectrum development, and she talks about the idea. some companies using uh being spect first or speak ankled. and a lot of it comes down to where they see the source truth being. so a lot of companies using things like kero or speck kit to define a feature, and then at the LM executes that spect, and you could iterate ate on proprocess, right? so so lot, we we always start off the user spect to write the code, and then they throw the back away and then get other companies that keep the speck. and then when the future changes that adopdate speck. and so there's all seeing a bit the code and a bit the spect is a source of truth until of what the system should do. and then you stannow start arting to get companies where they did even even look at the code. and they just look at the spect.

Speaker 1: no.

Speaker 0: and so we're kind of going on this spection. and she had some great terms for these. and i think for me, is like, is that the right path i don't know, i think, is a lot of questions about about we we d need to do to to get that. we have a degree of confidence about what happens when i ask you, you lot of javava code beyond the safety of the companer in the run time, you know, will that stuff there's a lot more questions to answer? i think before the whole world is ready to say, the speck is the source of truth of all assim is supposed to do productions always going to be the real tricks. so just to weekly.

Speaker 1: but but i think, is a good question. so just asked yourself, what how are your building and how we're using agents of where is the truth of your system right now is more? and the code do have a speck. is it scattered between the minds of people, uh, which which goes back to the the collaboration? because again, a lot of the reality of a lot of start tups ar is a few people hold these things in their head, right? right, which is okay, because and beginning, but of course, has you scale one, the reasons we used to love coal review, which is now going out of fashion. but we used to share information. and when someone left the company or or got l, or you know the the famous bus factor, it was okay, because other people people understood because they review, they were taken part. and that part of it. but now where we with but AI code of US, some some companies are like, well, we're not going to do it because of it's it's with too much code.

Speaker 0: that was wasn't the seseppof of paring right pairing. you wrote, if you do, if you do promisse as parent, you wrote at every day when someone leaves s everyone has got this same shared context. now it's what something people missed around the on the benefits appearing. i think it's a lot to be said around this i'was, trying to chris ford, ty sits of had a software. i had a technology for networks in in spay, and he said, the biggest problem we've got around uh around day.

Sam Newman: I do think software delivery is tacit knowledge. There's a lot of knowledge in our heads that we've built up that we don't necessarily know that we have. When you look at a code review, you could have a look at it: "It doesn't really feel right. I wouldn't do it like that." And sometimes it's subjectivity, but sometimes it's something about that you can't quite pinpoint. Maybe you can unpack it, maybe you can't, but it's also like, do you have bunches of things like that in your head about how software delivery should be done, about how the business works, everything else. If we want to step up that level of abstraction to work at a spec level, we actually need human beings to be able to unpack that tacit knowledge that we have and make sure it's in the spec, if we really want to be hands-off.

Sam Newman: I also think it is a fundamental problem that we've got in the current way people do software delivery. This comes back to cognitive load stuff, which is you've got a lot of people running off in different directions. I've seen a few people talking about pairing with their AI system—not a human, stop pretending it is. And so you're all going off, you're doing more in a different direction, that shared mental model of the program is breaking down, and these gaps between—AI is supposed to free us from drudgery, right? It isn't the most often developed. We have more and more code, and we're losing that big picture. So I think if we can work collectively on the specs, we can rebuild that model, then that could be the source of truth. I still think it's going to be partly our brains and partly the spec, right? But for that to work, there does have to be the collaboration piece, whether or not our dev teams go from being 15 people into about 5 people or 6 people. I don't know, but collaboration is really important.

Sam Newman: Jane Street, a proper company, right? They do proper stuff. They recently announced they're doing away with human-based code reviews, and they said, we can spend more time working on things like architectural design level. And actually, I'm okay with that. Because firstly, I trust that they know what they're doing, but also there's something about—I think a lot of companies are making the mistake of not doing the thinking, whereas Jane Street is saying, we're going to spend that time doing more important thinking, right? And I think that's sort of how we should view this. I'm okay not looking at the code as long as you're doing other things to make sure the system is operating correctly, and you're thinking critically about how the system needs to change and evolve. If you're just not looking at the code and you're not caring about these other things, well, then you deserve everything that's coming to you.

Speaker 1: Which also goes back to what Charity was talking about on how you want to have systems up, about having empathy, for example, for the work that QA and SRE folks have had to own code in production that they didn't write, they didn't necessarily read or even understand, but they know how to operate, know what good looks like in production, and they know what it looks like when it goes poorly.

Speaker 0: That's also a bigger thing that more developers are having to struggle with, because if you want to do spec-driven development, you have to know what good looks like. I run these HalfStack conferences online. We had one recently on software delivery with six speakers, and four of them were talking specifically around different varieties of spec-driven development effectively. And all of them basically said: you can't do this unless you can define what good looks like. And if you can't, don't bother. Some of those people were much more bullish, others less so. But one of the presenters—they've got like a small software team of three people—they're actively pulling in their observability data, their OTel stuff, right into the loop to understand how it is changing. I think that's exposing more developers into understanding—maybe it comes back to the practice of people doing user stories being always very narrow, functional requirement stuff. And now, I think we've got to be more grown-up about those things. The discipline is absolutely important, and the expertise is still needed, just needed in a different place.

Speaker 1: Sam made a really good point that you cannot hand your work over to agents unless it does these two things: one, define what good looks like; two, prove that your system does what this good definition looks like. You already know who I need to mention when it comes to proving that your system works as expected, and this is our co-presenting sponsor. Agents are writing more of the code. So how can we trust that our system works correctly without bugs? Code review is an increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes targeted testing and fault injection testing. By running the simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs—it gives you a perfect reproduction of every issue, which makes it a lot easier to fix issues. The first thing I thought when I heard about automated bug discovery and fully deterministic testing was that it sounds like science fiction. But actually, it's hardcore engineering when you look under the hood. Jane Street, Fly.io, and the etcd community ship agent-produced code with full confidence because they know it's been verified by Antithesis. To see more case studies and details, head to antithesis.com/pragmatic.

Speaker 1: I'd also like to mention our presenting sponsor Turbopuffer. We know them as a database that's really fast and extremely cheap, and they are doing something amazing: they are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. If you follow Turbopuffer, you know that their storage architecture was a massive part of their early success. Redesigning a winning architecture is a big deal. It's one thing for query plans to pass unit tests, which is already happening. It's another to bring each query plan to performance parity or better while maintaining correctness and reliability in production. And the team is sharing the messy reality of the rewrite, and how the v3 version is currently a lot slower than the current production one. For example, we can see how cold full-text search is 11 times slower than prod, and hot full-text search is 126 times slower. But we can already see some wins, like hot and cold vector search is at parity or better. I really appreciate this work-in-progress view and the real-time logs on what it takes to do a successful storage architecture rewrite. The Turbopuffer team is keeping a worklog of their journey, and you can follow along at turbopuffer.com/v3. I definitely will do.

Speaker 1: And with this, let's get back to Sam. On what QA and SRE folks know about running code that they didn't write—I want to talk to you about your latest book that is just coming out, Building Resilient Distributed Systems. What is the book about? And what made you feel this is a good subject to write?

Speaker 0: Yeah. Years and years ago, I went to Velocity, which is quite a well-known conference, an O'Reilly conference. And it was like the conference where a lot of the other dot-com people went to talk about how they made systems scale and all the stuff. It had been like one of my career ambitions to go to Velocity. I got to go after I'd written the first book, and I got talking to John Allspaw there, whom I still chat with now. He now works in the incident engineering space. He pointed me—I think it was him, or someone else—pointed me to this paper by David Woods. It was all about resilience engineering, and the paper was utterly baffling to me and absolutely nascent to me, but you know...

Speaker 0: Yeah, it's just not written for computer scientists to read. And for computer scientists, that was really true. But it was really fascinating seeing this whole other world about how people viewed resilience, and what we do in the software world. When I tried to take some of those ideas in a little bit, I found it very difficult to take some of those ideas from safety-critical systems.

Speaker 0: and that's my way of thinking and making any shape form palatable for normal people like me. and so i kind of must have been back, what 2016 maybe 2015. and so i go back and try and reread that paper, every six months or so, i'd read a bit around the agency, other smarter people than me that summarized the data better. and i was thinking okay these ideas now are starting to make sense. and then also, i kind of realize that despite my best efforts, lots of people seem to be building microservice architectures. even though if you read my books, most, my book is about telling you not to do it.

Speaker 1: do they read the book, which is microservices?

Speaker 0: let's be clear as long as they buy the book. that's the most important thing. whether they read it or not, you know. but and so then i realize that there are more people exposed to the, the horror, pain and suffering of a distributed system. and so it was like, okay. so i've got this stuff happening. i've got this like my sensitive that happen now. and i only thought that if somebody found themselves building a distributed system, working as part of building a distributed system. what was like the kind of equipment, a life ring, i could give them so they don't drown.

Speaker 0: and so the goal of the book is to start you off with really easy foundational, simple, kind of engineering concepts about how you can start making dealing with the problems of distributed systems, but also take you on a journey into kind of the whole concept of resilience engineering. and so it starts off without really easier ideas, and then steadily gets more and more complicated than you know. so i introduce kind of concepts, like, what socio-technical systems is the difference of safety management of resilience engineering is all big concepts. but second chapter of the introduction is the site, what's observability, talking about hotel, the third chapter is, what's a timeout? right? i think these are all quite tractable things taught,

Speaker 1: but they're building blocks, right, right, the primitives for, yes or for the systems, and the vocabulary as well.

Speaker 0: yeah, absolutely. and to give you know, to go through and sort of distill some of that knowledge down. and also to be a bit of quite happy being a bit opinionated on these things, but i'm also lucky now, because partly, because my new work, i get to chat to more people. so i contact people who this is their full-time job and trying to distill down with their knowledge a bit. and so i wanted that book partly is about being evergreen, but it was also about saying there are lots of people building distributed systems. put what type of distributed systems you've got off on side for a moment. there are lots of people building them whether they should or not. and they're really important to us now. so how do i? how do i help people start? and grow. and that's that's what i wanted to do. i mean, i enjoy the process of writing anyway, but this is a book i wanted to be able to give to people. that's always been right, right? a book. i want a book, even now i've gone sit down to write a book, i found someone else had done it, then it's already, say well that exists now. so this for me was a book that i wanted to exist and that's kind of why i wrote it.

Speaker 1: so now i think it's a book i certainly could have used like because i dropped into a place at Groupon where we did have what i realized were distributed systems. you don't just just large systems, handling a bunch of stuff, lots of services depending on one another, because even a simple thing of like knowing, like, oh, you should pass your observability context. i mean, you figure out after a while, but it takes a while, on your own.

Speaker 0: yeah, on your own. i mean, again, i mean, maybe this. this is my attempt to unpack my tacit knowledge to an extent. i've made so many mistakes and i keep making mistakes. i think making mistakes is useful, but i think it also creates opportunity. people can learn more from other people's mistakes, so it is a good idea too. and so in the book, i also try to pull in real examples and sometimes some big companies, sometimes some mistakes to kind of make this stuff seem real. i think the other thing i think i really try and get across is that things go wrong, right? engineering is all about making sure as many things go right as possible. it's not about making sure everything goes right. no. and the only way you can build resilient systems in general, is also by being honest about mistakes that have been made and talking about them and sharing them because that's how everyone gets better. if you don't do that, you can't put a resilient system in your company. and so this again, is my attempt to sort of highlight all the awesome work that has gone before, the fantastic post-mortem reports that some companies share.

Speaker 1: like when there's a system that we all use or many of us use has an outage. i mean, it's annoying especially when you're using it. but i always greatly appreciate when the companies, because they don't have to do this, but when they share a public post-mortem or incident recap, whatever that is, they go through what happened. for example, GitHub had an outage, and they shared a really interesting one on what, all the things that went wrong and the what what. and then you also have companies where something like this happens and they don't do anything. so of course, you're not obliged to do it anyway, but i always have a bit of a...

Speaker 0: regulations are now making them do it.

Speaker 1: which which i think it's great.

Speaker 0: yeah, think like DORA stuff of the UK, in Europe rather. and i think we should also separate around that post-incident report stuff that there's marketing PR versus, have you done a proper post-incident review? now i do think you have a duty of care that you just know as part of a proper post-incident review. but those companies maybe say nothing might have a fantastic process internally. it's also the fact that some companies that put those things out, you read them, you're like this does feel more marketing. i think if you want a good company that puts out some of the best self-flagellating post-incident review write-ups, go for Cloudflare for their stuff.

Speaker 0: i always, they absolutely go for themselves in those. it's like i feel like guys, take it easy on yourselves, right? everyone makes mistakes.

Speaker 1: but i quite like and do it, they did it rapidly, within 24 hours. and they are a publicly traded company.

Speaker 0: yeah,

Speaker 1: which which i think just goes back to the, like some of them, "unlike, we're publicly traded, we don't want to do it because it would give away information for shareholders." you know, like Cloudflare does it,

Speaker 0: and they're fine. exactly, exactly. and and also, the deposit of trust. a lot of how an organization is perceived comes down to how you deal with mistakes when they occur. right? i mean, yeah, nothing ever goes perfect for every company.

Speaker 1: so in in your book, you talk about the three rules of distributed systems. what are these three rules?

Speaker 0: so i really tried to distill this down because i could never remember the, the rule of eight that came out of some, oh, the fallacies of distributed computing. i always forget what those eight were, there's loads of complicated stuff. but i realized when you strip it all down, there's only three things that really cause all of the problems, right? the first is that you can't send information instantaneously from point A to point B. it's physics. i mean, now it turns out when you get in to talk to people about quantum physics, they say, uh, and there's a thing called quantum entanglement, which does allow the theoretical instantaneous read of electron spin states across arbitrary distances. at which point, i point out that A, that only works for reads as far as i'm aware, and B, not a networking protocol. so like, but it takes time, right? so i don't get from A to B instantaneously, and i can control bits of that as a developer. yeah, i could control how much data i send, and i could control the the payload size, technology i use to serialize and deserialize. i'm unlikely to be specifying the network cabling or equipment used or distances between them. so uh, not really under my control. and the moment you go to the public internet, BGP has a bad day, and your packet has gone down the wrong transit of the internet, right? and that's that's rule one. it takes time. and that's important to recognize because sometimes it looks instantaneous, it's not. second thing, sometimes the thing you are talking to isn't there, sometimes that... and we do things to make it under our control, right?

Speaker 0: like we would run, say multiple copies of service instance behind a load balancer. well, how if the load balancer fails? but i mean, like you can't control for everything. i've had services taken out by a fire at a data centre, a SAN caught fire, and basically one bit of the SAN. the SAN had a caching storage layer with SSDs. these two sets of SSDs, one of the sets of SSDs caught fire. the other set of SSDs are working fine. and so we had this weird latency graphs coming out of the SAN, like perfectly fine, if you routed through the ones that went on fire. but the failover is another one we're routing to be fair. it was routing around the bits that were on fire. i had this where we had ducting between office buildings and rabbits moved into the ducts. and ate the networking cable, i can't control for rabbits, right? and sure, things like that, that are real is the two things right? yes, you can control that and you can reduce chances, kind of know the chances, right. so sometimes, rule number two is sometimes the thing you talk to isn't there.

Speaker 0: and the third rule, i'm very grateful to a colleague for telling me for a website thing. i've missing is that resource pools are not infinite, meaning you can run out of computing resources, CPU, memory.

Speaker 1: i know whatever even basically a load balancer could be overloaded. and then just not respond even though it's there. and we know the latency, but it maybe queues fill up.

Speaker 0: the vast majority of the time when you have a system outage it comes down to that, some resource has been totally saturated somewhere. and it might have been that saturation might in turn have been caused by the fact that something is down and you're in a retry storm, which is in turn, causing you know, connection starvation, which is in turn causing resource saturation, which, in turn, goes on and on. yeah, but that that's the killer. and and again, another insight. and you know, to intellectually all of these things, but you can go through all the complicated concepts in distributed systems. and so oh, why spend time on consensus protocol thing. well, this is go way way down this bit like, what about CAP and PACELC. well, can i just like yeet this, put this up on one side. remember these rules first and then most problems end up becoming a little bit easier to deal with.

Speaker 1: and in the book after introducing these rules, you start, as mentioned, second chapter is observability, that kind of suggests that this is a really, really important part for distributed systems. why is observability so important?

Speaker 0: i mean, the way to think about observability is, is a property of a system, right? so the more observable something is the easier it is to understand how that thing is behaving based on the external signals. when you have a system, which is a single system, you actually get some pretty high quality signals coming out of that program. therefore, giving you a high degree of observability, you can get a lot of information out of that. i could attach a debugger to a process. so i could generate a core dump and run that in. i could just thread dump on the JVM to work out where my semaphores are blocked. if this is my whole system, like knowing the CPU stuck at a hundred percent is actually really useful information cause that's probably a problem right. so this basic information we pretty much get for free with a single program.

Speaker 1: this is a whole degree of observability for a single program running on your machine that you have access to, exactly right.

Speaker 0: but now we go to a distributed system, you don't have access to the computer the same way. you've got multiple computers. and so those basic signals are harder to get. you can't correlate the information without actual work. and those individual signals by themselves, don't really give you a proper sense of how the system is behaving. so you have to do more work to make the system observable. and as with these sorts of things, it's much easier to do that from the beginning, than it is to retrofit. and you know luckily, we've got things like now the OpenTelemetry protocol and specification, because that at least gives us an open framework that is vendor-agnostic. oh, Charity Majors gave me stickers just to be clear. i always work independently. i have to be really clear Charity gave me stickers, but like we get a vendor-agnostic open standard benefit. now it makes it so much easier for me to have conversations about this. but if i don't get the signals coming out of the system, if i don't have that stream of structured events telling me what the hell is going on with my system, i'm flying blind. but how do i know what an appropriate timeout is to a client when you're talking to a service? well, i'm only going to know that if i have a sense of, well, what is its normal response time and distribution? so i'm going to need latencies, know the success or failure of those latencies, probably pull up the histogram, maybe take a look at it. if i don't have that information coming out, i'm not going to be able to specify timeouts correctly.

Speaker 1: and then we're talking about a low level thing like latency and things that we can measure at the machine level. but of course, observability, i'm talking about like doesn't work for the user or the customer, doesn't it do what they do when they click the button to submit? does it submit? do you look at that? because we know observability, it's usually talking about these kind of latencies, the status, the 500 responses that's going bad. that means it's down. but there's always a part that i always wondered. and i found it hard even when we're working in teams to, okay, like we have the easy observability, like this is kind of textbook...

Speaker 0: not textbook, it's system metrics, it's the metrics.

Speaker 1: but then there's the stuff of does it work. does it do what people care about? like again, in the case of like an Uber, like when people hit a button to request a ride, does it actually request within what is usually five seconds or something like that, or is it stuck in an infinite spinner? which is bad, even if your logs can be as healthy as can be. but people are upset, right?

Speaker 0: you treat the raw data of making this observable as a stream of structured events. and from that, you build up the higher level abstractions and a few scenarios, is so a trace right, where we get sense of for a given click of a button, what happened in the production system. and this is a big blind spot, right? most organizations had to build our own tracing systems. this is where things like the OpenTracing stuff came from, and things like Zipkin and things like that, but the attempt to recreate like mine, most people just tried to recreate what Google built there. and it's an attempt to say, okay, can we at least look at a flow, what happened during that flow? where did these things go? well, that requires me to get a bunch of spans across all these different machines, and a span meaning an activity, something i did, and then a span can beget more spans. so i can say my first span might be a call that hit my server, right? so i've now got a span that starts with it that span. i might talk to my database. and these are operations, and i can put additional information on these spans, and then i could reach another service and then you get a trace ID that shared across those spans and you got timing. and now you get effectively a view of what happened across those calls, but that's all just based on individual events stitched together. and then you also start getting the system with a situation, which is is my system behaving correctly. and this is going away again, going away from vanity dashboards. this is moving into what we call the SLOs, right, service level objectives and SLIs. an SLI, service level indicator, that's our way of saying at this given point in time, for this thing, is this thing good or is this thing bad. and we use a collection of those. so one example of that might be a successful percent of response time, right? so that would be is it a good one or bad one. it could be a bad one because it's an error. it could be a bad one because it's slow, good or bad, and i collect those together. and that helps me know if i'm achieving my SLO, so i might have reaching my SLO...

Speaker 0: which is so I should be able to have response, right? The 99% successful requests within 300 milliseconds, you know. And that would be your statement, and that would be a commitment. Yes, that should always be a commitment, that's the thing our team is working towards. You then get this raw data; you can't create SLAs on individual things, and then you use it to work out what resilience is looking like. If it's green, everything is a good time; if it's red, people are unhappy. I mean, you can get it this way, we have new ways to get a great introduction to this stuff. But without those building blocks—because like I talk about in the first chapter of the book, nobody says "I don't want my system to be resilient."

Speaker 1: Yeah, no, no one.

Speaker 0: But at the same time, resilience is not a binary property. It's not like authorization; it's a bit like observability, a matter of degree. So then the question is, how much do you need? Because am I having a conversation with you about multi-cloud and, you know, doing complete rewrites of systems for critical spendings like Monzo do, for example—

Speaker 1: Yeah, they have a standby bank. They go down that course so that it's sitting there, it's costing them money for the most part. But then just a few weeks ago, their main application went down for a short period of time, and they actually activated that bank. I'm assuming it's costing millions of pounds or dollars just to keep it there.

Speaker 0: As a side bar, as I understand it, their standby bank has a real stripped-down list of features. It's much more bare-bones. It's all of the core ones: like you can freeze cards, you can take money out, and check your balance. It's a complete rewrite, so there's no shared code because they worry about a bug across different cloud vendors. It's also by design significantly cheaper to run. So they've got thousands of microservices in the full stack; this is about a tenth of the cloud cost, and they also run it constantly.

Speaker 0: Yeah, so that is always there. I know at Monzo, right, that's how that works. But there is no point starting it without those foundations. So actually, as I put observability, it's like you need the raw data, but also you need to make decisions about—comes back to what good looks like. Then you need to know: okay, do I need to worry about multi-cloud vendors, DR strategies, because that's where I need to be going? Or am I somewhere else? And so I think without that kind of sense of your North Star of how resilient do we need to be, without that defined, you don't know how deep you're going to get. It's like trying to say: "How secure does my system need to be?" Have you done a threat model? If you don't have a threat model, I don't know.

Speaker 1: And I guess, answering the question of how resilient do I need to be, it greatly helps—and maybe it's actually the requirement to make an informed decision—is how much will it cost me in terms of operating costs, in terms of time to build it, in terms of, for example, latency. Just a simple example: if the business says, like, "Oh, would you like multi-cloud?" and business is like, "We would love multi-cloud! If Amazon goes down because we heard they had an outage last week or a few months ago, I actually want to be able to fail over to Google." But then you'll be like: okay, well, first of all, this will mean we're paying two cloud bills. And if we want full replication, it might increase our latency because we need to wait to write to both, which might mean slower response, you know, customers might see things. So anyway, there's all of these trade-offs. And my experience is, oftentimes you might retrospect that, but if you can have the discussion upfront, that's great. But to have the discussion, you need to know what the trade-offs are, you need to know what's happening. And all these dimensions that you care about, like how long will it take to build, how complex would it be, how much money will it cost to operate.

Speaker 0: But also as technologists, we have got to get good at communicating with our business stakeholders, with our users. "What percentage of latency do you want?" Some business people know what I say when I say that, but how many experts in their field, who are great product owners or subject matter experts, but are not software developers, wouldn't know what I mean by that. And so we also have to understand the business domain, the product domain well enough to do the interpretation and help bridge that gap. And it is a dynamic thing as well. I mean, it's the whole thing about resilience in general, right? A resilient system is one that changes and evolves; if you don't change and evolve, you are a brittle system, not a resilient system. And so even those targets we set, all that conversation we have to have about what is good enough does change.

Speaker 0: But I think that's the case of, say, the US-east-1 outage in the last year. There were lots of AWS—the AWS us-east-1 region went down. It's the oldest region, where lots of people's first services were launched historically. When that went down, it caused a lot of people to say, "Do we need to be multi-region?" And some of these companies would have had the conversation before and decided "No, we don't," and then having seen the fallout, they have changed their minds. And that is fine.

Speaker 1: That is what you mean by a resilient system evolves? Yeah, with the real world.

Speaker 0: But also, it is about what problems you're trying to solve. There were some big-name companies that got taken out when us-east-1 went down, or got severely degraded in what they could offer. Companies like Netflix were severely affected. Do we think the people working at Netflix are idiots? No. Do we think they knew that if a cloud region went down they'd be impacted? Yes. They probably looked at that and said, "Well, to come up with a multi-region approach is going to have this impact, it's going to cost us this much money, it's not worth it." Let alone talking multi-cloud, because multi-cloud versus multi-region is a whole different conversation. And that's okay, as long as you had the conversation. And this is the other secret of it, of course: as long as what you think you need to do is in tune with what your users think you should be doing. That's why I made the point in the book talking about the difference between SLAs and SLOs. The SLA is what you're going to get sued for if you breach; if you only ever delivered to the SLA, your customers aren't going to be happy. You've got to do better than that, right? There is always that social contract.

Speaker 1: Now, with building distributed systems and building resilient systems, it really helps to understand concepts that are not as straightforward. I'm not going to go through all the things—you explain a lot of things in the book—but I do want to touch on a concept that I found tricky to understand and then appreciate, and you really appreciate it: idempotency. Yeah, what is idempotency?

Speaker 0: So there is a mathematical concept behind idempotency that I don't understand, but basically an idempotent operation is one that you can apply multiple times without it actually having effects multiple times.

Speaker 1: So it has an effect once, the first time.

Speaker 0: A really good example of an idempotent operation would be pressing a lift button. If you want to go down in the lift, you press the down button. It lights up, but if you press that down button again and again, nothing changes. The lift is still coming—we hope the lift is still coming, right. That's an idempotent operation. Whereas some light switches toggle the action: it goes on, toggle it and it goes off—that's not an idempotent operation. In computing, we have some operations which are inherently idempotent, like reads. A delete is typically idempotent, right? But the example I think I use in my introduction to this concept is a one-off payment.

Speaker 1: I was about to say that, I have a much better idempotent example, yeah.

Speaker 0: Yeah, payment is a good device. I'm going to pay you money, so if I...

Speaker 0: If I want to make a payment, and I send you "please pay Sam a hundred pounds." Yeah. This is, I'm emailing people back at home, right? "Please pay Sam a hundred pounds," and I don't get a confirmation back. I don't get back a response. From my point of view, there are two possibilities. The first possibility is that you received that payment request, you've done it, the payment has been completed. And the result has got lost to me for some reason, I didn't get a response, that's one possibility. The other possibility is that you never carried out the operation, maybe it didn't get to you, maybe you crashed.

Speaker 1: I think like a good example that I've seen in the past, maybe it's over, but airlines: you're about to buy this airline ticket for a thousand dollars, two thousand dollars, you've checked all of your details. And it says like "Pay" or "Reserve". And then often, some airlines will take you to an intermediate page which says "Please do not refresh this page or submit it again, because you might be double charged." And what you really want is you pay, you know, like pay. And you get the spinner, and if it's not idempotent, if you do it again and you get charged twice, yeah. But if it is idempotent, if you just refresh, you copy the same thing, you submit the exact same form...

Speaker 0: That's interesting. So the question then becomes, right. So I send that payment for my airline ticket, whatever it is. And I haven't had that response back, but I really want to make sure that the airline ticket booking can go through. This is you with the browser, or the people at home sending a hundred pounds, right? You think, "I must make sure Sam gets this hundred pounds right now." One thing in a distributed system, because of rule number two...

Speaker 1: Rule number two means there is rule number one...

Speaker 0: Rule number one, sometimes things go wrong, yeah, and accepting these things is going to... You just try to retry, because you really want it to happen, yeah. And retrying, there's the danger that if the first operation had been processed successfully, you didn't know about it. Well, the first operation had been applied successfully, and you do it again, and now you pay twice, you know, two tickets. And that's the problem without idempotency. You want to make it safe to retry operations.

Speaker 0: So to do that, to make it idempotent, there's two ways to do it really. One is good, and that is what you should do, but you kind of have to retrofit, which retrofitting is a pain. And one, which is one you should explain, which is useful, but has problems, right? The first one is idempotency keys. Yeah. So you generate, you have a unique ID. You use a UUID for this, yeah.

Speaker 1: And so when you generate the unique ID, and when you generate two of them, they should be different. There is like this tiny, tiny, tiny likelihood of clash, which might again, that would happen.

Speaker 0: I think that's like, I think with UUIDv7, I think it's like a 59-second chance space, something you can generate locally without round trips. Yes, collision chances are extremely low. It's a way to modify that we can work with that.

Speaker 1: And this, as long as your Amazon problem...

Speaker 0: But this has been, yeah, the AWS APIs you see, Stripe, PayPal, they all use the same key mechanism. So you send a UUID. So then if you send that to me, and say "Pay Sam, and here's the idempotency key." I don't send you back a response. Whatever reason you retry it, you pass that idempotency key. If I had carried out the processing, I could look at it, "Have I processed this? Yes, I have, okay great." If I have processed it, even if I get a second one, third or fourth, "Have you processed this?" I just keep on saying, "Yeah, processed. This process is fine." That's a nice simple thing to do. The issue with making things idempotent with idempotency keys, it requires both client and server to support them. So it has to be a required field to go through, and it has to be something the service supports. So if you're going to retrofit this in, it's a pain. But if you're building it, use an idempotency key, use it exactly.

Speaker 0: The other one you mentioned at the beginning was: I go back, I hit refresh, I fill out all the same fields again, and I submit the airline booking again, it should know it's duplicate. That's another thing you can do. This works as a good retrofit for idempotency. So if I've sent the same fields, the same person Sam, the same amount 100 pounds to my bank account, and I'm sending this through again, what you can do on the server side is generate what's called a fingerprint. You take these fields, yep, and then generate a hash that represents those fields. So if you do a second booking, it looks the same.

Speaker 1: And you're using that as an idempotency, as a retrofit of idempotency keys.

Speaker 0: Yeah, there are a couple of downsides. So that you do get false negatives, meaning that legitimate second requests might get rejected. Real-world example: this happens in Denmark, and it might happen elsewhere. I speak to people in Denmark who work on a payment system. You go in, you buy something for 40 kroner with this particular payment system, and you tap your card, 40 kroner, whatever. If within the next four or five minutes, you buy something else for 40 kroner in the same shop, the till handset rejects it. If it was 41 kroner, it would be fine. That's because the chances of you buying something for the exact same money is so remote. There's some idempotency to stop double taps.

Speaker 0: And so when you get situations where operations can legitimately look the same, but actually be different and carried out at the same time, the really good example would be the AWS APIs. It is completely legitimate for me to, in a for-loop, provision say 20 EC2 instances. Now there's better ways of doing it than cycling in a for-loop, but I could iterate through the for-loop and spin up 20 identical EC2 instances. If I just relied on there being a fingerprint, they are all the same. So that's where this stuff breaks down.

Speaker 0: And so if you've got situations where legitimately you have operations that come to look the same, but they're actually different, they could happen close to each other, you have an issue. If you've got a bit of space around them, what you can do is hold onto those fingerprints for a period of time: five minutes, ten minutes, an hour. There are still some potentials for issues, so it's not ideal. Idempotency keys are the way to go; fingerprints are for retrofitting. But if you use them both together, that can be really useful, so you can pick up situations where people change the parameters between retry attempts, which leads to confusion—a bit more of a complex topic. But a lot of public-facing APIs will still generate a fingerprint, and if you carry out this operation and you give the same idempotency key you gave before, but you've changed parameters, the server can reject it as being an invalid request. So having those things work together is great. The fundamental ideas are actually really simple, but there's some nuance about implementing them and how you store them on the server side.

Speaker 1: I think another interesting topic that comes up often these days, it feels more often, is thundering herds. Yes. It's when a system restarts and suddenly it faces a big load. Specifically GitHub had an outage a few weeks ago where it was networking related, and recovery took a lot longer time because there were retries going on, and it creates this thundering herd. What is a thundering herd?

Speaker 0: Generally I talk about it in any situation where you've got mismatched capacities. You've got too many requests for too few resources, right? Resources are not infinite, coming back to rule number three, yeah. So in the chapter, I talk about a whole bunch of different scenarios.

Speaker 0: So you've got everything from malicious... like a denial of service attack, right? Denial of service attack. Malicious attacks often come at short notice, and there's not that much you can do about them. The answer is you'd use Fastly or Cloudflare. I won't go into details of how that happens. But you also get things like kind of internal, self-inflicted onset. So a good example would be something like a cache collapse. The system relies on the cache not only for speed of operation, but also for reducing load on other components. Yep, the cache gets restarted, you then have a lot of cache misses, you'll hit the origin, and the origin goes down.

Speaker 1: Expected such load.

Speaker 0: Yeah. So a common thing there would be, first, you use flush cache snapshots to disk to reduce the chance of having high cache misses every restart. But also a lot of companies will just turn their service off until the cache refilling finishes. So that's kind of another example. Another one is retry storms. Back in 2017, Square had a big outage that restarted Multipass, which was their multi-factor authentication system. Multipass in turn pulled its data from Redis. They were using it as a database, because you can do that, apparently.

Speaker 1: I think... they are using it as a cache, basically, right?

Speaker 0: Well, I think they were using it as a database. There are some people who think Redis is a database. That is a different podcast altogether. I think you should start a "Stupid Technology Decisions" podcast, where we talk about using technology stupidly. Great. Anyway, so they were pulling state back from Redis. And there was a problem that the clients in Multipass, if the Redis calls failed, they'd try again, but without any delay between retries, and they had a hardcoded limit of 500 retries. So their retries basically smashed Redis, and they kept restarting Redis, and then they'd restart Multipass. And again, this was a long time ago, and they fixed it, right. They fixed it by having a more sensible retry policy and externalizing these values so that thundering herds would never happen.

Speaker 0: The last kind of section I talked about were ones caused by products being too successful, like Twitter or Healthcare.gov, right? This is where you have external, real, legitimate load that takes you out. And when you look at all of these solutions, although what happened is the same, the kind of solutions were very different. So with a thundering herd that's coming from external legitimate users, you've got different things to look at. Like, I would have more constraints on signups and have fewer invite codes. With Healthcare.gov, one of the issues—there were many others—was that they went live in like 36 states at once, and they were meant to be overloaded. In hindsight, maybe going live with one state first would have been a good idea, for example. With Twitter, they were going down all the time, so they could have constrained signups or put tweet limits on you, and things like that. Yeah, constraining those sorts of things. And they did those things as well, right? So although they look the same, the mitigations for each were a little bit different. So I quite liked talking about that, because they were all great stories I found.

Speaker 1: Of course, like, I think we're talking about solving it when you have the same resources.

Speaker 0: But then we get into things like scaling. Like, can you scale horizontally and add more resources? Because there's two sides to this, right, whenever you have it. At what point can you do it, or at what point do you face a bottleneck because of a central component or whatever?

Speaker 1: That's a great example of where sometimes mitigations... you take something like a startup that suddenly got successful. That success comes like a wave, right? The wave of success hits you, and then it recedes. You'll get a thousand customers in, and maybe you'll keep a hundred, right? That's just how things work. So you need to be up during the peak, and you probably need to shrink down again to keep costs low. So if you build your application to have dynamic scalability, and if you are on the public cloud or a similar place where you can spin up and spin down, that's a viable mechanism, right? Okay, we can scale up to handle this, that's fine. Would I use that against a denial of service attack? No, because then your distributed denial of service attack becomes a distributed denial of cash attack, right? You go from not only having no service, but also now having no money, because you keep spinning up nodes on your Kubernetes cluster and Amazon just keeps taking the money. It's great, and you still can't keep up. So that's a perfect mechanism for some things, but you kind of have to know the context, to say you can't just throw resources at trouble.

Speaker 0: Right, I think we're going back to what we say a lot of times: as an engineer, you want to know about the business. You want to know what creates value, what creates revenue, what is important for the company versus what is not, and where we want to maybe invest with our infrastructure and our reliability versus where it doesn't matter and it's not worth doing. It's all about trade-offs.

Speaker 1: And I think it's about... you don't know unless you get information about your production system; you don't know how close you are to the red lines. That's what observability is key to—it's what trade-offs are all about. And that's why I do a whole chapter on formal methods, I talk about CAP and PACELC. And I say that these things are complicated, but actually it really all comes down to a conversation about understanding the business context. So for example, at an e-commerce company, if I want to buy a physical item online, and at the moment of the purchase you're not sure if you've got it or not, most e-commerce companies will say, "Yes, sure, you can have it." Because they know there's a good chance they do actually have it in stock. And if they don't have it in stock, they can get it back-ordered and deliver it. And in the worst-case scenario, they can even say sorry and refund the money, right? So that's not great, it's not ideal, but if they say, "We don't know, therefore we're not going to sell it to you," they're missing out on a sale, and I could go buy that somewhere else, where they probably might have had it. So e-commerce companies in that situation understand the relationship with the customer and the impact if they get it wrong—they can try to fulfill the item later.

Speaker 1: Keep that problem exactly the same, but now change the business context: now it's concert tickets. So you're getting tickets to go see Taylor Swift, right? You go to see Taylor Swift, and we're not sure if you've got that Taylor Swift ticket or not because the backend system is down. But the site tells you, "Here's your tickets." You take your kids, take your wife, fly to Portugal, all excited to see Taylor Swift, and then it turns out you don't actually have the tickets. The problem with that is, the devastation is entirely different. They can't back-order a ticket, yeah. And so in that situation, the impact to the customer is so much greater than the cost of the flight and everything they put into it. So what they do is make a different decision. In that situation, if they don't know if they've got a ticket, they do not issue a ticket.

Speaker 0: I'm not sure if we covered it in the book, but this is the fail-open versus fail-closed debate, right. When you have an error and you're unsure, are you going to let it go through or not? And as you say, this can be a decision where, for the same problem, two companies can make completely different decisions. Or a company years down the road could make a different decision. Uber had this classic example when it came to payments: when we weren't sure the payment went through, do we allow the ride to finish or not? In the early years of the company, we always failed open: yeah, just let them have the ride, we might get the money later, and if not, we'll deal with it. But as the company started to go for profitability, we failed closed: like, we're not sure we have the money or that the charge went through...

Speaker 0: You're not going to get the right and you get the same company different focus. And if you're the developer you needed to pay attention to like what is important for the business. It was growth initially. It's okay if you lose money, then now it's I don't know, unit economic profitability.

Speaker 1: But also that open close as a layout open discussion isn't one size fits all for the whole system. It's on a feature by feature basis and even...

Speaker 0: Even in the system.

Speaker 1: You could say this taxi ride is 10 euros versus it's 1,000 euros, you can make different decisions.

Speaker 1: Now if you approach this from a mathematical and from a computer science viewpoint, we talk about CAP theorem. That was a conversation about CAP theorem, right? So I talk about CAP, PACELC, which is the extension of CAP. I explain what these models are because when you get deep into architecture you hear about them. But fundamentally, that's what this is about, right? When you have a partition, CAP theorem defines: I can't talk to a part of my system.

Speaker 1: Remember, it's rule number two: I can't talk to a part of my system. What do I do? And all you need to know as a developer: I can't talk to that, what do I do? Sometimes we can make a technical decision: I'll try again. I hope I don't bring my system down with retry storms. But I'll try again and again, if that doesn't work, what do I do?

Speaker 1: Well, actually often the answer is, yeah, you look at CAP and PACELC or whatever else, but go have that conversation with the product team: "What should we do?" The resolution of that often ends up being a discussion about: are we trading off consistency versus availability, PACELC, harvest and yield, you're going into that evolution of Brewer's conjecture. But don't worry about that. That's in the book. That's why it's a later chapter, chapter 11 or 12 I think. Don't start there, start with having a conversation: rule number two, I can't talk to this dependency right now, what should we do?

Speaker 0: And that starts... when talking about resilience, you talk about the dimensions to think about resilience. What are these dimensions?

Speaker 1: I start in the first chapters. So climbing back, that paper by David Woods I mentioned that was impenetrable to me at the start, and he comes up with four concepts for resilience engineering. And I like the four concepts he comes out with. It helps us think broadly, right?

Speaker 1: So it starts off with robustness. This is where we can basically, in his words, absorb known perturbation. So we model the system and something goes wrong, and we can just deal with it in a way that no one cares about, nobody knows. So a very simple example would be in Kubernetes if a pod fails, Kubernetes creates a replacement. Now, the pod failing, that's a perturbation. It's going to go a bit wrong, but we deal with it, great. So that's the start.

Speaker 1: And the thing is, most engineers I speak to, they focus entirely on this and didn't look at the other three. They focus entirely on this. The other annoyingly true thing about robustness is the things we often do to make our systems more robust in turn increases the complexity of our system, which in turn increases the surface area for failure. I bring Kubernetes in to deal with the failure of a container, now I've got Kubernetes and that becomes... and that's not solvable by the way, that's just a thing to be aware of, right?

Speaker 0: Yes, that is also why sometimes people don't go multi-region or multi-cloud because it's increasing the surface area of the failure types.

Speaker 1: Yeah, that's robustness, right? For things to be robust, we have to know about them ahead of time. This is predictable, we deal with it. Then there is rebound, which when something does happen and goes wrong, how quickly you can recover.

Speaker 0: So this is typically the time to recovery, well time to detection and time to recovery.

Speaker 1: Yeah. That's the metric of measuring it. In DR chat, I talk about disaster recovery, but that's like how quickly could we get it up and running. So we've got into degraded functionality, how quickly can we get it up and running? That's thinking about the rebound. And again, for that, you have to be very humble, because robustness is "the system will never fail on the inside", but what if we got it wrong?

Speaker 0: We have to admit things go outside and as for how you can recover, this is where teams that are serious about this do simulations of this going down. This is where Netflix's very famous Chaos Monkey comes in, where it goes and shuts down part of the system to test it live. And there's drills that are being done on how our on-call process works. That's a hard part. And doing it once is not enough, because you need to do it regularly because the system keeps changing.

Speaker 1: I'd like to shout out to Uptime Labs who are based in London. They do excellent incident training that you can do online, fantastic. So you've got that recovery process, how to get up and running. And this links next to the third one, which is graceful extensibility.

Speaker 1: And so graceful extensibility, it's a slightly weird name, it's basically dealing with surprise. When we make our system robust, we're dealing with the known problems.

Speaker 0: Yeah, what was expected.

Speaker 1: What about things we don't expect, right? If we don't expect it, this is now not about computers, it's about humans. Technical systems are fundamentally socio-technical systems, people and technology working together. How do we deal with what hits us when it turns out badly? But if you're in a corporate environment which is very command and control, hierarchical, with a very narrow job description, you're often constrained in what you're able to do. Those organizations tend to deal with surprise badly because a corporate structure is structured around the known, and so structurally you find it difficult.

Speaker 1: You mentioned drills earlier. Drills are great for helping with surprise, having people from the outside saying: "This has happened, how do you deal with it?" Google had a thing, and SRE teams do it around the Wheel of Misfortune. They spin the wheel: "Today there's a DNS poison attack by a rogue state, this is happening, how do you deal with it?" That is all about people, experiences, trying new things, and constantly pushing yourself.

Speaker 0: Because learning also from others. I remember this very famous outage happened a few years ago at Meta, where their DNS system through BGP just all went down, WhatsApp, Meta, everything. To fix this, they pushed out an update to their BGP table which severed DNS to their servers. Now to fix this, they needed to get physical access to the computers because they couldn't reach it remotely, so they needed to go into that building. Now in going to the building, they had to go through a door that protected the system which was running on their own domain, which was down. So they had to physically break in. But again, this is a thing where they shared the details, which was great, as now everyone else could think about: do we have any of our physical security tied to our own domain? Because if you have it, you want to change that, learning from what Meta learned about.

Speaker 1: And this links from graceful extensibility, which deals with surprise in the moment, into the fourth one, which is sustain adaptability, which is: how can we learn from what happened? Not just the learnings—do we have the environment in which we can learn and share and grow, but then also adapt the system based on that learning?

Speaker 1: And it's called sustain adaptability because it is a completely continuous process. That's not just today, that's not just when you have an incident, that's when you don't have an incident: what are we doing to learn? Because I think your point you made there was excellent, which is: I like reading other people's incident reports. There's a little bit of schadenfreude with companies you don't like, but it's always...

Speaker 1: So a little bit of it is like, what have they learned? Can I learn from them and do something that results in my system improving? So when you don't have an incident, read other people's incident reports. There's a place called The VOID. It hasn't been updated recently, but there's like hundreds and hundreds of incident reports, so it is worth going through.

Speaker 1: So that idea of sustained adaptability is all about us adapting and changing over longer time scales. For those last two things, to say that's reaching graceful extensibility, this all comes down to people and culture, like having proper environment. Psychological safety is just absolutely required. If you can't raise issues, if you can't challenge in a safe way, you will deal with surprises badly, because things can't get raised, and you won't learn, because things will be hidden. Yeah.

Speaker 1: And so that all comes down to behavioral stuff. And so the last three chapters of the book, we focus on that, looking at some of the socio-technical aspects of that. I've got like twelve, thirteen, fourteen chapters. So about the technical, and then the last three chapters will round out that side of things. So like what we do to make sure we have that environment of psychological safety, because without it, you can't actually be resilient.

Speaker 0: That's very amazing because so far throughout this whole conversation, we've talked about building resilient systems. We talked a lot about people, about how to think about resiliency. One thing that we've simply talked about just in passing is AI, even though like this book is coming out now, and I know it has a section on AI as well. But it's just interesting to reflect that even though like AI is changing how we're building software, it seems like some of these ground truths of how to operate reliable software are a little bit just outside of this realm, right?

Speaker 1: Well, I mean, is it? No, it's not. I mean, at the micro level, there's some stuff, there's tooling and stuff that can help. I detail quite a few examples where root cause or correlation analysis and tools help support around post-mortems, and intended automation is doing amazing stuff in this area.

Speaker 0: Dealing with a lot of probability?

Speaker 1: There's more fundamental kind of things. And the first thing is the micro and macro level. At the micro level, they're useful tools, we can talk about that. But at the macro level, none of it stacks up financially, right? So if you're taking a longer view about resilience: if I'm building my products based on a market sector that is fundamentally flawed, these companies don't have a viable path to sustainability. When you look at OpenAI and Anthropic, and the hype cycles, conservative analysis by The Economist showed that they need to be generating $2.6 trillion in revenue to justify their CapEx expenditure by 2030, while the entire software market is $1.4 trillion.

Speaker 1: So at the macro level, I don't know how that gets sorted out, I'm not smart enough. I also don't have the shares of these companies. But you know, that's capitalization, I can't solve that. What I can start thinking about is: okay, that's the environment what I'm operating in. How do I deal with that? I can't deal with all of that. What I do if I'm taking future resiliency into account is being very careful about how I use AI.

Speaker 1: For example, you know, if I'm thinking and taking a longer view, being more flexible about model choices, that's going to help me because I've got the opportunity to use open-weight models from different vendors. Just basic things like OpenAI and Anthropic are pretty terrible when it comes to the availability of their core models from their own services, which is why even if we use Google or Amazon or Microsoft as their primary place where they host these models, they have better uptime than things. So there's all that stuff we've got to deal with at that level.

Speaker 1: There was a conversation about non-determinism, and Charity Majors caught me on this a bit. And I said, the thing is that, you know, we're going from deterministic software to non-deterministic software. You know, that's a challenge for us, for engineering work. And she said to me, our distributed systems were never deterministic, and now they're really not deterministic. And of course, she's right. And now we take our distributed system, and we add people in beyond the retries and the failures, we always had a degree of non-determinism, we're just amping that up a bit. Yeah.

Speaker 1: So I think that's a continuation. So for me, I think it's a confusing landscape, not good or bad. I think on a macro level there are real issues, whereas on a micro level there are really great things that we can do.

Speaker 0: That's the tactical things that we can do, right.

Speaker 1: I think really what I mean by that is I'm taking the macroeconomic situation into account, and I say that there are things that we can make use of. There's some good stuff in that that we can make use of. But I also think we need to do long-term hedging. If you are building AI into your product, there's a simple hedging thing: being multi-vendor, being multi-model. For example, rather than having a single agent, breaking that down into more workflow models. Therefore, you can take bits of your workflow and realize actually, this could now be swapped out for a bit of deterministic code, written efficiently. I think these are all sensible things that you can do. We can't fix the macro world, but we can at least deal with our own world, and I think there's sensible things that we can do in that space.

Speaker 1: What are the things that we know that we do not know about AI? I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is, let's be real. Why did the LLM delete my database? Well, it has no concept of causality. No, that's what it comes down to: it has no concept. Of course, like there is a glass here. If I push that glass onto the floor, you'll work out if there's carpet down there, good. If asked what's going to happen, you say, the glass won't break, it may be fine. If I push it onto that floor and it's a hard floor, that might break. You're carrying a bunch of calculations in your head. You've got an idea about the physical properties of the universe. You've got a sense that if I push the glass, this is going to happen. LLMs don't do any of that.

Speaker 1: They have no concept that if I do A, B happens.

Speaker 0: Yeah. Well, there's a push for world models, but these LLMs are not world models.

Speaker 1: Causal models have been around for a long time, they predate LLMs. They can do stuff like that. So they could make decisions like, if you have the appropriate model and appropriate inputs, they could at least get a sense of: if I do X, then Y happens, and I've been told not to do Y. This is why guardrails around LLMs are really not, for a long time, going to be a complete solution. And so I think we expect LLMs to do more than they can actually do because they seem so smart, yeah. And so I think that's the problem when that leads to overestimating their capabilities a bit too much.

Speaker 0: I think so, or maybe over-relying on them without knowing, and we might have severe incidents later actually.

Speaker 1: Absolutely. And I think we need to know when and how to use those things. And I think what we don't know right now is: where should I have the LLM do it, and where should I just write some deterministic code? There's a lot to be said for just having the agent call a script where you know that script works, rather than having it implement the script in its magical brain for you.

Speaker 0: Well, there's something to be said that it seems both leading labs, Anthropic and OpenAI, have realized that coding is so important. So oftentimes, when you ask it to do something, it will start generating code because it tries to write a deterministic program, because that part will be deterministic. It's very interesting. When you ask it, people often find it pumps out Python or TypeScript or whatever, especially when it has to do with comparing this or that. And they have trained these models to default to coding. I'm not sure how this came about, but it seems to be working to some extent.

Speaker 1: It's all very testable, right? You know, they are trained to generate plausible code, just as they are to generate plausible words, right? And in the context of code, they generate some plausible code that we can now validate: does it execute? Yeah. And so that's a really nice set of tractable aspects, and there are some places where it works really, really well.

Speaker 1: but the other rate is way. it's terrible like I write my book. I used very little of it. Don't let AI at it at all.

Speaker 0: well, you don't use it for the writing, use it first.

Speaker 1: and you just not using it for writing at all. I use the fact that we are doing the research. But in the context, the research is, if I used NotebookLM basically, but in that context, what is plausible? But then who's going to click the links, it is me. And I say, was that there or not, I am checking the work. It has done way more work than I could have done going through all those papers. And God knows how many missed papers. You can read through it, right? But in that context, I feel completely fine. Unable to do that in other settings. You get into what is practically cognitive surrender, which is where using AI to write documents that you're never going to read. That's really, really a big problem with the LLM technology. That's the problem with us.

Speaker 0: okay. We talk about this cognitive surrender and cognitive debt.

Speaker 1: yeah, cognitive debt has kind of been around for a while that sort of explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools. There's been a bunch of research into this, there's been some studies that showing that people's ability to think critically is impacted the more they use chatbots and things. And I think we've also seen other examples that Margaret-Anne Storey has got a great article about this, and she cited some of the research around it as well. And that's explaining, I mean, her thing is what she's talking about. We talk about technical debt building up in the system. We've got to be careful of our use of AI, and it's a cognitive debt. We need to have a shared mental model of what this system is supposed to do, thinking about it narrowly through the lens of a program that we're building. And if our use of AI, for example, we go off in different directions, that shared bridge breaks down, that becomes a cognitive debt. Cognitive surrender is a different thing, which is like, "Oh, yeah, fine." It's basically LGTM, right? Uh,

Speaker 0: but what the AI tells me, yeah. LGTM was typical, "Looks Good To Me", approving large pull requests where you cannot be bothered, or you tried to look at it, and for whatever reason you didn't want to go through, or it came from someone who you rubber-stamped.

Speaker 1: I wasn't able to tell on all that stuff, right? And so LGTM was literally a sound pattern match on and takes as an approval. So you say LGTM, that person's approved, right? And that was it. That was where I think that maybe economically came from. Not sure.

Speaker 0: I think you can never, yeah, yeah,

Speaker 1: but it is this idea that it has become a shorthand. Does AI look at it? Yeah. And I think that psychological surrender thing is coming out in lots of ways. And you know, we're getting AI to write documents and no one's going to read. We've got AI that's reading documents that weren't read by other people. And so the cognitive surrender piece is really that the humans—it comes back to the human being in the loop has very subtle differences in those workflows into the current differences. It changes like—there's a big difference, like code doctor talks about. It's a big difference like if you are say a cancer oncologist or a cancer specialist. It's a big difference to use oncologist working and LLM doing sort of detection of similar scans. You're looking like done. "Uh, you should look over there. By the way, you might miss that." That's actually quite useful, right? Versus the LLM-powered AI going, "Here was scans I've looked through. Here's the one we need you to sign off, right." The last one, the second one, right, takes the expert out of the loop and makes them an approver. And often in that context, they've got rid of a bunch of the oncologists. You've got one person in the final loop. That's going to lead into issues, right? The other model is we are making people better. I think that's come back to the original idea of AI. Also, the pitch, right, was it was going to free us from drudgery, no, just like the Industrial Revolution was going to free us from drudgery. But like it would elevate us, we'd have less toil. We can be this. We can focus more on critical thinking and smart. And so much of our current use of AI is not freeing us up in critical thinking, it is causing more context switching. It's causing less time for thinking. People are working longer hours. We know because studies show this now. That's not necessarily AI's fault. It's how we're using it, it's like we need to address that a little bit.

Speaker 0: can we talk about complexity? AI, maybe adding it. I'm going to quote you from about it from a few months back you wrote: "I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify, at least in the platform space."

Speaker 1: yeah, can't remember the exact context in which we had that conversation. But I mean, platform space was all—I mean, platform space is kind of interesting, like because that was an attempt to make things simpler for developers because Kubernetes became popular. And it's not developer-friendly.

Speaker 0: that's not always designed for them, but most developers have found it such a nightmare.

Speaker 1: I mean, well, if you have a platform, don't have a platform. Yeah. Yeah. There's a platform team's problem, right? And that's what they get paid, they do what they get paid for. But and so I think there was this idea that I was sort of hoping that there'll be a bit of a complexity collapse around that space. I had this hope the things like Wasm for example seem to simplify down that programming model a little bit. Um, and I think there's starting to be a bit of a conversation trying to be had about some of that stuff going on. What then happens, of course, as AI has come out, and it was like, "I've got to AI all the things." And so rather than thinking about what we need and try to strip complexity back, we've now just got another piece that requires—it still have all the old complexity, adding new complexity on top. And somebody has got to look after all that stuff.

Speaker 0: but I think it goes back a little bit of like we seem to have more work with AI. And probably doesn't help that they add more complexity, especially when we're not being clear of what we want, we're not being critical of the output when we have cognitive debt or cognitive surrender, especially of a cognitive surrender. You know, "it'll just be right." So these things might be amplifying the fact that there's just more and more work, more complexity, right?

Speaker 1: yeah. And I think there's also that connection through to what the alignment is doing. Sort of comes full circle a little bit, which is the idea of potential. Not looking at the code isn't to say that you don't care about what the system is that you're building, it's to say that you've got a better way of understanding what you're building. So one of the concerns is, if I'm using AI to create, because it's going to create awful loads of debt and a complicated system that no one's going to understand. Well, we can't let that happen. We still need to remain connected to the system that we're building, which is why I'm just a big fan of like modularizing systems. Whether or not you put those modules on different computers, I don't actually care. Getting AI to work within the modules. Having the humans think carefully about what these modules are and the connections between them. Let the AI go hog-wild inside. Those boundaries are easy to change, the gaps between them. That's where the human beings need to think about the mental model of how everything fits and slots together. That was a very sane way for us to balance our time, our energy appropriately.

Speaker 0: so to summarize, that probably a sensible approach is to have the orchestra in your head, know how your system works, know how the boxes work, know where there's AI inside it. Maybe it's generating code. Maybe it's running your incidents response in isolated ways, and then spend your time talking with other people in the business, on your team, with your customers, because you're—as you said, systems like software is people's ideas connected. And you have things in your head, and you write it down. But now using AI as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions because maybe you have more time to, as an engineer, understand more on the business on what is working, what is important, those kind of thing.

Speaker 1: yeah. And also to make the structure and the model, this is a bit more explicit as well. I mean, that's one thing a good modular structure does, and it is why Shopify, for example, have been so successful at keeping their things a monolith, right? Because they spend a lot of time building the monolith application as a modular architecture, right? Because it makes the structure easier to understand.

Speaker 1: You can zoom out and see how those things fit together, but also in the short term, while we're having these debates about are we going full software factory or not, we don't have to make a one-size-fits-all approach. We can say, look, this is our architecture—be it a modular monolith or microservices, architecturally it doesn't really matter. I know that over here, this area is where we're trying the full software factory approach where we don't look at the code anymore, but we specify the interface. We have verification around that. We know what good looks like for that module, right? We've got functional tests, and we look at say some of the operational requirements as well.

Speaker 1: But these areas over here, we're still wrapping. These are really key, really important to us. We're not ready to give up on this yet because we don't have the trust level yet. We don't want to take that risk just yet. And so over here, we're going to work with the AI in more of a collaborative way. While we work together, we stay involved in the code because we're not ready to make that step here. But if we approach our whole system as one big black box, then we get forced into making an all-or-nothing decision. Continuous learning and incremental steps—that's healthy.

Speaker 0: Do I understand that? You're saying that you think that the software factory or the dark factory—where you give it a spec and the AI is building it or operating it—that you see that as viable?

Speaker 1: There are some companies already on this, but not everyone should be doing it whatsoever. I think if that is our future, that is our lowest-risk starting point. The question is about how you go about adopting and trying it. What I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way and in lower-risk environments.

Speaker 0: And maybe simple domains, those kinds of things, right?

Speaker 1: Exactly. Also, a lot of dark factories don't create finished products in the real world. Beyond the fact that in the real world dark factories aren't many, most create components, they don't create the finished product. Yeah. So in the same way, if you think about a module being a piece of the overarching system, right? Well, let's have a factory on these bits here. And actually, looking at it, like if we had to rewrite that from scratch, it might take us—with the collaborative way with AI—a few weeks to rewrite. If we had to, we'd actually be quite relaxed about it, even leaving it as is, and it's been working fine.

Speaker 1: And so it's a module around which you've got great test coverage already, for example, or where it's not on the critical path from a performance or latency point of view, or it's a service where you've got really good observability data and really well-defined SLOs and a good steady state. Well, we can start running it as a factory now and seeing what's happening in production. We could have a look at the SLOs. We maybe add some special SLOs around there.

Speaker 0: I hear you on that. You're saying that as a professional, someone who wants to be up to date with the new techniques that we can use, if you have the opportunity to start to build a software factory—a somewhat autonomous place in a component, in something lower-risk—it's probably worth trying it out to see how it works, to see what you learn, to see if it even works or where it breaks down, but do it in a way where you are minimizing the blast radius. It's a little bit like when microservices came out: the concept wasn't to start by moving your biggest thing; you start by building a small service that didn't really matter all that much, and you were okay to learn from it when it goes down, and it wasn't on the hot path for the business.

Speaker 1: Yeah, I think the thing to start off with is getting your head around the spec. So looking at spec-driven development—whether that's Tessl or Spec Kit, whatever that is at different levels, right? Have a look at GitHub Workspace or tools like that as a good on-ramp to get ahead of it. Start off by playing around with that because that's the stepping stone. You have to define what good looks like as an input to your factory, so start with the spec as input.

Speaker 0: And also for the output.

Speaker 1: Yes. And then it's like how you verify—how are you going to go about verifying the output of the factory? You have to think about that, and not just in terms of the normal behavioral requirements. I agree with folks who hate "non-functional requirements," because if something is called non-functional, it implies it doesn't function. So he breaks it down into development requirements, operational requirements, and business requirements. I think it's nice doing that.

Speaker 1: So across those dimensions: how do I verify that the software being created by this module has matched my developer requirements, such as maintainability; my operational requirements, like latency and uptime; and my business requirements—is the feature working? So as long as you've got a way of validating the software around that. And if you don't know what good looks like for those three things, guess what, it isn't working. So you can write specs, but you can't run a factory. There may well be some points of analogy here—this could be Model-Driven Development all over again, which we do not have time to get into and get very angry about. But it could be that; however, I have hope it might actually work this time, unlike MDD.

Speaker 0: And then as we're closing, I think a lot of what we talked about is that it's important to hold the architecture in your head, to think about architecture, and to think about how to modularize. Now for engineers who want to get better at this kind of high-level thinking, strategic thinking, and architectural thinking—outside of reading your book, which I think brings a lot of really good ideas on architecture as well and not just on resilient systems—outside of that, what would be your recommendations for professional engineers who want to get better at architecting systems and know what good looks like?

Speaker 1: I think there's a few books I can point people to. I think Fundamentals of Software Architecture by Neal Ford and Mark Richards is good. To be clear, they're friends of mine, so I'm not unbiased in recommending it. Around modularity, Vlad Khononov's book on Balanced Coupling is fantastic. His DDD book is also very good, but that Balanced Coupling book is really great. I think he's also developing skills and coursework to take some of those ideas forward, so definitely take a look at those.

Speaker 1: And I think if you're looking at things architecturally and thinking in terms of modules, for me—while Martin Fowler's book has lots of great stuff in there—for me, that's where you start. The edition of Building Microservices was really all about smuggling in getting people thinking about information hiding. A very old idea from the 1970s that still comes back. So to me, modules are still the whole thing. I would read Vlad's book, and if you want to go read Parnas' original papers from 1971 and 1972, they are still worth reading. Those would be good jumping-off points.

Speaker 1: I would say the nice thing about thinking in terms of modules is even within an existing codebase, you can start to modularize it in some small ways. You can start to think about bounding things, about clear separation, and using encapsulation to tuck information away.

Speaker 0: And I guess it's fair to say: look, if you have an idea of how you can make your existing codebase a bit more modular, you have zero excuse not to do it. I mean, we don't trust agents fully, but this is a textbook example of where if you can say, "Here's my goal, refactor this," where if it would have taken a huge amount of time by hand, it is vastly faster now. So you should be empowered to experiment with this.

Speaker 1: Especially if you've got a decent test suite. And if not, start with that. If you've got a statically typed language, even just a good old-fashioned IDE does an amazing job, right? In a dynamically typed language, now you don't have the excuse either, especially as modern ones are well understood by the LLM. So if you're using Python or Ruby, it's so much better now than it used to be for refactoring those codebases. Absolutely. You can start testing your way around that and start modularizing. And that goes back to making sure the architecture isn't only in your head, right? We want the code to represent that mental model. Yep, because we communicate through the code, and maybe in the future we'll be communicating through specs. When you make those things more explicit, I think that helps everyone stay on the same page.

Speaker 1: There's a George Fairbanks—forgotten the name of the book now—but in it he has a pattern called an architecturally evident coding style, and that we should code in such a way that the code communicates the architecture of the system. And I think our programming languages are quite bad at having good conceptual models in general. So I think we have to go a bit beyond ourselves, but we could do more to make the architecture of systems that little bit more explicit. This is also why I spend time in production, because especially if your module structure ends up as a process structure, otherwise it's process boundaries. I can go look at the network to find out my architecture. I shouldn't have to, but at least know production is truth, it's like football is life.

Speaker 0: Production is truth. Where can people get your book? Can they go order it now?

Speaker 1: Um, if you can go buy it from a local bookshop, that was a great idea. It's also going to be available for people on the O'Reilly platform in sort of digital form, so you can go and read it as part of a subscriber. Everybody can get it there as well. But lots of people love having the physical copy.

Speaker 0: And we have these in the links, but also check them out to the book directly. And also, you've been doing some videos, right?

Speaker 1: Yeah. Well, I've been doing some stuff on Dave Farley's Modern Software Engineering channel. It's kind of a good excuse to do some semi-rant, you know, monologue-esque videos and have some fun chat. So yeah, go to the Software Engineering channel on YouTube, and you can find videos from me and Dave Farley, and Kevlin Henney is there, lots from Kent Beck, Trisha Gee, Dan North is there as well... I'm pretty sure I've forgotten somebody else. There's another couple of people as well, but we try and pull videos together once we can, it's a bit of fun.

Speaker 0: It's a great channel, we also like that one, yeah. And if people would want to work with you, where can they find you, how can they find you? What kind of stuff can they get help from you for?

Speaker 1: I do internal talks for people, like internal events. I do that a lot. My training courses cover a whole variety of things around distributed systems and security and microservices. I also do a lot of consulting for clients, helping them around rearchitecting their systems, stuff like that. To know more about that, go to my website samnewman.io, and you can contact me there. You can also find me on LinkedIn. But if you come straight to the website, use email—I'm always a bit old-fashioned like that. And I work with clients all over the world. So if I'm able to help, yeah, head over to samnewman.io, my contact details are there.

Speaker 0: Wonderful. Well, Sam, thanks again, this was great. Thank you. Thanks very much.

Speaker 0: As for this conversation, I really liked Sam's three rules of distributed systems: One, it takes time. Two, sometimes the thing you want to talk to isn't there. Three, resource pools are not infinite. Sam said how most of the outages happen because of his last one—resource pools not being infinite and services are getting saturated. But I was chuckling when he was talking about the rabbits chewing through networking cables between office buildings. Then again, a few weeks ago with thousands of flights stranded down the east coast of the US, it was because workers accidentally cut a telecom cable in New Jersey used by the FAA.

Speaker 0: Another thought from Sam is how failure should be handled differently depending on the type of business. An e-commerce shop might decide to show you the item when the inventory service is down. Worst case, after you order and pay, they just refund you because, look, we don't have the item. Basically failing open is fine in this case. But on a concert ticketing website, you don't want to sell a ticket that might or might not be there, and so failing closed is the right call to make. Because again, as discussed in the episode, if you sell the ticket, people might book flights, they might get upset, you might lose them as customers.

Speaker 0: Finally, I appreciated how level-headed, perhaps even cautious Sam is about AI. He's got no doubts that AI is very useful at the micro level, basically tactical at things that we can already approve, but he is worried about the economics of the model providers. His suggestion is pretty pragmatic: hedge your vendors and your models, multi-vendor and multi-model, and wherever you can, swap non-deterministic AI agent workflows to deterministic sequences—basically have LLMs generate deterministic code. I find it very, very refreshing thinking. Check the show notes below for the story about scaling Uber, the early days with Uber former CTO Thuan Pham that we discussed, and it's an awesome story. And for more of The Pragmatic Engineer deep dive on resilient systems. If you enjoyed this show, do subscribe in your favorite podcast player, and thank you very much if you recommend the podcast to your friends or your colleagues. Thanks, and I'll see you in the next one.