Host: What do you think really made Kubernetes breakthrough?

Kelsey Hightower: The number one success criteria was Docker. Now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers. We were already off to a running start because you could just reuse the same Docker containers. And I remember I get this email from Satia, the CEO of Microsoft, and I'm like, man, he wrote this nice email and I open a PDF and there's a zero get added to the equation. And so you're looking at this like, I didn't even know that they do that. We know that it happens, but the person that graduated from high school in 1999 that chose the A+ certification didn't know that was available. So, I was serious about going to Microsoft. I'm not just like a Genai hater. I just don't like the naive promotion and adoption of it. I think it should be way strategic. And since I think about Genai as a tool versus the great human replacement, then I can use it in way more primatic ways. And when AI is involved, the one thing I just do before the thing kicks off in this meeting, do not say AI because Kelsey High Tower is known as one of the most influential voices in the Kubernetes community, but you wouldn't guess from how his career started. At 19, he dropped out of college to be a DSL model installer, became a self-taught developer, and still went on to become a distinguished engineer at Google. At age 43, he then retired at the very top of the industry. Today we cover Kelsey's unconventional path into tech and how he kept creating new opportunities for himself often unknowingly. The inside story of the container wars, puppet, Docker, Terraform, Coros and how Kubernetes eventually won. Going from a Google IC to executive level and how he rejected a Microsoft offer from Sachio Nadella himself and still doubled his compensation. His grounded on pragmatic advice for software engineers worried about being commoditized by AI and so much more. If you're an engineer thinking about your long-term career trajectory, whether that's getting into a staff plus level, going independent, or even quietly planning to leave the industry, this episode is for you. This episode is longer than a normal episode, frankly, because I was so glued to my chair, mostly listening to Kelsey's stories and thinking. This episode is presented by Anticys. Verify your systems correctness without human review or traditional integration tests and avoid bugs or outages. Before we start, I'd like to mention our presenting sponsor, Antithesis, and maybe offer a little history lesson. Over the last two decades, software development has gone through a mindset shift from an imperative approach to declarative one. Infrastructure is a perfect example. Think about how tools like Puppet and Ansible allow declaring how individual servers should be configured. Then came Terraform, the ability to declare the desired end state of your whole infrastructure, servers, networks, databases, and their relationships. And then with Kubernetes, we stop scripting container life cycles. Instead, we write manifests that say things like, I want the replicas of this application exposed behind a surface with this much CPU and memory. Once we didn't have to specify every little detail of our infrastructure anymore, deploying software became much faster. But then the bottleneck became how quickly we could test and verify the code to be deployed. Testing remained imperative. We had to write tests for every little detail. And now with LLMs, we're on the verge of a declarative shift in the way code is written as well. Just tell the model what you want and let it figure out the details, and it's going to make the verification bottleneck a million times worse. Anthesis is a declarative testing tool that can keep up with your AI coding agents. You state the properties you want your software to have, and antithesis figures out how to check them for you. Verify your code as fast as agents can write it and ship with Growlin confidence. Head to insys.com/pragmatic. Kelsey, welcome to the podcast. It's so nice to see you in person.

Kelsey Hightower: Yeah, I'm actually happy to be here mainly because I kind of look at your stuff over the years. So, it's honor to be here in Amsterdam as well.

Host: How did you make your first dollar at a job?

Kelsey Hightower: Oh, my first dollar at a job McDonald's, right? That counts. So, in high school, you get the job that's closest to you. So, it was in walking distance of my house. As soon as I turned legal age, uh 14, get a work permit. And I went there and it was one of those jobs where, you know, you go, you fill out the application the same day. You typically get your information or you're going to get hired the same day. When can you start? I'm like, right now. They go get a shirt for you in the back. You What size do you wear? Uh, men's large. And the one thing I liked about that job is you're dealing with real people that are in a hurry. I guess one bad part about the job, you know, a lot of people don't respect people who have that job. So, they kind of look at you as just like this intermediary thing between them and what they want. But there's so many things that go into a restaurant like that. It's very efficient. Um, you know, people have expectations. And I learned how to run the whole store. So, by the time I turned 15, I was a assistant manager. So, nights and weekends, you know, other managers would leave. They would give this 15-year-old the keys. And I knew how to do everything there, including close out the store, right? So, you have to count all the money. You have to fax this huge spreadsheet to corporate every night. And then my mom would pick me up. And so it was really good learning how to really be responsible even for adults at that age. So that's how I got my first dollar.

Host: How did you get into tech? How did you get into programming

Kelsey Hightower: in high school since I moved from California to Atlanta? So right, you're going from one side of the country to another side of the country. And uh I missed maybe 3 to 6 months of school and in order to graduate on time, I had to take some extra classes. And so as someone who played sports, ran track, uh played football, played basketball, and it's like, you know, there's this computer programming not, you know, computer club, technology student association. There was a class component and then there was after school component. And I was like, I don't know, man. This computer stuff that's for the, you know, you know, I'm trying to be a cool kid. But the one thing I did, I really enjoyed it, right? So I I had a liking to AutoCAD. I even competed at the state level in AutoCAD. So we we drove down and you compete. They give you a task and you sit in front of the computer. It was my first year doing it, but I really like the idea of like taking a specification, designing it, and I probably would have gotten first place if I would have got the product to work because you also have to print it out so that the judges can review your work. So I got second place even though I didn't.

Host: This is the 3D modeling.

Kelsey Hightower: Yeah. AutoCAD, you know, just AutoCAD. Yeah. So part of the curriculum was, you know, you build bridges. We did this thing with um chapter team where you know you have a code of arms and you're kind of doing like a debate. Do you kind of learn all of these things uh related to business but CAD was one of the things I like most? Also in that class one of the classmates taught me TI basic.

Host: Is that a version of of basic?

Kelsey Hightower: Well so TI basic so you know the graphing calculator it's like a TI86. Oh yeah.

Host: You can program it.

Kelsey Hightower: right. So in class they're like hey you know it's not just a graphing calculator you can actually program it. And I was like what's that? And it's like look we can you know every at that one at that time you would create the snake game right so it's basically get a magazine copy and paste the code and then you run it and now you're playing snake uh based on the code you wrote and so you would toy around with this concept so that was probably the first introduction to programming was literally programming my my TI86 calculator

Host: and after high school did you go to college or you considered college right

Kelsey Hightower: I considered college because in Georgia at the time and still today there's a thing we're called the hope program. So if you have a B average or above, you can go to any public school for free. And public schools in Georgia include Georgia Tech, Georgia State. These are pretty good universities. They're really good. And so I decided to go to one that was near me. The first two weeks I was like, um, this is too slow. This is not this is not the pace that I want to move at. And also remember it's 1999 when I'm graduating. And so when you turn on the TV, people are standing in line for the next version of Windows. There's a lot of euphoria. AOL is starting to phase out and we're starting to touch on highspeed internet and and I was like, yo, look at look at the pace this is moving, but also you're hearing the narratives. Bill Gates drops out of college. These people are not necessarily glorifying the degree anymore. It's all about the skill. Now, unfortunately for me, I didn't know anyone that was a programmer. I didn't know anyone that was like a system administrator because at that time all the systems with like Sun Micros systemystems or IBM mainframe those are still the things that are in the enterprise. So when I looked at the job openings I'm seeing a bunch of skills that I don't even know how to acquire. And so instead of going to college you know I'm still doing fast food delivering pizzas at this time at Pizza Hut. And I remember going to a bookstore and they had the A+ certification guide and I looked and some of the job postings said, "Hey, you need to be A+ certified to take this support role or whatever it was." And I was like, "You know what? That doesn't require college. The book is only $35." And I remember buying the book and it is an official certification process. It looks like it was part of the job market. And so I remember buying that book and reading it cover to cover over and over again. and you're learning all the fundamentals, right? You're learning about, you know, how motherboards and how memory and all these things work and there's an OS component and then there's a little practice exam in the back. And so for someone like me, having that fast feedback loop of like you put the CD in, you take the exam and even though it was multiple choice, you kind of felt like if I got anything wrong, I would just go back to the book and make sure that I understood what was written there and then you go take the test again. and it had a little randomization to it so you couldn't just rely on absolute remembering everything. And I remember going to the facility to take the test and you know you're in that little room and they want to make sure you don't cheat. So there's a camera pointed at you and you're just going through it. And so the nice thing about those tests, there's no trick questions. Either you know it or you don't. And I think they maybe give you an hour, hour and a half. And I remember finishing that thing in like 10 minutes. And when I walk down, you know, you wait, the dialup goes and they calculate your score and say, "Hey, you passed." And then you walk out and you're like A+ certified. And that was like the first time in my career that I felt like, oh, so if you put the effort in, you can gain the certificate. And when I got that certificate, I remember there was like a job fair where, hey, anyone that has A+ and network plus certification, you can be part of the contractors that were replacing people's dialup with DSL at the time. Okay. And so that's how I I guess officially got into tech.

Host: Is it fair to say that you saw that this could be the most efficient way to get into tech at the time?

Kelsey Hightower: I think I said I saw it as the only way.

Host: Why was college never like telling you like, okay, that could be a way. Was it just you didn't see examples or

Kelsey Hightower: I Yeah, I never saw the examples. I never saw the endgame. A lot of the stuff that they were teaching the curriculum, it didn't make sense that you would pay all that money. You know, look, maybe it wasn't a good school. Maybe it was the wrong class that I took. There's so many factors that could have went into this. But when I looked at it, none of the people that at the time that I was looking up to, this is not the path that they seem to be taking. And so I had enough of school, right? If you're 18 at that time, you're like, look, that that's enough of this. Because at the time, I kind of felt school was this because it was so easy for me actually. You know, it's like it was easy to get straight A's. I didn't feel like there was a serious challenge. So it's like, hey, I want to go and do four more years of this. And I would later learn that look bachelors is a lot of the similar that you go through through K through 12, you kind of remember stuff, you listen to the lessons, but then masters you challenge the material and of course if you make it to PhD ideally you're adding something new to the field and I never saw anyone that has made it that far. So I never put that in part of my calculus. So, but just having that immediate feedback loop of like getting this A+ certification and feeling like, oh, I'm ready to participate in the actual economy, the ecosystem. So, to me, I was like, this seems like a better path. And it felt like a path that I would control.

Host: Yeah. And then what was your first job that you could get with with the certification? This was the comta, right?

Kelsey Hightower: Yeah. So, at that time, Bell South was, you know, the biggest telco uh probably in America. you know they had been broken up by that time from the AT&T days at that time the people who did phone lines right so those are the official Bell South technicians they drive the fancy trucks they have all the equipment and when they made the shift to highspeed internet that means you had to actually touch the computer and I think as a union they're like look we don't touch the computer we don't even go into the house we we get to you know the demark and we stop and so they had contractors come in and the contractor's job were to come in, have to do a little bit of wiring. So, if you had to run some cable, you did that. Create Cat 5 cables, you did that.

Host: Ooh, yeah. You you I one of my first jobs was actually cabling. So, I still I forgot the exact color code.

Kelsey Hightower: Orange, green, white, green, blue, white, blue, something like that. It's burning in your head now. And you know, so you did whatever it took. And the other thing you had to do was you have to open the computer. You have to make a decision, right? If they had a new enough computer, they can use a USB modem. Those were terrible. They always broke and you would always come out for a repair. Uh but for a new install, if you really wanted to do a good job, you install a a nick, right? Uh Cat 5 port on the back of their computer. And at that time, like we're talking Windows 98. And so usually, I don't know, 20% of the time as you're installing the drivers, the computer would crash. And now you have a whole another situation. You have to now troubleshoot getting this thing back online or back operational. But if everything went smoothly, they now had a network card and then you had an external modem that then you connected to, you know, the phone line and they had this high-speed internet connection and then you connected the network cable. And I did that for about let's say a year and then I started doing the businesses. So you go into people's homes like you're going door todoor and then when you go to a business you would hook up one computer but there's obviously eight computers there and only one of them has internet access. And I remember at the time, you know, the business owner, it could be like a small insurance company and they would say like, "Hey, how do we get all the other ones online?" And the first time someone asked me that, I'm like, "I I don't know, man. I don't." We put it on one computer like pay grade, right?

Host: Yeah. We make sure it works. But then I decided like, "Well, let me go learn." And that's when I remember like going to like Office Depot, right? They sell computer equipment and things like this. And I went in the store and I asked him, I was like, "Oh, you can get one of these Lynxys routers, right? infamous blue spaceship looking Lynxis router.

Kelsey Hightower: I still remember them.

Host: And those things were probably like 50 bucks. And I remember just buying one and figuring out how to get multiple computers to to use one connection. And so eventually I was like, look, I can't do it as part of the job because we we have to do this and we have to leave, but here's my carp. And then they will call you. And I remember one of the first installations that I did on my own. Uh they wrote me a check and I was like, "Yep, you could just write it out to Kelsey High Tower." He was like, "No, we don't we don't write checks to people. We write checks to companies." And I remember right there on the spot, I'm like, "Oh man, I need a company name." And I just made one up digital gateways. And they wrote that on the check and I'm sitting there like, "So, how do you cash this?" So, I went to the bank and they're like, "Sir, you have to get a business license. The business license, you could just do business ads. You have to do this." So, I'm figuring out now I'm 19 years old. Like, okay, I got to go get a business license. Have to figure all this stuff out. So, I do everything. I open a business account just so I can cash the check. But at that point, I'm like, "Oh, this pays more than this does." And so, I got really good at doing those network installs. I really got good at troubleshooting cuz sometimes someone gave them a USB modem, lightning comes, the USB modem is fried, and then you would swap them out for a network card. And eventually, I decided like, I can probably do my own business. And I decided to get some office space. So, I opened a small computer store right outside of Atlanta. I would buy parts from the distributors. I was just like 19, 20 years old. And I wasn't buying enough to really qualify for an account. But luckily, one of the smaller computer stores I used to buy parts from gave a recommendation to the distributor. It's like, hey, this is our guy. He's just getting started. And they gave me an account. And I was able to buy motherboards and GPUs. And people would come over and like they bring their kid and they would have a parts list. I want a computer with all these things and we would assemble machines and you know sell them but also it was the headquarters for all the other service calls. So I did that for like three, four years, you know, that ended up evolving into at the same time or now were talking like 2000, 2001, a lot of the music studios were moving from analog gear and the large mixing boards to ProTools, right? The little rack mount unit. And they all needed max. They all need these conversions. So I added that to my uh abilities. And then I had a small setup in the store and artists and musicians would come in and say, "Hey, we want exactly that in our studio and I would get the order and I added it to the list of things I could do."

Host: I mean, at this point, you now have a small business. Sounds like it's it's going well. And suddenly you take a job at an employee job at Google when I look back, how did that come up up to now? It's it's almost like this is like, you know, the story often times will continue. You become an entrepreneur, you grow your business, you you know, you just take it from there. Even during that time as that store owner, I managed a comedian and we went on the road. And

Kelsey Hightower: so you were you were helping out your your managing

Host: I had a buddy from high school. He was a comedian. Turns out he was actually really good at it. We even I went to go see him at a club. He's like, "Hey, I need a manager." I was like, "Are you even funny? Like I know you from high school, but I don't remember you being like funny enough that I would pay to see you tell jokes." And um he was like, "I have a show tonight." And so I gave him a ride on the north side of town. And the interesting part, it was a predominantly black audience in Atlanta. Okay, makes sense. And he did these jokes and they laughed and it was one of these comedy clubs where if you're not funny, you have like 3 seconds and they would just boo you off the stage and that's the end of it. And he held it. I was like, "Wow, you survived that. That's um that's incredible. And you were pretty funny." And then we drove about an hour north and the audience is predominantly white. And so on the drive there, I'm like, there is no way you can do those jokes in this room. I got to see how this is going to go. And at the time, they're only paying the comedians like $50. So, you don't make a lot in the early days. And he totally pivoted the set and he held that audience, too. And I was like, "All right, I can be your manager. I know business. I know kind of logistics. I know how to, you know, make a plan together." And I did that for a number of years. and you know he won some televised competitions. We went on the road bands like Earthwind and Fire and some large comedians from Kings of Comedy and Queens of Comedy and I actually picked up some IT work with the company behind it called Letham Entertainment. they had been doing movies at this time. And so now I'm like, you know, doing it for them and I got the small business. I got the comedian. And so, look, I was able to save a lot of money, but man, luckily I was 20 years old cuz I had all this energy, but I was working quite a bit. Eventually, you settle down, you get a family, and you do the math, and it turns out people kind of over glorify entrepreneurship. I think a lot of people believe there is tremendous upside, right? the type of entrepreneurship we talk about with software companies, the the upside is crazy. But when you're doing like selling parts or service business, unless you plan to open lots of stores and you know grow a larger employee base, it's not the same growth trajectory as software companies. And so I kind of did the math after four years of doing that. I said look, I want to settle down. And if you've been an entrepreneur before, you know employees get paid first, the owner gets paid last. And there are months where you get paid less or you don't get paid at all and now you're kind of drawing from savings because it's not their problem. But I did well overall like I did very successful. But I remember it's like you know what I think I'm ready. And so I looked around and Google had data centers nearby and I felt like I had a great combination of skills. I understood you know the racking stack part of the world. I understood the physical part of the networking stack. I understood everything from Linux to Windows. I had an entrepreneur mindset. I didn't think there was nothing I could not do. And I remember going to that interview and they were hiring like data center technicians. What it paid? And in my mind, I'm like that you only have to work for 8 hours. You don't have to issue any invoices and you get paid every 2 weeks no matter what. No inventory. This is crazy. No way they're doing this. And so I go and I remember doing that interview and I didn't know Linux that well. And luckily for me, I knew FreeBSD well. And as I'm answering the questions, I'm like, look, I am not an expert on Linux, not the way Google was asking these questions. And it's like three people on the other side of this table just rapper firing. And I remember I was like, I know FreeBSD. I swear I got lucky. This one of the interview, I think they had a FreeBSD tattoo on their leg, the little beasty logo. And I saw this logo. I'm like, "Oh, I'm saved." And so we started going down the FreeBSD questions and I pass and I get this job in this data center. And it was good because it's like, hey, I'm I'm working with my colleagues, but it felt a bit slow because you only get one job. You come in, you do this thing. And I got really good at it cuz to me, I kind of saw it as like a bit of a competition. Who is the best data center tech here? What are their metrics look like? How do I exceed their metrics? I want to learn how to do every particular thing in this data center because previously as a business owner, the more skills you have, more money you can make. And then I just started switching jobs every 3 to 6 months because I just wanted to explore everything to just amass my abilities. And doing the math, I think every jump was like a 25% pay raise. I mean, coming from a small base, it wasn't it didn't feel big at the time, but after a few jumps, my salary doubled. puppet was uh was a a bit of an inflection point in your career.

Host: You know what and I would say the biggest inflection point in my career was there were two of them before I get into Puppet Labs.

Kelsey Hightower: Okay.

Host: So you come from Google, you see this huge operations. There's hundreds of thousands of servers. The cables are perfect. They're immaculate. Oh, by the way, can you help us imagine like what a data center back then looked like at Google and and what what did you do as as part of the job?

Kelsey Hightower: Yeah. So in 200 maybe four maybe 2005 that data center is like a warehouse. I mean it's huge. So think about a place where and I'm pretty sure they always exaggerated the numbers. So exaggerated number to think about is like think about 200,000 servers in one place. Everything is immaculate. So a lot of people have worked in data centers and it's a mess. Wires are all over the place. You know you're ad hoc adding and removing servers. But Google was systematic. Those machines came off a truck. They were wrapped perfectly. When you wheeled them into their spot, you connected the network cables, they would pixie boot. They would burn in. So part of the job was, you know, you walked around with a crash cart. So depending on what your abilities were, some people had pretty straightforward jobs. You had a crash cart, had all your tools on it, and you would walk around and you would find machines that were needing of repair. So if you have 200,000 machines, it's okay if like 300 of them are broken, right? the system can route around that. And but if they were broken, they needed to be repaired. So you would go to rack a server 7, you will pull it out and you will look at it and oh yeah, the SATA card is on fire. Like it's literally burned. It's burning. It was burnt. And so he's like, I can diagnose this one with my eyes. It needs to replace the SATA card. But the thing is, you would go into the system and you would say, this SATA controller needs to be replaced. So you would be making your prediction and then you would replace it and you would go through its burn-in process. It would join back to the fleet and then the way you were measured was how good were your predictions.

Kelsey Hightower: Oh, right. So you said it's a SATA controller and to me moving fast you look as if I think that SATA controller I'm not going to waste any more time on this one because I'm trying to get my numbers up. So I just swap the SATA controller. You bring in all the cables, you slide it back, it goes through his process, and you move on to the next machine

Host: before you get the feedback.

Kelsey Hightower: Yeah. And fun fact, there's a guy named Tim Hawk is very popular in the Kubernetes community. He's like the network lead. So back then, Tim Hawkins is working at the other side of Google, like the bigger side of Google. And one of his first projects was built a little tool that you would put on the motherboard. It had about nine lights on it, and the lights would flash back and forth, and it would tell you what dim slots were potentially bad. The technicians that were fast, they didn't waste time like running a program to do a extensive test of the memory. You learn how to use this little device and you would walk up to the motherboard. So the thing I would do is like before I do anything, put this on the motherboard, get the reading, and I learned to trust it over time, dim one, dim two are bad. And the goal would be all right, I have all this memory on my cart. You swatch those two dim slots. Make sure that that's done first. Reboot the machine. And you might be like, I think that's the only thing wrong. And again, you were put in the system. I believe I only need to do DIM one and two. And then the way you were measured is how long before that machine gets kicked back to repair. So if it doesn't get kicked back within, I don't know, 30, 60 days, you did a good job. If you didn't, your scores would be low. So for the technicians that were just reckless, right? They wouldn't even try and you're just swapping the wrong part. You're swapping the wrong hard drive. And so your stuff is always coming back for repair. You were not efficient. So I got to the point where I can maintain high 90s but also repair let's call it three times more machines than other people. So lots of machines rate of return. And then I learned how to do the network switches. And then there's power audits where you're lifting up tiles from the floor and you're making sure that everything looks good. You got to be careful not to touch them because you could die. And so you learn everything about a data center like the service loop. You know you're running all this cat 5 cable and it has to have a perfect service loop. fiber runs on a different part of the rack, right? So you don't ever mix these things. So as a person still like I'm in my early 20s. I'm thinking this is how all data centers look. It's not the case. But it's it's crazy because just as I think back, you know, I was a manager before as well and of course a software engineer for a long time, but the way your performance was continuously measured and fed back to you and you were evaluated based on it. It feels way more strict, should I say, than you know like folks who work as software engineers including at Google. I mean the frequency, the expectations.

Kelsey Hightower: The reason why I didn't feel as bad as some of the metrics people are using now is because I felt like I can control the outcome. It didn't feel like it was a thing that was just a metric that didn't do anything. If I felt like my score was taking a hit and I was like, you know what, I am being a bit sloppy and how I'm diagnosing these machines. And I remember one time where I almost had my score dip below 90. I started writing additional shell scripts to starting combining different functions together. It's like, you know what? Can't be moving this fast. There's a way for me to diagnose multiple things of the machine at one time. And so, I would diagnose the Saturn array, all the hard drives while I'm doing the memory component. And then when it rebooted, I would just run the script one more time as I'm putting my cart back together to catch that one more thing. And once I started doing that, I can move as fast as I want it and the scores are right. So to me, when the scores actually match the things that you're doing, then it's a healthy feedback. And again, no one really talked about it unless you needed to talk about it. And so I kind of leveraged it for a personal thing. I pulled it up in the morning. I kind of looked at my performance metrics and in many ways I calibrated my strategy based on this detailed feedback that I was getting. So, I think I appreciated that level, that granularity back then because it felt like it was uh something that was helpful for me, not just my manager.

Host: And you said you had two two kind of big big inflection points. One of them was

Kelsey Hightower: Google was a big one. Like that was definitely one of course my entrepreneurship. And when I got to web hosting, I went to a company called Pier 1. They're a spin-off of Rackspace and they were all about fully automated self-hosting, right? So,

Host: this was back in like what 2005 67?

Kelsey Hightower: Yeah, this is like Yeah. 2005, 2006. Oh, they were already about that was their thing. Their tagline was latency kills. And so back then you would go online. A lot of the customer base was like um people hosting their own game servers, right? So if you wanted to play a game, one thing you could do back then is host a game server, but you needed a game server that multiple people could hit. And so you would go along to server beach. So this is a spin out of Rackspace. So Rackspace is more like, you know, we'll buy a server and once you get it and it's a lot of manual steps. Rackspace was more of a let's automated everything. So the machine would pixie some PHP scripts would run. If you ordered a RAID setup, then we would configure the RAID while the machine was net booted and then we would put you on the VLAN that you belonged on. We would install the right operating system based on what you've ordered and we just took a form. We just went through all of these steps and then when we were done, and it took maybe about an hour. When we were done, you had an IP address, login credentials, and if you wanted email and plus, you know, website management, all these things, you got it. And when you were done, you gave it back and then we put it back into the pool ready for the next customer. And so when I saw how we were doing that, I was like, yo, this is okay. You can automate things end to end. And the other thing that was important here, we were doing things like updating the firmware for RAID controllers because once you pixie a server, now you're in memory and you have access to all the hardware. You haven't committed to an operating system yet, but you have enough to do whatever you want. So if you want to configure the RAID controller and back then there wasn't like clean APIs. We were literally running curl scripts and command line utilities trying to get this machine into the right shape and then when we were done we would put it into the fleet. So at that early age I'm like oh there is nothing you cannot do. When we had to automate the Windows 2000 servers back then we would just build tools that would literally screen scrape log in to active directory and then we would screen scrape mouse movements so that way we can patch software on those Windows servers. So the concept of like I need a specific tool to do it's like no. Back then it was like you do whatever is necessary because these people are only paying like $99 a month. I don't have time for you to spend a whole week when they call random customer. You don't know their setup. You don't know their infrastructure. You have minutes to get them back online. So everybody learned to move quickly. No complaining. When you get that ticket it's on you to figure it out. Maybe you lean to your teammates for help. So I kind of learned how to move fast. But the the inflection point came from I started that job in tech support. So people would call my SQL isn't working, DNS isn't working. They can't even describe what isn't working sometimes. And so I realized that we were all in the phone queue. When the phone would ring, it would just round robin between everyone. Then if you couldn't solve it fast enough or you couldn't solve it at all, you created a ticket. And the ticket sat there and then ideally you get to it later, but the ticket queue was just getting long. And when a shift change happened, we just had all these tickets piling up. And of course, customers are now mad, 3 days, no response. So one day I said, look, I'm just not going to log in the queue.

Kelsey Hightower: I'm just going to resolve the tickets. And I'm back in that entrepreneurial mindset. I'm building little scripts to take a ticket. I see the issue. This is my SQL issue. We need to vacuum the database. This is easy. Run that one. Ticket close. Hey sir, everything is good to go. Please try again. Close. This is not even an issue. Close. Plus, oh, we got upgrade PHP. close and then the ticket queue is zero. It's just empty all day. And so someone's like, "Hey, Kelsey's not logged into the phone queue. He's not even on the phone. He's not taking calls." My manager pulls me in the office like, "Hey, Kelsey, why are you not on the phone queue?" I said, "Because we don't all need to be on the phone queue. We can just have one person making the ticket queue stay at zero." And then I would tell my colleagues, "Hey, look, if you can't figure it out fast, just open a ticket super quickly. I will take care of it." So some people specialize in Windows. They got a Linus call. They didn't know what to do. I said, "Don't worry about it. Just put it in the queue quickly. Move on to the next call. I got you." And so I became super efficient. So I explained this process to the manager and he thought about it. His name is Mike. And Mike was like, "Yeah, I like that. We're going to change it. We're going to have a couple people stay out of the queue, but the promise is that queue has to stay zero." And it's the first time I learned the difference between activities and impact. activity is you being an all-star answering a bunch of calls, but the ticket queue for the team is still high. You making this jump and saying, "Look, maybe I stay out of the queue and my promise to my colleagues is impact." The ticket queue is zero. So on Wednesday when we do the team turnover, we're handing off a empty queue. Now, of course, I didn't teach them that because I didn't think about it. And when we would come in, the queue would be high again. And it's like, yo, whoa, this is not you guys keep handing off a bunch of burden to our team. And of course, we would clean it up, but then the management team was like, yo, everybody's going to do it. And then I was like, man, you can change the process. So that was like the huge inflection point. I think the next one was more about being a mature person. Stop job hopping so much because again, after maybe a few promotions and some impact, I'm now off to the next thing. and I didn't have to necessarily live a long time with some of those decisions or you know stay long enough to really impact more of the culture. So when I got to financial services, it was the first time I got restrictions put on me, right? Because in these companies, everything is about moving fast. Google, we got to move fast. We have to compete with the big guys. We're going to do with cheap hardware, smart people, any means necessary. Web hosting, we're not charging a lot. Margins are thin. You got to move quick. Financial service like, no, we making money over here.

Host: You joined the financial services company.

Kelsey Hightower: Yeah. So that was the first time my my salary doubled. So I went on my lunch break for a job interview.

Host: That must have felt awesome.

Kelsey Hightower: Oh my god. I don't remember. My manager was so upset. I went on a lunch break.

Host: The previous manager, the one you

Kelsey Hightower: I met Pier One. He was a good friend of mine named Joe Rodriguez. He's the first person that made me a software developer. So, I showed him some ideas I had on like modernizing our virtualization, uh, optimizing our Pixie Boot process. And he was like, "Man, you have a lot of good ideas." So, in the interview, he's like, "Show me what you built." And I'm so excited. I'm still that entrepreneur thinking, and I got the job. So now I'm a software engineer working on this automation stack. And uh I remember seeing jobs that I used to be afraid of 5 years ago. The same jobs that made me want to just go get an A+ certification and open a computer store instead of even trying. And so I looked at those job descriptions again and it was like you need Linux. So like I got that even at that job I got Red Hat certified, right? So I was like I got all the qualifications now. And I remember the job. I didn't know how much it paid before I went, but you had to wear a shirt and tie. It was the first time in my whole life had to wear a shirt and tie.

Host: Oh, so you had to go and buy one.

Kelsey Hightower: Yeah. And I remember my home teacher, she him my pants for me, right? I was like, "Hey, I got a job interview. I know. Does this look right?" I'm driving up on my lunch break and I get there and you know, this is enterprise. I'm like, "Wow, this is this is the big league." I didn't even think Google was big league. I thought financial services big league. Obviously, if you have to wear a tie to do your job, you must be doing something so serious that you need a shirt and tie to do it. And so, I get to the interview and I'm sweating like, "What? They're going to ask me stuff that I've probably never heard of or seen before." And they started asking Linux questions. I'm like, "No, you got to use this flag. That's not the right flag." No, look it up. That is not the right flag. You can't do that with Grip. Nope. You got to pipe it this way. That the PS table doesn't show you that. Not on that version of Linux. not with that curl. And I'm like, it's too easy. And part of me is like, either I'm really good or something else. And so I'm thinking like, yo, that was maybe there's another round. And so I'm driving home and I'm loosening up the tie and I'm calling my wife like, "Hey, I I think I did a really great job." Like, you know, I think and and this has been a pattern, you know, I get good at something and I find a better job. I got good at something, I find a better job. But now it's like this is like a career. And so I'm talking to my wife and I was like, "Hey, I got to call you back." And it's the recruiter calling. Hey, you have a second? It's like 100%. Uh they want to make you an offer. I'm like, you know, make you an offer is different. Like it's usually this is how much the job pays.

Host: That's how it was until then. How much were you making?

Kelsey Hightower: 45K.

Host: So you were making 45K at that point.

Kelsey Hightower: $90,000.

Host: Yeah.

Kelsey Hightower: I almost drove off the rope. like n like double. This is insane. I'm thinking 90 and you're doing the you know retirement calculations. We can buy a house. You you're thinking of all of these things that you could do. And so I'm on the way back to the job and I remember and they were like then when can you start? It's like Thursday. I'm telling them we can start Monday. Like uh two week notice. I'm like I don't know man. Like it's devil for this. I have to. So, I remember I called Joe and Joe's based in San Antonio where the headquarters of Rackspace was. I was like, "Hey, Joe, man. Hey, man. I got to I got to I got to quit." He was like, "What, man? You just, you know what I'm saying? You're doing so well on the team. How you going to quit on me? I made a bet on you." You know? I said, "Hey, Joe, calm down, bro. They pay 90K." He's like, "What? Are they hiring?" And so, I got to I got to that company, Total Systems, Tus. And um it did feel a bit slow. They had run books, change windows, everything was regulated. This is a financial institution. We're processing credit card payments. We're doing work for the government. We're doing all of these things. And I remember the team was just everybody just moving at this pace.

Host: But isn't it crazy when you get in there? I I had something similar when I moved into London and my salary more than doubled going for Edinburgh where you have all these expectations because you're being paid so much more. the tie. Same thing for me for the tie and then it's a bit disappointing and I think did you not not feel a little bit like am I missing something like this should be you know higher this should be you know the interview was okay maybe not as hard but it it was something I mean I did because I was naive because I didn't understand the consequences of a mistake and so yeah I was like oh these people are just moving slow and I looked at what they were doing and everything was like a risk if you made a mistake you remember you only get seven milliseconds, 7 seconds to give Visa a response. If you don't, then it's a default decline. Now everybody's losing money. And so the cost of getting a good change, it was just worth waiting. So I didn't know that in the beginning. So I'm just like, oh, everybody's moving a little too slow. I was seeing how they were doing deployments. I'm watching how they're provisioning servers. I'm like, you know, you can automate this whole thing cuz I've done it multiple times. And then I learned to be a little bit patient. All right, hold on. I learned how to deal with the nose. I learned how to deal with the executives. I learned how to talk not just to engineers with the senior leaders and get their trust. And I won't talk about everything that I did there, but I was there for about three years. So the job hopping stops and I remember there was a task where we were using Apache and some Java plugin to talk to our J Boss instances. And so they were using a lot of memory per connection on the load balancer. And so we got a new we're moving off the main frame. We're moving into this new, you know, Java world and the servers just kept falling over. We couldn't handle all the transactions and I'm just watching the team go into these change windows and then we fail. We would fail. We would fail. And the CTO at the time was like, you know, this is costing us money. These we're getting chargebacks. We're having to pay out penalties. And I was like, I had a dev environment when I was using EngineX. I got rid of all this Java stuff. Like it's it's just HTTP. You don't need a Java connector thing. I'm reading the spec. You don't need it. And my memory usage is a fraction

Kelsey Hightower: by getting rid of it.

Host: Yeah, you don't need it. And engineext had a better threading model, all these things. And I was like, I got a perfect config. I've ported the Apache config to this one. I even handle all of our redirects, all of our routes, all the legacy crust. This thing will work. It's like, I don't know, Kel, this is not certified stuff. It's like, no, no, no. I got it from Red Hat. RPM install engine X. It's in it's in Red Hat. And they're like, okay, it's in Red Hat. Okay, sure. gave me one chance.

Kelsey Hightower: Yeah. And about after a week, they gave me a shot and they let me be in the change window. And

Host: what what is the change window?

Kelsey Hightower: So change window typically in the financial institution is like we can start changes at midnight and we have to be finished by 6:00 a.m. and you have to notify every customer. We're going to change something in the environment. Things may get a little weird. We would like your permission. And if if it's something that can be detrimental to your business, now's your time to speak up. So once we got permission, we had a window. That's the only window you got. And if you can't finish it on time, you got to shut it down and get us back to where we were. Yeah. Roll back. Roll back. Roll forward or if you could roll back. And I remember I put engine X in place. So at this point, it's all you. No one at this company has EngineX experience. Also, most people don't want to do this. They're like, "Hey, we think this is a bad idea." And so now it's your reputation on the line. There is no hiding behind the team. It's almost like everyone's sitting back like you got it. Now luckily the leadership was like we are supporting you and we're putting our careers on the line as well. I mean they probably would have been safe but after about 2 hours I got everything working and we just watched your memory pressure just drop. So if we were at 90% on utilize I don't let's call it 32 gigs and we're just blowing stack every time everything's crashing. this thing is hovering at like two gigs of RAM. And everyone's like, yo, like are we getting peak load? Is this legit? And we're all just sitting there like not sure. And the way we used to test the platform is we would someone would drive to the gas station, get a credit card, buy some gas, and we wanted to see the transaction land in the Oracle database.

Host: Yeah, you could track it.

Kelsey Hightower: So we can track the whole thing. We're like, yo, it works. And so the thing is, you can't really be comfortable until about 10:00 the next day. So now it's 3:00 a.m. We all go home. I don't go to sleep. I'm like, "Yo, we've changed a big part of the infrastructure in production. Let's see." So 10:00 goes by and we looking around and there's no naggios alerts going off. I'm like, man, we might be we might be good. You know me, I got my I got top going. I'm looking like memory usage is holding steady.

Host: Yeah, top is uh for those who don't know that that's when you uh it shows it show the CPU PS tree in a loop, right? So just like if you ran the PS aux command, you see all the processes running processes.

Kelsey Hightower: And so you just run top and then it's basically that every second being refreshed or whatever timer you give it. But one of the bets I made was if I get this to work, you have to buy pizza for the whole floor for a week. Every day we get a new order.

Host: And cuz he promised like Kelsey, if you get this to work, I'll buy you a steak dinner. And at the time I still am. I was vegetarian. I don't eat steak. But here's here's what I would like instead. you got to buy pizza for the whole floor. Now, I don't know how strategic I was thinking, but I figured that I could score some points with the whole team from turning it into a I succeeded to a we succeeded

Kelsey Hightower: and then people eating together was one good way of doing it. And I remember after it worked, he was like, "What's the order?"

Host: And I just ordered, you know, pizza from one place and we just put it in the middle. Everyone was like, "What's the pizza for?" Right? Cuz usually you only get this for special events, like for the thing we did. And the next day some more pizza and some more pizza. And so when I got that accomplishment, I was like, "Okay, this is what maturity looks like. It's not about just having the right solution. You literally have to get consensus, buy in, make sure and then your reputation's on the line." And I got that change in. It just kind of changed the way I thought about what success looked like. And so that was the inflection point. And then of course I brought puppet into that organization repeated a similar process of automating everything with this tool.

Host: Kelsey just talked about what CI/CD looked like in the 2010s and how difficult deploying to prod was which leads us nicely to our episode sponsor build kite. This type of manual deploy era was a time of build kite was created. Injuring orgs were trying to figure out how to ship more frequently more than once a week and those running into wall first turned to build kite. Shopify started running on build kite in 2015. Airbnb, Canva, Uber, and OpenAI have all been using Built Kite for 7 to 12 years. Throughout the 2010s, the hardest CI problems in the industry for global software leaders were getting solved on Builk Kite pipelines. That's exactly what makes Built Kite the CI platform most in tune with what's happening right now. Genai throughut is pushing everyone's build queue to its limits. Coding agents are pushing commits 5, 10, 50 times volume that your defaults were built for. The same architecture that absorbed Shopify's traffic at scale five to 10 years ago now absorbs a shared push from Entropic, Cursor, Meta, Mistral, Coher, VLM, XAI, Lambda and more who are all Buildcad customers. Where other CIS are cracking under the weight, Builkcite is running about 1.4 billion job minutes a week. Agents run on your infrastructure or on build kites. Every artifact and log is captured and when something fails, you see the why immediately. If you're hitting ceilings on your existing CI or downtime is becoming more problematic for your delivery, head to builkite.com/pragmatic 30-day all access trial, no credit card, and an actual human engineer on the standby. His name is Ola, and he is very helpful. I'd also like to mention our season sponsor, Sentry. AI agents don't have a feedback loop from production observability logs, but should they have it? I mean, we probably don't want to build a system where after each production error, AI agents automatically push out a fix into production without supervision. That would be a recipe for disaster. But what if we did something more modest? When a production error fires, an agent investigates this with context from Sentry. Sentry already has all the context in the error. After all, Sentry MCP is one way to plug in Sentry to agents that support the model context protocol. Cloud Code, Cursor, Codeex, VS Code, Copilot, they all do. After you hook up the MCP server, you can do some very useful things. For example, you could do this when an already resolved century issue resurfaces. You can kick off a cursor agent to investigate the regression, read the relevant code, and open a PR with a suggested fix. There's a little work involved to get all of this going. You need to connect Sentry to your code repository, add Sentry MCP to Cursor, define the instructions for Cursor's agent to investigate, configure the trigger that launches the automation, and test that it all works. But once you have it up and running, you can get regressions fixed faster whilst you're reviewing every and all fixes. This feels like a sensible and helpful use of AI and CP and Sentry to me. Check out Sentry at centry.io/pragmatic and start monitoring and fixing regressions today. And with this, let's get back to Kelsey and when he introduced Puppet into his organization and automated DevOps processes.

Kelsey Hightower: And I remember we had someone from Puppet come by

Host: and and just for again those who don't know Puppet, Puppet is this configuration management tool.

Kelsey Hightower: So puppet is we went from you know lots of shell scripts you know doing random things and pseudo automation but most of the time you just did a manual thing you had a ticket come in someone wanted a new SSH key on a server someone want something installed you did it manually copied and paste the output in the ticket and you closed it and you did that every day over and over again and that's where the saying that I I say sometimes some people have 20 years of one year experience some people in it have been doing that 20 years on the row they never the leap to automation or learning new tools or skills. And so I saw like man I can bring in a tool and I brought the tool called puppet at the time 0248 and this is right before DevOps becomes a word but it's like all right I'm starting to learn how to write code like the for the Java production stuff. I'm writing puppet, you know, manifest using the puppet DSL. Sometimes you have to write a little bit of Ruby to build a new resource type and I'm contributing and also I get introduced to open source like, hey, there's something there that doesn't work. I'm going to contributed upstream. So, I'm doing all this behind the scenes getting ready for act two. And I remember my manager went to a conference and he came back and like, "Hey, Kelsey, I finally know what you're doing." I was like, "What's that?" He was like, "Uh, you're doing DevOps." And I part of me was like upset. how do you get to name what I'm doing? And so, you know, everyone's talking about DevOps and he's like, I met a guy named James Turnball. And James is an Australian dude. He worked at Puppet Labs and he wrote the first puppet book that I bought and read. And so I'm like, James Turnpaul? That name sounds familiar. Oh, I'm looking at my desk, James Turn. So, he's one of the co-authors of this book. And it turns out James wrote a lot of tech books back then. And he's coming to the office. And my manager wanted him to check out how we were using puppet. I was like, "Okay, so you know, this is world-class expert." And by that time, I had puppet hidden behind Jurro tickets. And so you could just open a Jurro ticket. I had RPMs for everything. You picked the RPM you wanted from a drop down. You picked your environment from a drop down. And magic happened and you got results. Most people didn't even know Puppet existed. Most people never touched it. They just knew they can get anything on demand.

Host: You could go to Jira and it does it. So, so for example, you could like provision a new machine or

Kelsey Hightower: whole environment. I want database. I want IBM MQ series message server. I want these three apps and I need this firewall setup. No problem. Open the right jar tickets. Get them approved. When they're approved, we had this little I I called it Mr. Receti. All right. One of my buddies gave me a name from some video game. Once it got approved, Mr. city would own the ticket and we would just extract the custom fields and we would just call the right puppet manifest and we would take the output and update the ticket and so PMs developers they just got what they want damn near instantaneously so we built all the systems they didn't know nights and weekends I'm contributing to puppet because you know wasn't allowed at work at the time this is 2007 2008

Host: they probably didn't even really understand what open source was anyway

Kelsey Hightower: so they was like hey you got to do it on your own time so I would just do it nights and weekends so James shows up and I remember we all get in the elevator. We're going to the third floor. My manager's introducing James to me, to our team, like, "Hey, this is this person. This is this person. This is Kelsey." And James turns to me, he's like, "Kelsey High Tower? Oh, we know you. We love your contributions." And my manager is just like, "What contributions? We're We're not doing contributions." And James is like shaking my hand and he's just talking to me like we're friends because the thing is I have been working with the puppet labs team through like openource open source contributing to puppet and so we get up there and you know my manager is like hey let's show him our setup I was like all right James we have a lot of puppet manifests over here I'm using external node classifiers I'm setting brightening config from data you know I read my burst is promise theory and we got it all dialed in but from an interface perspective I just didn't want everyone to have to modern puppet. So, we hooked it up to Jer. I'm showing him this entire workflow and he's just sitting there like, "Yeah, I have no recommendations at all. Like, this is this is amazing." And here's the thing that kind of the game changer. So, he came all the way to Atlanta. We're 45 minutes from the airport. He spent time with our team and uh it's the end of the day. And then he's on the way out and he's about to call a cab to go to the airport. I'm like, "No, I you're in my hometown. I cannot let you go to the airport. I will give you a ride to the airport." So, I get in my car and he's in the front seat. My colleague's in the back seat. He takes off his tie and rolls up his sleeves. I can see all the tattoos. I'm like, "Oh, so you're like a legit text like, "Yeah, I don't do all of this. I just did this for you all. This is your dress code." I was like, "Oh, James is cool." He was like, "Man, yeah, you guys are doing great work." And you know, now I'm talking to the real James Turnball. And uh he invited me to give a talk at Puppet Cough. and I went to go give that talk. It was my first like real conference talk and I talked about some low-level kind of puppet integration stuff and I had a little comedy in there. It was like the first time I felt like I could just be myself on stage. Before I came home, Luke Kenise, the founder of Puppet, we sat down in a little coffee shop in Portland, Oregon, and he was like, you know, would you like to work here? I was like, yes, like of course, 100%. And of course there was a nice salary increase and I could work remote. I didn't have to leave Atlanta immediately. And I remember coming back and uh my manager really appreciated all the progress I was making over the years. I kind of made a name for myself. And I remember he gave me a nice little raise just out of the blue. And he slid this envelope to me and I opened it. It was a nice number in there. I was like, "Oh, that's great." I was like, "But here's a thing. I'm resigning. I'm going to go work for Puffet." And then he said something that was dope because we had a kind of, you know, difficult relationship depending on how it went. But overall, he helped me mature. That's one thing I will always say. He helped me really be mature. And uh he was like, "I'm surprised you stayed here this long, right?" As like this ultimate compliment that you stayed here beyond the just getting better or making an impact. You literally changed this culture. And the full circle moment happened like 7 years later when Kubernetes came out. I remember coming back to that office and they had already been running CD in production, Kubernetes in production and even some of the old tooling I had was still running and to me that was like a really important feedback that sometimes if you make an impact lasting impact is way different that even when I wasn't there they still had the culture to say there's new technology and we know how to bring it into the stack.

Host: So at this point containers were starting to start right. How did you first come across containers and very early on you realize this is going to be big and important?

Kelsey Hightower: I didn't.

Host: Oh you didn't?

Kelsey Hightower: No, because I'm at puppet labs. You know, before that I was contributing to core Python stuff like virtual imp pi on the team that was trying to integrate some of the Python. Python had a package management problem even back then. And so I'm all in on Python and then when I'm learning Puppet, I'm all in on Ruby at that time. I'm thinking DevOps plus configuration management is the end all beall for everything. And really at that time we're talking 2011 2012 in my mind the competition was only between Puppet Chef and Anible. That was it. And so I'm all in and I'm working at Puppet Labs. And so we're kind of in some ways I felt like we were dictating the future. We were going to companies and getting them to understand configuration management. We were talking about the benefits of like speeding everything up and you know doing all these compliance. We were moving people from SharePoint to executable code to implement these things. So we thought we were just getting started. There was no world where we then think that we needed 10 more years of that, right? We even got money from VMware to integrate Puppet into that. Chef was being integrated into AWS at that time. So this was like the thing that finally were finally getting there. We just got to teach system administrators to become engineers via DevOps and started embracing these tools. And then the first thing that I saw come out was Golang.

Host: Mhm. And I remember sitting at my desk and I was like,

Kelsey Hightower: you know, at that time we were hitting performance issues with Ruby, right? The global lock interpreter. You can't do more than one thing at a time, not easily.

Kelsey Hightower: And so we moved to things like J Ruby. We even did closure for some part of the puppet stack. And I remember we were thinking about should we start doing stuff in C++? Mhm. Should we start doing stuff in C? But the problem is contributors. What would we do with all the Ruby contributors, right? we were relying on things like Ruby gems and it's going to be it was going to be a hard sale and I remember downloading go for the first time and the thing that sold me on it I remember um one of the prototypes that I built was factor so factor is one of the agents in puppet that gives you facts about a machine so this is Red Hat this kernel version this is what the users look like and it would give that to puppet so that your configuration code had context on what to do and that was written in Ruby and sometimes it will run slow because you're reading all these files serly. And so I remember I was like, "All right, let's try this Go Lang out." And I remember I wrote the code on my Mac, compiled it, SCPD it to a Linux box. I was like, "Oh, this is crazy." And I remember running all the facts in parallel, and it came back in a fraction of the time. I was like, "Yo, we got to use Golang for stuff." And I remember the team was like, "No, it doesn't work for Solaris or AIX because Puppet was now moving into the enterprise." I was like, that's the criteria? No way. But I got it, right? I've gone through this maturity thing before. And then Terraform comes out. But before Docker, Terraform is trying is starting to challenge things a little bit, right? And it's starting to use things like Go, at least that was on my radar. So Terraform is like, who cares about the node? It's all about the APIs. And all of us come from the server world and everything's about an agent on a node versus the cloud starting to take over at this point, right? So,

Host: So Terraform was built with the cloud in mind, right? You you could configure your cloud environment infrastructure as a code.

Kelsey Hightower: Yeah. And we were trying to do this from a node. We were trying to teach puppet how to talk through by having this indirection where you have to go through a node to configure a server in the cloud. It was so weird. But then Terraform Mitch Hashimoto, right? Like he was a big part of the DevOps movement early too. Then he was like frag Vagrant was written in Ruby. So he was part of that same ecosystem. And then he splits off and there's this going. I'm like, look at that. They're using Go for this. And I remember when Docker came out, Puppet at that time was the one pushing innovation. We're asking people to think a little different, get out of their comfort zone. And then Docker comes out and most of the office was dismissing this as like, nope, this is a fad. They don't even have config management in this thing. They don't really understand enterprise. This is just kind of not like a toy. They were not being disrespectful, but no one saw this as a challenger. what we were doing and I looked at it and I didn't get it immediately until I left. When I left Puppet, I built this tool called CompD. I went to go be a VP of engineering and I got to write Go code. So, we started rightfully so. We looked at all the Java heavy usage and I earned the trust of the team and we started rewriting some of the microservices in Go, shrinking our cloud footprint and we sunseted it and got it all into production, the Go code. I was like, we don't really need Puppet anymore. And I open sourced this project called CompD and it would pull variables from CD and generate just enough config, just the parts of Puppet that I thought made sense. And then Docker was out. And then I was like, wow, we can probably stop moving Python files around. We can probably package them up. Not in our case.

Host: And the idea with Docker was right, that it it would define a virtual machine or a container, right? So to me the big value of docker at the time was previously I had definitely did the work to make RPMs for every app even the custom apps RPMs are the red hat package management y so if you're on a red hat system you can do yum install engineext yum install postgress but most people even today don't package their third party apps like the apps that a development team would write you're like no we just put cicd we'll copy them over there maybe put them in a tarball but we usually never went to making official Debian packages or RBM packages But puppet meant you didn't have to go through all of that work and you can still end up with a package something that was repeatable. So all the stuff we used to do with Python and virtual in all the things you used to do with Ruby gems and you know the virtual environments we have for that we got rid of it just squish it all into a docker container and you got rid of a lot of dev tooling. So this is why I think it resonated so much with developers because we cleaned up the mess of working on multiple projects and so I was attracted to that comp was compatible with that and then coro was the thing that I was like you know what I think I know what I want to do but I didn't understand distributed systems not to the degree what coro was doing. So when I moved to the the what was the idea behind coros was like Google's infrastructure for everyone else and so at that time we did there was a tool called MSOS um the board paper had already been published I tried to read it when I was at puppet labs I don't understand any of this stuff is hard to install I couldn't justify it but we would see kind of the rumblings of the Twitter folks talking about the distributed system and all this maybe big data stuff is going around but I was like I just don't understand it seems incompatible with the stuff that I think about even as someone who's worked at Google that had a system like this, right? They had Google cluster file system at this time. It didn't seem as complicated as this MS thing. So I was like, I dismissed it. But what cores did was build on top of Docker. Cor is like, what if we had an operating system that only had Docker on it? Everything is written in Go. We can have a little key value store where you can put your config and we would just synchronize it to all the machines. And that was so compatible with the comp dway way. And so I'm looking at this cores thing. And I'm like, yo, this looks a little bit more like the future because that's what we used to do. You know, I had at this time I was a software developer at Puppet. I was a software developer, VP of engineering at the other company. And I was like, you know what, ops can learn a lot from the uh Docker way of thinking about the world because as a system administrator, we always try to make the OS small, remove things you don't need so it can be secure and repeatable. Docker was like for the things that need to change, just put it here and isolate it. And Cororos is like, "What if you had an operating system designed only to do that?" And I met the Cororo team at Gopher Con, a Go language conference. And they saw me present. And what I learned from the puppet days, every presentation is an interview. I don't think a lot of people think about it that way cuz people are looking at you on stage and they get to see what you're about. And so at Gopher Con, I remember also Rob Pike and the original creators of Goldinger. Oh, this is like the first Gopher Con, right? And so the original creators, Russ Cox, all these people I look up to, Brad Fitzpatrick, and they're all in the audience. And I remember I had a talk, two things. I'm sitting in the audience waiting for my turn. I'm number four on the list or something like this. And the two people who started Gopher Con, they're just from the community, Brian Kettleston and Eric St. Martin. And they're new to this conference scene. Like you could tell, right? You know, they were the best MC's in the world. But they were like, "Hey, welcome to Gopher Con." And at the point in time, Ruby is still dominating. And down the street, there was the Ruby conference going on. And we were just in this building. And I remember the first talk they introduced Rob Pike and he gave this amazing keynote around Hello World. He went up all the way down to the compiler. He went all the way up and he described how the language took shape. It felt simple on the surface, but it went super deep. And it's like Rob Pike had also come full circle from his AT&T days through Unicode through all the stuff that he's ever done. And now we get to see one of his best works, Golang. And my talk was titled Golang for system administrators. I wanted them to see that there's a better way. The way they introduced Rob Pike, though, I was like, "Oh man, this is Rob Pike. You got to you got to have more energy than that for Rob Pike." So, I'm sitting in my chair and I don't know them, but I'm sitting with a buddy of mine from that company where we rewrote everything in Golang, Billy Click. We're sitting next to him. Say, "Hey, Bill, I'm going to go ask them, can I be the MC?" He's like, "What?" Okay. So, I'm whispering. And I walk to the backstage. It's like, "Hey, Eric, uh, Brian, uh, can I try my hand at being the MC?" And they're like, "Sure." Right? And they knew who at least who I was because they accepted the talk. And so I don't know if it was after Rob, but I came out next. So like, "Hey, I'm Kelsey High Tower. You may not know me, but I'm going to be your MC for the rest of the day." And I don't know what happened because it was my first time doing that. I'm cracking jokes. I'm having fun. And I said, "Hey, from here on out, when anyone comes to the stage, it needs to be loud in here to the point where we can get kicked out." And so I was, "All right." Hey, so our next speaker and I think it might have been Derek Coloulston, you know, he's like a Google. He was talking about some ghost stuff and I remember it got really loud and then they come on stage. So if you were a speaker ever before and it's like you walk up to basically a standing ovation, you're energized, everybody is excited. I don't care what you do, the energy level is high. And so he comes out, he's having a good time, the audience is having a good time. And then eventually it was my turn. And I realized like who introduces me. So I go onto stage and I was like I'm next. So we're going to try it like this. I'm going to go back and I'm going to come out and you guys gonna make a lot of noise for my talk. And then I remember just like sliding slowly behind the curtain. So everyone's now laughing. And then the music comes on and I walk out surprised like hey. And everyone's clapping and I do this talk and at that time it's live demo or nothing. And at the time they had this thing called go present. So Rob Pike team, they wanted to make sure they could use Go for everything, even generating a slide deck presentation.

Host: It was all in Go.

Kelsey Hightower: It was all in Go. So you had this nice HTML representation of your slides. You just open up your browser.

Host: Hopefully it won't crash.

Kelsey Hightower: Hopefully it doesn't crash. And you can run executable code in your examples.

Host: So any code snippet you put there, it was formatted nicely. And you had the little run button so people can see the output of the Go code. And so I was like, hm, what would a system administrator value you can get from go? How can I prove it to them? And so I wrote a ipixie server in Go. And I remember in VMware Fusion that runs on your laptop. So if you want virtual machines on your laptop, you can use desktop parallels or VMware Fusion. The thing about VMware Fusion on your Mac, you can create virtual machines, but you can also swap out the network card. And I did one where you can have a network card that talks to IPixie. So, I wanted to show them how it would um boot up multiple machines from a Go Pixie server running on my laptop. So, part of that demo would start the I had this kind of snippet of my Pixie server and I hit the run button. So, now my Pixie server is running in the background. I was like, "So, what can you do with it?" And I remember looking in the front row and Rob Pike and team are just intimately looking at this thing like, "Wow, this he just started a pixie server from his laptop from Go Present the slide deck tool." And so this thing is running and I'm like, "All right, let's bring up VMware. I got to make sure I switch the adapter to the one that supports iPixie. Got to do some firmware thing." And then I remember I booted the VM. You can see the logs in the Go Present of like HTTP handing off the image, giving the IP and the virtual machine is booting up and you can just see the amazement on the audience face like did you just do that? And then I booted another thing in the thing and I'm going through why I think that this is a gamecher for the craft that we have. We finally have a tool where we write high performant things with the simplicity of imperative programming and we can go beyond just scripts. We can actually build systems. The audience was dialed in and it got loud like after you know things were working people were clapping. I look in the back and there's a whole bunch of people now standing and I recognize some of the names because they're from the Ruby community. These are people that are writing the Ruby books. And then, you know, after I'm done, there's a break and I walk back there. They're like, "Hey, we saw on Twitter that you guys are just like going crazy over here. It's out of control is electrifying. We left the Ruby event to come here." And so I felt like, man, it had arrived. But guess who else was there? The Coros team. The Coros team was watching me because I was pixie booting Coros. This is what I mean is every job's an interview. the team was like all right we can see that yeah you should join the core o team so that's how I ended up at coros this is how I really felt like the container movement had legs and this is what maybe a year or two before kubernetes comes out

Host: and then kubernetes came out u and you were starting to contribute to that as well

Kelsey Hightower: yes so my core os and I think the thing that's very important a lot of software engineers sometimes they look at the people on stage and they ask questions like does that person know what they're doing is this person just like an evangelist. Uh, did someone write this demo for them? Did they give it to them and they just run the code? Yeah, we we think that, don't we? Well, I mean, I can see why because sometimes it's hard to value skills you don't have. So, a lot of software engineers are terrible. If you put a soft some, they can't talk. They can't simplify concepts. Maybe they can write code really well, but this is a set of skills that they may or may not have. And so, when you see someone like that, you're you're you're questioning them. And I remember giving a talk one time at Strange Loop about Cedd and CRO OS and someone was like do you understand uh is isd like a CA system or AP system like the cap theorem and the question was kind of loaded like we don't think you even know what that means. I was like um CD is going to always favor consistency. He was like that's not correct the RAF paper blah blah blah. I said listen you said Etsy D. Let me show you this is the raff log. This is my three nodes. I'm going to turn off two. And you notice I can't write any keys. So availability has been sacrificed. Consistency is being preserved. I'm going to start another node. There's going to be a handshake. There's going to be quorum. I will be able to write keys. Now it works. That's the cap theorem in reality. So it doesn't matter what the rap paper says that you're talking about a raff log on a single implementation. Raph doesn't talk about cluster membership, leader election, how it's implemented, and what you should do in the different modes. That choice is yours. And this is how D is implemented. And I remember he was like, "Oh this guy actually knows what he's talking about." Being at Coros, we were working on our own fleet management system called Fleet. We're using systemd and we're trying to synchronize configs through CD and remember in a core os cluster all the nodes communicate viacd. So imagine using systemd for those that have never used systemd you put a unit file like I want this process to start with these flags you know bind to this port and you put it in a directory and then systemd will start it and if you had a thousand nodes of course you could SCP that file to all 1,000 nodes and then there you have it. So we decided instead of you copying all the files of all the nodes just put the unit file incd and the node that should run those things would then pull from and just run those unit files and we called it fleet. So we had our own vision of giving people a distributed spouse or distributed system. About a year goes by Kubernetes comes out and everyone was like what's that? And we're all we got like a day notice. So the Google team reached out like hey tomorrow we're announcing this thing. Here's the GitHub repository. Here's you guys are under embargo. Don't talk about it. And so I'm thinking we're not part of this story. like our our names are not in here just say as Google and Red Hat and they're going to announce it at Docker Con. There's no core OS in here. So, what can we do? And I remember I stayed up all night. I got access to the GitHub repository. I reading all the Go code trying to figure out what all these binaries mean cuz there's no docs.

Host: Yeah.

Kelsey Hightower: And so I got everything working on core OS. So when they did the official Google announcement, of course there's a famous Docker keynote where they unveiled it to the surprise of even Docker to some degree. And I remember they post their number one on hacker news and then we post hey what they just said but here's how you run it on core o and some examples and commands and I had to do a few patches to get it to actually work and I had to build some binaries to get things glued together. You did that overnight. Overnight I didn't go to sleep. I'm just like hey guys I finally figured out how to compile everything. I think the kublet does this. You have to put this there. They had like a coup up SSH script, but I had to reverse it because I'm not using Google Cloud. We got core OS. We're on bare metal. So, I have to pixie boot some things. But I think if you put all these things in the right place, there's this CD thing that's ours. We know that. So, we know how to use that. But I think the API server connects to that. There's no volumes. There's no config maps. So, all you can do is get this thing stood up and then you can submit a config and then it will just basically use Docker in the background. Okay, I can document that. So, we get everything to work and I write this nice little guide. Someone in our team, they publish it to the official core west website. So then we launch that on hacker news and then we go to number one and everyone is like Google just announced this thing. We don't know what it is. And then Kelsey launches this thing. We now know what it is. We know how to install it. We know how to run it. So now people are downloading core OS just so they can play with Kubernetes. And I had a keynote probably in a week. I don't know what I submitted to the conference cuz usually you submit like months ahead of time. I was like, "Hey, I know this talk was supposed to be about this, but it's not today. It's going to be about Kubernetes." And people like, "What's that?" It's like, "Yeah, it just got announced. I'm going to show you." And I started giving people live demos of Kubernetes and how to make it work, you know, using Coros and all these things. But the team still wasn't sold because we had our own road map.

Host: Yeah. and and you had your own fleet management software.

Kelsey Hightower: Yeah. We had our own Kubernetes was now competing with it with fleet specifically. Yeah. And also we didn't know we can trust it, right? Remember Docker is the king. Docker's number one. There's Docker Swarm, right? And we're with Fleet. And right now, we're like, uh, Google also launched years prior a thing called let me contain that for you. It was a container runtime written in C++ to compete with Docker. And no one cared. Yeah. And so we didn't know like maybe no one's going to care about this either. It's only when I started going home and starting doing small contributions and starting to read everything and getting a feel for it, I was like, "No, there's something here." And so for maybe two or three months and luckily the founders of course were really nice, Alex Pov and Brandon, they were just like they didn't get mad or anything. I was contributing nights and weekends kind of like I was at that other company. Uh all of my keynotes were more like Kubernetes plus core OS. And I remember at some point it was inevitable. We all got in a room. I became the product manager of core o at the time and Alex is like I think you got the vision here and we got everybody in the basement in San Francisco in the office we were like hey guys all in on Kubernetes fleet deprecated all these things that we're building deprecated we're going to go all in and I'm glad we did because um Alex Py came up with the name operators which is like a core idea in the Kubernetes community we put all this effort in there me and another guy working on a thing called CNI which is the networking layer for Kubernetes And what Kubernetes really meant for me was that previous 15 years of experience as a practitioner in the data center learning promise theory learning Puppet where it works well where it doesn't and then understanding that Puppet wasn't the only way and then making going through all these loops. And so when I ended at Kubernetes it felt like this would be the thing I would build if I only knew how. And that's the way I explained it to the rest of the world. And at that Gopher Con, maybe the next one, there were people from the now early Kubernetes community. It's a small company called Kismatic. They were kind of a cores competitor to some degree, but they came up with the idea like we should have a conference just like GopherCon. We called it KubeCon. Joseph got the logo going and the Kismatic team kind of put up the money for the first event and we really welcomed the entire community and now it's been 12 years later and the CNCF has done a fabulous job of keeping it going. Now there's like what 13,000 people here in Amsterdam keeping that thing going.

Host: So I asked this from Cat Cosgrove as well who was on the podcast. What do you think really made Kubernetes break through and then just just become the de facto way of of orchestrating nodes and and and just winning again? There was at core o you were building fleet docker had swarm like as you said in the beginning it it didn't seem like this this will be this big

Kelsey Hightower: I think the number one success criteria was docker so remember there was mos and msosphere and they had their own runtime corp had come out with nomad and they had their own runtime but the biggest runtime that had already got global consensus was docker so by that time there were so many docker containers and docker workflows and docker Swarm maybe the Achilles heel to Docker Swarm was its design. They tried to take the Docker API which worked really well for one node and expand it across multiple systems and it was not the right API to scale to another type of thing that we needed. And so they kept trying. They tried to add storage. They tried to add networking but the Docker API was never meant for that. And so the Kubernetes team was smart. Instead of trying to say Google's better than everyone on everything, they did a couple of things correct. Let's just use CD. Let's just use Docker. So you take those two things and you take the experience of the people who wrote the Omega paper which is kind of thinking about what would come after Borg and at least the things like MSOS.

Host: So Omega was a system where what Google would have built after Borg but they never built it, right?

Kelsey Hightower: Yeah. So they had elements of it. So like the omelette, you know, this like agent that would like be more declarative. a lot of hints from the Kubernetes world that will come later and and then just to be clear Borg was and is still Google's way of managing their back then hundreds of thousands now probably millions or tens of millions of of servers and they were best in the world with this or they still are right so they learn I think Borg was one of these things where you integrate the hardware the software the package management the configuration management map reduceuce right Borg is this thing that just expands and grows in some ways I guess it's extensible but all that insight and knowledge, but then they get so much experience with that. If you were to do it again, what would you do? And you read the Omega paper and it's like here's how we what we learn from scheduling. It doesn't need to be that complex, especially for certain workloads. You don't need this like high performance overengineer thing. There's a simpler way to do scheduling, especially if you can give the scheduler a bit more metadata about the workload. They're also big game changers. Now instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers. And so I think that's the number one success criteria that we were already off to a running start because you could just reuse the same Docker containers. You didn't have to rebuild a new image thing. So given that what they tried to do in the early beginning was just fill in the gaps. And in many ways, yes, it's a new system, but it fills in the gaps. The one gap that they filled in was Docker had an entry point. So if you needed a Ruby app that needed EngineX and your process, you used to have to write a little shell script, the entry point script that would do all this magic almost imitating an init system. Kubernetes is like, no, no, no, you don't need to do that. You can just make separate containers and then Kubernetes would run them as a process tree. And so for many people, it's like finally now we can have a clear way of thinking about application architecture like blocks. like blocks now instead of like you have to open the entry point to see what we're going to do versus full life cycle management independent. So it solves that number one problem. The other big one that I think that they solve number one we went from infrastructure is code to infrastructure is data and infrastructure is code is like if this do that um bring in this module for loops all this stuff and Kubernetes is like no no no you have to specify exactly the containers you want how much memory that they need and then we have the status field to tell you if they were running or not and you would take this data object that you could write by hand give it to an API and then the control loops would operate on this state so That means it didn't matter if you had Ruby, Python, or anything. You can just take your IDE, write some YAML, give it to another tool, manipulate the YAML, and then pass it down to the API servers. You can build any combination that you want it without having to be a compiler first. That to me was a fundamental game changer that I don't know if a lot of people understood why it felt very easy to onboard to Kubernetes. Cubectl apply object off you go. And the last thing I think credit to Brendan Burns, the ability to extend Kubernetes in a first class way. OpenStack didn't have it. MSOS didn't really have it. In MSOS, you have a scheduler and you built the other part of the scheduleuler. So you can have Spark, Hadoop, Marathon, but you had all these other tools sitting on top of a thing. So an extension in MSOS was heavy almost like a whole another system. The thing that makes Kubernetes powerful, there's a data model. We gave infrastructure a type system. So instead of imperative shell scripts, you finally had types. So if anyone's ever come from like Python to a type language, types do a lot for you in terms of cognitive overhead. Like you really know what goes into this function. If you put the wrong thing into this function, it doesn't even work like you can't pass a string where an integer needs to be. Kubernetes brings the same semantics to infrastructure. And finally, now it's much safer to automate things because you're gluing together things that actually have structure and types. I was about to say the reason we love types and every language is goes towards them is safety and it eliminates a a whole class of errors. Yeah, you can do things like static analysis. You can have other tools compile different things and ensure that they have the exact same thing and you have this validator tells you that's not the right object, that's not the right field. And so once you have all of those things, you can build a really nice deployment system. So coupl deploy these containers, no problem. But what about everything else? So instead of trying to evolve Kubernetes to do everything, Brendan Burns, I remember sitting next to him, he's like, "Kelsey, let me show you this thing, you can extend Kubernetes just by giving it a description of what you would like your object to be." So if you were thinking about this from the rest world, hey, I need a user, here's the credit operations, and just give it to the thing and it does everything else for you. And so when that came out it's like so if I wanted to manage let's say a firewall like yeah you can describe a firewall and you give that to Kubernetes and all the tooling works you can now say coupl apply a firewall and so now we got tools like search management where if you want a certificate from let's encrypt you can just say what domain you want where it should live give it to it and now you had a first class extension you didn't have to uninstall it you have to make some magical binary and it really didn't matter what what language you on it because once you put the data model in place, you got the machinery and if you care, if you like Python, you can have Python running a loop, grab the data and then make it so if bash was your thing, you can literally pull the config using a bash script and make it so and just update the status field. That opened up the entire ecosystem. So Cisco could come in, do what they wanted to do. Red Hat can come in, open shift and do what they wanted to do. To me that was the gamecher that brought the rest of the community in.

Host: And then you joined Google not very surprisingly at this point I guess from the outside of course but but h how did that go? You've been contributing to to Kubernetes as well. The team team was there. Did you join the Kubernetes team?

Kelsey Hightower: No. So by that time I'm at Coros Kubernetes has definitely taken off. I'm giving lots of keynotes now. Everyone wants to know my opinion. I'm making all these prototypes. I'm kind of moving things forward. There's a Kubernetes book now, right? I'm a co-author with Brendan Burns, Joe Beta at the time. I'm like, you know what? I am I'm thinking about my exit. So, in our careers, we do a lot of work to get into this field. All the certifications, the boot camps, the studying, some of us college, and then once we get in, we're thinking about career progression. Dev, senior engineer, principal engineer, distinguished engineer, and we spend almost our entire lives on that trajectory. And our field is so young that some of our pioneers are finally like no longer here for the first time. We're not used to that and we're not really used to people retiring. Like Rob Pike just retired. The concept of an exit for individual contributor or a leader in the tech space is we don't we didn't have a lot of those. Lionus is still at it. He's not retired, right? So we don't spend a lot of time thinking about the exit plan in our field. If you're a professional athlete, your body will tell you when it's time to go. And so when at core OS we reached this peak, I felt like I've done everything I've ever wanted to do in in the tech industry. In the tech industry from 1999 to being unsure of myself to seeing myself on the side of museum buildings, full landscape view because people are coming to see what I think about where technology is actually going. And so it comes full circle. You get a bit of taste of the fame. You can look at GitHub and you meet people. We use your libraries. We used your command line tools. I started my career like some people were not even born when I graduated high school and they started their careers from those books. And so I felt at the time that I had come full circle and I was starting to think about the exit part of that journey. I remember spending time with Jet Propulsion Lab, JPL, part of NASA. And I remember being there and and I was so excited because the movie The Martian, had just wrapped up filming there, and they gave me a tour of the facility, like the Mars rover, the new one before they launched it to Mars. They were QAing. It was just going in a circle around the track. And I'm like, had a little laser on there so it can split rocks. And they were showing me how they improved the wheels over time. I went to another lab and there were a bunch of scientists working in there and it looked like a fish tank and I described it that way was like yeah we we determined that if you want to replicate parts of Mars surface and I might be getting this wrong that the rocks that you find in like a fish tank we can replicate some of this stuff and maybe slightly different than that. As I'm watching these people work and he showed me like how spacecraft has evolved over the last 20 or 30 years and I'm like wow you all seem to have an actual purpose. For the first time I've seen people using technology not to just make more apps not to add numbers in a database but to actually have humanity do something. And so they were not all about Kubernetes and Docker and Python and Go. They were like we're just trying to get a person to Mars and back again. I remember in part of the interview process that their interview questions are if you had to deflect a meteor, how would you do it? But it has to hit one state. So now you're in the leadership position. What would you do? And you're just explaining the answer like you know I would kind of bring a bunch of experts and then you know you got to think about the ability to evacuate people and you have to explain yourself. And the core part of my answer was you would have to explain yourself almost 24-hour live stream. Here's the trajectory. Giving everyone the countdown. Explain every decision you're making. I chose transparency. I chose truth. I chose like, look, we have to deflect it. There's no way to make it zero. So, we've chosen this state and this is the evacuation plan. And we estimate that this number of people won't make it. We're just being honest with you. No need for conspiracy theories. It's live. And I was like, wow, what an interview. A 20-year career at that point. never had an interview that made me feel that way. And so I actually was going to go to NASA after Coros. I even signed the employment agreement. I was going to move to Pasadena, California and work at NASA on the Mars mission and lead up the infrastructure and the infrastructure teams. And of course Google called. I was like, hey, come to Google. And at that point, I was like, for what? You have hundreds of thousands of employees. I admire Google. I've been there before, but not in that capacity. I was going to go to the headquarters. Yeah. not not not to the not to the data center but to the headquarters and I and I've always admired people like Brian Grant, Eric Tune, Don Chin, all of these wonderful people that I got to work with through the Kubernetes community. In many ways, I felt like I was already working on the team because by that time I had commit access to Kubernetes. So, I kind of felt like all the things I wanted to achieve in that regard was there. And so, I was like, why would I come work there? I'm just going to be a cog in the wheel. I'm going to go there. I'm just going to disappear. you're going to just make me work on Google stuff all day long. What's the value in that? I've seen the peak of this. And they were like, we won't do that. I was given the opportunity to do Devril and it's the first time I ever did it. But devro represented freedom. No tickets, no write code measured against squee benchmarks. I was like, I don't want that. I want to be able to make impact. And so the team was smart. They were like, look, we got this area called Devril and we'll let you define what you do. So you got to write your job description pretty much. Yeah. But as an entrepreneur, I know how this goes. How does it go? If you come in and you really do Devril stuff, in my mind, you're going to get fired because if you limit yourself to the external perception of Devril, you go to conferences and you become more of an evangelist, you do tutorials and guides. For me, those are activities. I'm a person of impact. And so, the first thing I did when I got to Google was like, where's the customers? How do we make money on cloud? So, I need to figure out how to talk to the customers. So, hey, where's the sales rep? If you ever need anyone with Kubernetes expertise, call me. I will fly to Disney. I will go to Walmart on site and I will whiteboard for 6 hours because that's the revenue component. So globally went to Australia, Canada, doesn't matter. We don't even have a region there yet. I'm going to bootstrap it. So now I'm growing my impact on the revenue side. And and the reason you said if you would have gone as Defo, you would have been fired because you would have not been generating any revenue. I just felt like I was going to be fired. But but this is just like thinking as an entrepreneur as like you want to make money. I want I want to make sure that I'm impacting the business. And for most businesses, revenue is the criteria. And no one ever made me do that by the way. It wasn't like Demetri. It's like no no. To me, I understood the value of revenue. So I would go out and I was able to do it in an authentic way. I'm just talking about the same things I was talking about before. And then product impact. This is cloud. Why limit yourself to Kubernetes? There's serverless, there's databases, there's metrics, there's there's so many things here. So now I'm like, I need to learn everything and I want to employ all of my skills. So it turns out my time in financial services means I can be an exec sponsor. I knew how to go from hello world to hello revenue. So if I got into an exec briefing, I didn't waste everyone's time showing them the latest feature of Kubernetes. Doesn't make sense. They want to know how these tools come together to lead to actual app impact and outcomes. And so I matured there and I also got smart. You got to go to other teams and you read the OKRs, right? So another team might say, "Hey, Kelsey, we're really trying to get more adoption on our metric stack." And I remember the first thing that I started implementing at Google was a thing called empathetic engineering. So you have a lot of smart Googlers. These people are brilliant. I mean extremely brilliant to the point where the hardest problem Google had in my opinion was what to build. Not how to build it, what to build. They could do that. And in some cases you end up with like five messaging systems and but the thing is what to build seemed to be the most pressing problem. And so the one thing that I tried to do was like how do you convince other engineers, you know, their manager, how do you get them to trust you? And I started these empathetic engineering sessions where the first one was like get the Kubernetes team in one room. All of them engineering off-site. I want you all to install Kubernetes but you can't use any scripts. And remember these some of them are distinguished engineers and principal engineers. Some of them worked on Borg. Some of them are just the original creators of Kubernetes itself. And it was so fun to watch them struggle because it's like do we install Docker first? What version of Docker? Can we put this on your bunt too or does it need to be Red Hat? And so an hour goes by, teams of four are like, "Nah, man. This is doesn't work." I was like, "Great, you all can stop. I'm going to show you how I would do it." And of course, I know how to do this because I get to prepare, right? So, not a knock against them. I'm just like, "All right, Debian, tune the kernel this way. Put Docker on there. Put FCD, put the API server, put all these things. There you go. That's how you do it." And I was like, "Yeah, I mean, of course, you had prepare, of course." And so, the question then was from an engineering perspective, how will we make this better? And then people are like well if we had OS packages this could have been appit install and we could have just used local machinery like that's a good idea. Someone was like we will make that happen. Another person was like even if you had those packages though you still need to know what order and where the config files go. So coupube admin was born which was a command line tool that gave you a procedural thing. But the other thing that I remember from my career was I don't want just a tool that abstracts everything from me. I want to know how it works. So I wrote the guide Kubernetes the hard way. And that guide is what I used to help teach people at GitHub in the early days pre-Microsoft how to like run Kubernetes on bare metal and and walk them through that guide. And so it was that empathetic engineering that helped me make a huge impact on cloud because I can go to every team, every org. And instead of guessing what their road map should be, given someone who had spent time in the field, given someone that had this enterprise background, hands-on experience across lots of tooling, I knew where people were coming from, I would say based on where Google sits in the landscape and in the competitive landscape, given what our abilities are and what our customers need, I think this is it. But I never said it that way. I would get everyone in the room.

Host: You you would have them discover it. have them discover it. So you and nudge them. So you you you knew what you think the key the most important problem areas were for example and then you orchestrated a session to help people. I try to always know the two things in multiple product areas that would make the most impact aka revenue that would get adoption and so and then launches versus landings. You would put all these things in motion. They would land at different times. So launches is we ship the thing, people have a big celebration. I know that doesn't matter as much as the landing. People are actually paying for it. So when I got good at that cycle, the promotions were a little predictable because I would be able to make impact, right? Sometimes you build prototypes and sometimes you would contribute to things like cloud functions. But the overall goal though was to use every skill you had. So working with PMs, we got to add Go support to cloud functions. We're the team that made Go. Amazon has support in Lambda for Go. We need to do this. And also, I have a keynote at the next Gopher Con and I'm talking about serverless. And I have two choices. I have to use Lambda or we can use Go Functions. And so, we sprint. We get it done. We get all the Googlers reality. We get it checked in. It's in alpha. And I tell the PM, "Hey, you want to go to Gopher Con? We'll launch it on stage." We launch it on stage. We tell people how we designed our worker, how the internals work. and then they can just come sign up. That was the trifecta. You do the work, you have the vision, you execute, you launch, and you land. Land. And the landings compiled over seven years is how you go from like I think maybe L5 to distinguish engineer L9. That's like four promotions. Yeah. Four promotions, seven years. But it's interesting because a lot of times when you ask someone on, you know, how they got promoted, let's say, yeah, four four times in seven years at a place like Google, I would expect just being naive that they will tell you like, okay, this is how I planned or like this is what you need for each level. But sounds like it's it's a very different you just had landings that that created impact like and and you were focusing, do I read it correctly, that you were not focusing on your promotions or the next next level? You you just wanted to, you know, do the best work that you could.

Kelsey Hightower: Oh, no, no. I was focused on your next levels because that was the goal. Because think about it, if my whole career, I've always tried to acquire the skills and make the impact so I can move to the next level. And so at Google, the levels were expressed as promotions. And there was a point in time in the org that I was in, there was no level seven for an individual contributor. So now we have to make a level seven. And then once you get to a level seven, now you kind of pave the way for the other people that want to come up through that icy path. And then level eight isn't the same as level seven. Level eight isn't the same or level 9 isn't the same as level eight. And yes, there is a formula to some of the promotions early. So from 3 to four, four to five is a little formulaic. You have a ladder. There are things that are expected of you. And the decision making on those type of promotions are localized. Meaning your manager, maybe a director. But then outside of that though, when you start to go higher, it's now expanded where there's now other teams that have to understand your impact in a way that can't be biased by a local team. So if you think you've made a lot of impact and then people across the org do not agree, that allows you to really throttle what impact means, right? Because if it's just your team, I really like this person. Now everyone's a distinguished engineer. And so I think Google did a really good job of saying, look, it was okay to be like L5 or six for your entire career. Yeah, that's their terminal level. I think it used to be L5, now it's L4 actually. Okay. But they moved it back cuz I I think L5 has gotten a bit tricky and they don't want to fire really good L4s. Exactly. So, and I think for a lot of people, you can follow the formula and get to where you need to be. But I was like, at the time, I think there's like a couple hundred distinguished engineers. So, as an entrepreneur, if you show me the top of the mountain, I want to get there. So, then you were targeting, you're like, all right, how can I get to the next one? To the next one. and impact was the name of the game, right? Yeah. And I guess the only thing I would probably add to the moment or to the situation was I wanted to do it authentically. And so there was one part of the time I remember I didn't get one of the promotions, but I was getting promoted pretty fast. So of course people like, "Dude, you need to actually make an impact before you get promoted again. Stop it." And that was good feedback. But I remember I took a chance. So there's a pattern to doing a promotion packet like structure. There's examples and you go through all of this stuff and you get feedback. I remember one year I was like, I'm not doing that. This year I'm just going to talk in like the first person. Hey, I'm Kelsey. I work on these things, not these things. These things are important to Google. So, here's exactly what I did, but more importantly, here's the people I brought along. Here's the teams I've impacted and here's the results of this. So, I did this project and here are the results of that. And I'm talking in this way like I'm ignoring the process. I don't really care about the template. And as someone who at that point in time was on some of the promotion committees where we're looking at the promotion packets and making a decision as a team, I decided to write my packet for those people so that when they got it cuz look, if you're having to do a read a lot of these, it's it's hard. You're like, "Oh man, they're all so dry. Everyone's being very safe. Everyone's only telling me what I want to hear. I don't even know the person from after reading this whole packet." I said, "I'm not going to do that. I want them to see me as if I was in the room advocating for myself. And I remember getting feedback on that package like, Kelsey, this is like, come on, bro. This is, you know, things. I was taking it seriously. It wasn't I wasn't making a mockery of it. I just wanted to make sure that they understood what I was doing and I was aware that I wasn't just trying to play the game. I was trying to approach this process authentically. And I got promoted off of that packet. And look, it could just be sometimes the work sometimes speaks for itself, but a lot of times people can't see the work if it's not presented correctly. And so that kind of slingshot I mean and the reason why I tell that story is that every distinguished engineer doesn't get there the same way. Everyone thinks like, oh, how much code did you write? What complex thing did you build? And for me, I think it was the impact on Google Cloud's culture. The empathetic engineering thing became an official thing that they used to onboard other engineers. It became things that had a whole team behind it. They used to go give them and a product manager that would evolve the program and integrate into HR the philosophy around other engineers saying, "Kelsey, we just did an empathy session. I want to show you the results of it because we're about to ship it." And then engineers started really thinking about the customer. I mean, Amazon was always known for that to some degree, but to help Google get on the same page, I mean, I'm pretty sure other people had an impact, but there was a direct line of impact from those type of programs and also me diversifying, moving away from Kubernetes into the serverless realm, moving to the world where you're helping out the Postgress or the Spanner team, add a Postgress interface, the Go team, getting a little bit better with cloud, just making other people successful around you is one of those things that helps you become distinguished. is the impact, the ability to influence.

Host: Before you got to the distinguished level, you you shared a story about Microsoft in an offer. Can can you tell us about that?

Kelsey Hightower: Yeah. So, I'm one of those people. I don't like ultimatums. That's it's hard for an ultimatum would be you're working at a company for a couple years. You're doing a really good job. And I'm just going to make up numbers here just to protect everybody, but just say you're making $100,000. And for a lot of times, you could be happy with 100,000. And so a company says, "Hey, we're hiring." And you go there, you do an interview, and they say, "Look, we're paying $120,000." At that moment, you are actually worth $120,000. Cuz you now have evidence from the market. The $100,000 you currently make doesn't look so good anymore. You can't unsee a job offer for 120. And so now you have to make a decision. You can commit to that. Leave your job and go make 120 and start over. Or you can do the ultimatum thing. If you don't pay me 120, then I'm leaving. And that puts everybody in a weird predicament because sometimes it doesn't have to be that adversarial. Sometimes it's just this is evidence that, hey, look, I want to advocate for myself. I know we're not in promotion cycle, but I believe I'm worth 120 and I would like to have that conversation. And someone would say, hey, well, you need to go prove it. It's like, well, I have an offer if you want to see it, but I would rather be here. And that can turn into a you went to go look for another job. It's like, oh my god. So I'm stuck. They can have weird dynamics, right? Especially with your manager or or your management chain. Exactly. So I was always weary of that, but I also knew that in business, that's just how it goes. And so I'm really at the peak now. I'm thinking like early retirement. I'm probably can get out of here at what 55 60 if I continue on this pace. Yep. And then Microsoft is like, "Hey, come through." And look, I never had an executive recruitment process. So by this time, I'm probably considered an executive at Google. Once you hit like L7, you're kind of considered like an executive. What What is an executive? So an executive just means that you probably have uh an admin to help you with some of the tasks you're doing. You're probably going to be asked to be an executive sponsor of things and programs. So if a team wants to have a program for something and there's going to be a budget for that, you may have to help oversee that program. So they need executive sponsorship. Another engineering team knows that they're going to need budget for something and they need someone that has a little bit of political capital, a little bit of weight in the organization to help endorse them. So that can be executive sponsorship or you might be assigned to one of the largest c customers the company has. But that executive set of duties means that you're going to be making impact above and beyond yourself typically to support other parts of the organization. So I grew into that at Google. Microsoft was like, we want you to start there. And here's the thing, I didn't think about my role that way there. I'm still the old Kelsey from Coro West days, right? I'm just doing well here. And I wasn't going to interview at Microsoft. I'm like, for what? I'm at Google. I mean, I had a lot of freedom at Google. I was making impact at Google. I had a good reputation. And so, I'm like, "No, my trajectory is fine. I'm not doing that." And to be honest, I don't like Windows. I I didn't like Azure. I don't like .NET. I like GitHub. VS Code is nice, but my whole career, the majority of it, has been rooted in authenticity. I've been working on the things I actually like and care about. this would be one of the first times maybe outside of some of the enterprise roles that I'm going to go work on a set of technology that I wouldn't use on purpose. And so I went there and I met the Microsoft team. And the weird thing though is I had a recruiter and they swapped the recruiter out and it's like, "Hey, our mistake." I'm like, "What do you mean mistake? This person was super nice and they asked for a resume and I didn't have a resume cuz it's been seven years cuz you had to create one." Well, it's been actually 15 years at that point since I ever had to show someone a resume. So, I'm like, "Reme? Oh, man. I haven't made one of those in a long time. I'm looking online for a template. Like, what's the style of résumés these days?" And so, I'm like trying to figure out a resume. I'm like, "See, this is why I don't waste time interviewing. I don't got time for this anymore." And they swap the recruiter out and it's like, "Hey, sorry about all that. You we don't need no resume. Sorry, that's not this kind of process. What we want to do is make sure that you meet the right people who represent Microsoft." I was like, "Okay, that's different." So, I get on site. I'm like, "Okay, what kind of quiz questions are we going to be prepared for?" You're not doing quiz, bro. You had commit access to Kubernetes. You wrote the book on Kubernetes. You're leading these things. You helped start K. Come on. Like, you have a Wikipedia page. Like, there's We don't need to try to figure out whe you made the impact. We can just look at GitHub. There's evidence there. And so, I'm meeting all these leaders. They're bringing all these people who are behind the scenes. I remember meeting Scott Gunry for the first time. And I didn't really know who he was because I just didn't know how I didn't know the lineage and the history of of Microsoft. And you know, of course, I'm looking up who these people are and and I was like, "Wow, this is they're courting me. Oh, this is what that feels like." I mean, Google did it a little bit too, but not like this was like I'm I'm looking I'm looking at the these are the executive directors of this is like reverse interview. They're not interviewing you. They're trying to sell themselves. You know what? And that's the thing that I appreciated a lot because they were like this is your career. You built a fantastic career over there. We can't ask you to throw that away without understanding what you would be doing here. And so it was like you can if you want to understand the business if there's someone you haven't gotten a chance to talk to just give us a name. And I'm talking to them and the one thing that I really respected about Microsoft was I think by that time they had also acquired GitHub. And so they had a big vision for themselves, a lot of diversity. And I was like, "Okay, there's a lot of opportunity here." They're also all in on Kubernetes. They had just acquired people like Das and Brendon Burns is there. So I was like, "All right, Kelsey, you can come and make an impact there. There's room to grow." I'm like, "All right, I might do it." All right. So I'm glad I did the interview. And I get home and it felt like that time I doubled my salary. I told you on the way home and I remember I get this email from Satia, the CEO of Microsoft. And I'm like, man, he wrote this nice email, Kelsey, that I heard you had a good experience with the team. Remember, I did the interview at the Microsoft headquarters, right? So, hey, heard really good things from the team. Just wanted to let you know, you know, you're going to be respected here. We're going to support you as a team. I'm like, damn, support as a team coming from the CEO. And the offer was like a PDF. It's an attachment. So, I read this thing and so number one, what an honor. This is the CEO of Microsoft. Yeah, so many more important things to be doing than to be emailing me about a role. And I open a PDF and as a like very often in your career there's a zero get added to the equation. And so you're looking at this like I didn't even know that they do that. We know that it happens, but the person that graduated from high school in 1999 that chose the A+ certification didn't know that was available even while I was at Google having all this success. And Google paid me pretty well, too. But I know you can add another zero still. And so I'm like, whoa, this is this is crazy. And I'm like, wow. So I um I showed my wife and she was the one that said, "You should just go interview. Like put your ego to the side and let's go see what's out there." So, shout out to my wife. And so, I get the PDF and I'm like, okay, this number is perfect. Honestly, I don't know what to say, but let's just just find out like, is this really the only number? So, I remember given a counter like, you know what, I think it should be this. And the funny thing is Microsoft counter back higher. So, we're not playing around. I'm like, oh, whoa, now I understand that I don't understand this part of the game. Yep. And so I have this offer and I knew that I wasn't going to go interview there if I wasn't serious about taking it. So I was serious about going to Microsoft been almost 6 years at that time at Google and I went to my manager. I had the same manager which is legendary at Google. I had the same manager for 6 years straight even with all the reorg manager the whole time. He was a director. So even when I got leveled up they allowed at least someone one level above still to report to a director. And I had a such a good relationship with them and I told them what happened. I said I wasn't looking and they asked me to come in. So I went in and I'm going to take it. Number one, it will be financially irresponsible to not do this. So that will be the driving force and also I get to stretch myself in another way and see if I can make an impact again. He's like, "Okay." Um, but it was no ultimatum. I was like, "I'm leaving." And he's like, "I'm just curious what Google would say." I said, "No, not great. I don't want to do Ultimatum." Yeah. You don't believe in it? I mean, I do believe in it, but I didn't want to do it. You don't want to do it? Yes. I didn't want to do it cuz you understood the dynamics. The dynamics, especially I in in many ways, Google had been really, really good to me in every facet. So, it wasn't ultimatum time. It was more like I've earned it. And he said that too. I gave him the PDF and he looked at it. He just started smiling and like, "Oh, wow. Whoa." And then he said something that was really dope. He says, "I want you to know you're worth every penny of this." And I was, "Oh, the whole That's That's a game changer because he knows me like no other person I've ever worked with. No one knows me as well as Greg does. He knows my strengths. He knows my weaknesses. He knows my ambitions and my motivations. He knows all of them. For him to say that, it was important." And so I'm thinking if Google wanted me to be at that level, they would have done it already. But also, pragmatically speaking, no one really knows your situation. Like all these people have thousands of people they manage. They're not targeting you or being mean to you. At least in my case. And so maybe they thought I was making that kind of money already. Maybe they thought that I was already a distinguished engineer. That how would everyone in the or know? They got other things to worry about. So he presents the thing to them and I think within hours they're like this is no problem. Like hey here's the you know don't worry about that number. Here's a bunch of stock and all these things and I'm looking like whoa whoa. So now I have the money and I'm at the company I want to be. And you didn't do an ultimatum. I didn't do the ultimatum. So I felt good about the relationship. I didn't feel like um there was going to be some retaliation. I had no fear about that. And I continued to be successful. eventually got promoted to distinguished engineer at Google told the Microsoft team no but the one thing that people saw from this I didn't talk about this at that time but I did do a tweet and the tweet was different company same team a lot of people's like what does that even mean different company same team but people were you know retweeting it and they liked it and people oh it's because of the community stuff and how the different people in the community work together and that was part of it but a big part of it was that moment that Microsoft got me the biggest raising my career at Google and about 3 to 6 months later I was in San Jose and Satia was there and his admin is like hey um Sate would like to just you know meet me with you I'm like the CEO of Microsoft got time to do anything so I'm in San Jose and then I go to this hotel and the admin meets me downstairs like hey Santi kind of ready to see you now and I'm going up and now he know I'm coming right like let's just say the meeting is at 1:00 He knows I'm coming for like days and then you go into the hotel room and like the the doors are open overlooking a mountain range and Sati is sitting there overlooking the mountain range like a vanity fair photo shoot and I remember before on on the plane ride there I'm reading his book hit refresh and I remember the opening chapter he talks about starting Microsoft as a developer advocate and now being CEO of Microsoft advocating for the soul of Microsoft because he had been there so long. You saw the birth of cloud and all these things. I was like, "Oh, this and the book was actually a pretty good read about his trajectory in the industry and his time at Microsoft." So, now I'm done with the book and I'm about to meet him. So, I have all this context in my head. So, when I walk through, I felt like I had this rapport with him for some reason because I read the book and then he's looking out over the mountain range and I walk through the door. was like, "Sate, why are you sitting here like this? You knew I was coming, so you're just posing like we're doing a photo shoot." And he just started laughing. And so the tension, at least for me, was way down. And we had a discussion. I won't repeat everything here, but he said something dope. He said, "We were sitting around at the table and we asked ourselves, what executive did we want that got away? That means that you still were in the minds of of at least some people." He's like, "You were on that list." And uh we had a discussion and he was very transparent and he said um we gave you a good offer. We think we gave you a good offer and at that time Thomas Karen had just come from Oracle. My personal opinion I liked his leadership at Google but I can understand why some people were afraid of the Oracle DNA being brought to Google. And I think maybe some people in the industry felt like oh this is a moment to go and maybe poach a few people that didn't want to make that transition. and he said something like, um, we gave you a offer as if you were running away from something and we should have gave you something to run towards. And I was like, damn, that that's poetic. And so we wrapped up that meeting and I really felt like, man, I actually belong to the industry. It wasn't like I'm a Googler or Kubernetes person. I really felt after that moment that I was an industry person. So I was very comfortable at that point retiring within the next year or so.

Host: You talk with Sachio Nadell and about a year later you retired. How how did that thinking come up to because you were mentioning that you kind of had your retirement. You were thinking about your kind of endgame or exit game. ... (Host jumps into the AI query towards the end) ... One one thing that a lot of experienced software engineers are worried about right now is all the AI we're all using AI agents. They do generate code really quickly. Uh which is something writing code used to take a lot of effort. It took a lot long time to be good at it. Now with code reviews, there's now more tools coming in and some people are like worried like okay like what is happening to my profession like the craft of writing code seems to be something that we can offload and more and more people are offloading including prominent people. What will this do to software engineering and what advice would you give to people who are experienced software engineers? They're a little bit worried because it's it's a big shift. they still want to, you know, like be the whatever great engineer will look like in the future, but what steps might be able to take and especi especially like your your take because you're I think you're pretty grounded in just looking at this from a vantage point that some of us are not.

Kelsey Hightower: I think as a software developer, the first step you have to do is have a bit of reflection. For the last 20, 30 years, you have been automating a lot of industries away yourself. all those programs. I remember seeing an maybe it was like a diagram of every device that has been replaced by the iPhone. The radio, the calculator, the compass, all of these tools people used to buy individually. The top 30 of those electronics from the last 40 years are all in your iPhone. All of them. That means some electronic makers have gone out of business. They're gone. You did that. Not in a malicious way, but you were part of that. And so the software developer has been glorified for a very long time. The internet some people would say caused the downfall of magazines and newspapers because of the convenience of having a software approach. And so you have been part of the change to other industries and other people yourself. What did you think about that? Did you even think about it at all? So let's not be surprised if you find no sympathy from all the other professions that you've helped force change upon. So I think that's step one. You have to go do that reflection because if you don't do that reflection, you won't know how to behave now. You're going to be complaining and people are going to look at you crazy because where was this empathy before? You might be very excited about this and not realize you're only excited because you're in position to benefit from this. So if you work at an Anthropic, of course this is the future because it's in your hands. If you're at Nvidia, of course this is the future because you will be selling the picks and shovels. And so you got to ask yourself, why am I excited? Now what I don't want you to do is necessarily feel guilty about it, but I need you to see the big big picture. It's going to help frame everything else. The second thing I want you to do is ask yourself, what was my job? Remember, there was a point in my career where I was lying to myself. I thought my job was to be the less best Linux administrator ever. You as a software developer, you may have thought your job was to be the only person in organization that can write code. And since no one else could do it, you were safe, right? And so you didn't learn any other skills, networking, product management, design, talking to customers. Nope. All you had to do was write code and you were safe. And you probably made more money than everybody and you were fine with that. Now you got caught. The only thing you were good at is now been commoditized. And again, you did this to others. So let's say the vision you have for yourself is only in this very narrow realm. You're going to be very afraid of this trajectory because all you know is software developers write code. That's it. Some software developers still don't write test, still don't know how to deploy anything. And so they are really afraid because they can't see any other way that this plays out. Now if you're a full stack engineer, you're probably like, man, there's so much more than just writing code. You have to do architecture. You have to do design. You have to do so many other things that I love clock because now I can focus on those things and I can use these tools instead. So I can see why that person would have that perspective. Now, I understand why that full stack person has a perspective of watching the same people commentate that the code generation piece replaces everything else. They're going to be like, "No, you don't know what this job is." It's way more than just writing code. Writing code is the last step. If you're a security engineer, you're probably like, "We never figured out security for the pace of the current enterprise." The one before. Yeah. like everyone thinks they're moving slow. I remember I took a security um training thing and most of them aren't that good because they can't go super deep. They just tell you, "Hey, here's how to avoid fishing. Here's how to not leak information, adhere to various laws and things like that." And then they said one thing this time that I learned that was pretty good. They're like, "What's the key to protecting yourself?" And they say, "You know what? Just go slow." A lot of attacks are, "I'm about to board a flight and let's say you're an admin or you're a VP and the CEO texts you right now. We need to wire the money to Oracle to pay for the license. They're going to cut it off right now. This needs to be done immediately." You look at your phone, it's definitely from the phone number of the CEO. You have a good relationship. Everything looks right and it's moving fast. So, you're like, "Man, I'm on a 10-hour flight. I need to do this now. Turns out the attacker knows you're about to board this flight. They've seen all your previous text messages. They know how your manager talks to you. They know that you've moved fast in the past. And so they now are primed to get you to do the exact same behavior again. And you could be the VP of security. So you should know better. And sometimes that naive confidence will make you feel like I'm obviously not being fished. I've done this many times. This is definitely the CEO. Who would know how we actually operate? And who would know that I can actually do that? So what do you do? You make the transfer. And just like that, 10 million has been wired to the wrong place because you moved fast. It wasn't because you were not smart. It wasn't because you were not productive because in this case, you were productive, but you did the wrong thing. So when I think about code, there is value in having a healthy pace. Let's say you're an insurance company. You sell insurance. Hey, make model. How old are you? Have you had any accidents? Okay, here is your insurance for the year. Simple, very simple thing. If you're an insurance company, that's all you are. You're kind of close to being done. Now, you could say with JAI, we should get into payments. We should compete with Door Dash, right? We have all these tools. Let's go and we could build it. Yeah, we can build it. So, but the thing is, should you just build it? because you can and the answer is typically no. So we usually optimize ourselves as humans around the pace needed for the task and when we don't need to do that work anymore we move on to something else. So now I think what we're going to end up with is people not realizing a lot of this stuff we were doing in software engineering was decision-m what database to use what schema should we really collect someone's social security number or should we avoid it not yeah I can write code to parse a social security number really fast like no no should you even do it and so when you write code it almost makes you slow down again because there's been times where I thought I had a good design that's that phrase writing is thinking So is writing code. So as you're writing the code, you're like, "Hey, wait a minute. This loop is ridiculous, right? Not only is it going to make the computer warm, this is not the right thing to do. There's a better data structure than the algorithm that I'm using." So then you stop and say, "Hey, the data structure is wrong. We need to change the architecture from the top down. So decision making sometimes does benefit from slow. And when I'm saying slow here, we're not talking waterfall 6 months. Yeah. No, we're just talking about maybe one more day before you go at it. And I think some of us are going to miss that part because clot spit it out, ship it. Yeah. And that's also one one thing that you always have the the more experienced generation be worried about the young generation. I remember when I joined the industry, ReSharper had come out. ReSharper uh and the experienced old guard was like, "Nah, you're you're not a real developer if you use ReSharper cuz you know you're not going to learn the library and you need that and and you know, like that's what makes you a real developer." And then I remember when I was now five or plus years of experience and stack overflow started to become big and I was like nah you don't want to go to Stack Overflow because you're not going to you know learn the real thing. So but now what the current old guard is saying which is you know we're I'm I guess I'm part of it is like well if you use AI you're going to miss learning the basics and when you have learned the basics it's so much easier to use AI. And I wonder if we're just repeating the same the same mistake as as the previous ones did which is the new generation usually figures out the tools they understand how to do it or are we rightly concerned that some people who are coming into this AI native they're now learning to code they they can jump through so many layers that they will just not you know see what's under understand or are we just like making assumptions that might not be true. Here's where I think we can it can be right on both sides. Do you need to learn how to code to make an impact in this industry? The answer is no. You do not have to. There are some people who use these no code platforms where they drag and drop and they produce a really good app. There are some people who have built a consultancy business by just using Wix, right? They go there, their website actually looks pretty good and so they got really far with that. Now, for what they're trying to do and accomplish in life, they'll probably be fine. But let's just say you are a software engineer. And the idea behind software engineer is not limited to just producing apps. Software is the interface between hardware and things people want to do. So there's a whole bunch of things you need to learn. So if you want to be that type of software engineer, you got to learn hardware, too. If you don't understand hardware, you can never work at that level. And look, if that's not your job, then so be it. But you will never have that creativity. I remember seeing someone was like, "Hey, you can do isolation without a VM." I was like, "How would you do that?" He's like, "Oh, because when you boot the kernel, there's a thing you can do before the kernel loads to isolate it in a way that you can lock down processes." The only reason why this person knows is they know the full boot sequence from firmware to switching to the kernel and the tricks you can do in between. Now, for me, that doesn't work at that depth. I'm thinking there's only virtualization, CPU isolation, things like G Visor where you intercept system calls. But never did I think about the boot sequence. And so yes, you can get very far, but as someone like we applaud every version of Opus that's released or chat GBT, but there are versions of yourself that get deeper from these new trainings. So no, you don't have to. But if you ever want to get better at anything, and sometimes that depth, that nuance is the thing that leads to an invention, right? If you know how a compiler works, if you know how memory management works, that might give you enough information to say, "Oh, I can make a new programming language." If only thing you know is the surface, you can't even imagine how you can create another programming language that is better fit for the task at hand because you never gone that deep. I'm not saying everyone wants to do that. So I think it is fair to say all I want to do is come in get a job and if that job can be done by using AI tools I think the side effect of that is then that job will be commoditized. It has to that's just the way it's going to go. But I've always seen myself for my entire career I want to learn more. I want to go deeper. I want to go so deep that I can create. And I think a lot of people who are doing this, the reason why we're having this reaction, some of us, some of us, part of our careers have been the creation part, there is no spec for this. There's no protocol for this, but we're going to make it work. A lot of people that are doing like the reverse engineering, the hacking, there's like there's no framework for what I'm about to do. I just know how memory works and I don't care what your security tool does. I will make it do what I want it to do. You need to go way below the surface. And so for a lot of us that are saying this, we know the value of the fundamentals that lead to the other stuff. And so if you tell a next generation, oh, you don't need to learn these things. It's like that may be right in the short term, but we know for a fact your career will be limited and that may not be a problem and you have to decide. But make no mistake, if we put this much effort in training the model so that it can spit things out, you better make sure that you are willing to train your own model. So my advice to people would be and maybe we should talk about it different. Maybe we shouldn't have so much fear-mongering around it. Maybe we wouldn't should put it a versus this. We should just say great artists tend to know how to mix colors and it isn't your benefit to understand the primary colors so you can mix them to get the other colors. And it's a superpower, right? So you don't have to go buy, you know, imagine an artist trying to go buy 16 million colors and put them on the desk because they don't know how to mix colors. If you teach a person how to mix colors, like you can get any color you want. And I think that's the way we had to approach it. It's just another skill that if you had it, you might just unlock some creativity. So I encourage you to learn it.

Host: Kelsey, thank you very much. This was just an amazing conversation.

Kelsey Hightower: Awesome. Thanks for having me.

Host: I will admit I was glued to my chair for the whole of a conversation. Apologies that it took this long, but I hope you agree that this specific one was worth it. Kelsey's past was just so unlikely. He's someone who was raised by a single mother, dropped out of college in favor of installing DSLines doortodoor at 19, and still ended up as distinguished engineer at Google Cloud. Only a few hundred people who hold a title at the company. And he retired at the top 3 years ago when he decided that he no longer needs to work for others. What an inspiration. One thing I took notes on was when Kelsey said how every job is an interview. When Kelsey was giving the Gercon talk and PXE booting Cor from his slide deck, he had no idea that the Coros team was sitting in the audience and that's how he ended up at Coros. When he was contributing to Puppet at nights and weekends, he didn't know that James Turn would walk into his office one day and recognize his name. He just kept showing up and doing the work in public. It's a lesson worth remembering. Do the best work you can at work. it might unknowingly be your job interview for your next step in your career. I also found the Microsoft offer fascinating. Kelsey did not use the offer from Microsoft as an ultimatum at Google, even though he could have. He just told his manager the truth and then Google matched the offer. Obviously, at this high of a level, there's no universal composition negotiating advice that always works. But being a straight shooter with high integrity is something that is good to keep in mind. I was also inspired by Kelsey's focus on minimalism. He treats money as freedom tokens and made sure that his lifestyle never inflated with his salary so that early retirement was always a real option, not just fantasy. Finally, Kelsey's AI takes his point is grounded and pragmatic. AI does not change what software engineering is actually for. The job was never just to write code. The job was and is to solve human problems. The engineers who understand this are going to be fine. Do check out the show notes below for related the pragmatic engineer deep dives on Kubernetes and other related topics. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show.