What does it take to thrive at a fast growing company as a software engineer? Charles-Axel Dein was engineer number 20 at Uber and saw firsthand as the company grew from 20 engineers to more than 2,000 in just 5 years. He also happened to hire me at Uber and was my first manager at the company. Today, Charles works at Cloud Kitchens and has been maintaining the wildly popular GitHub repository, Professional Programming, for 15 years.

In this episode, we go into what it's like to work inside a rapid growth startup like Uber during its high growth times, how to be a standout software engineer, including tips on how to lead projects efficiently, and the importance of the "shift stuff and lift people around you" mindset. Charles's personal productivity tips, including why he uses flash cards to memorize things like architecture patterns and data science methodologies. How Charles thinks AI will change software engineering and why migrations is the best use case Cloud Kitchens has found so far for AI coding tools. If you're working at a fast-paced startup or plan on working at one and are looking for tactical advice on how to do well in such an environment, this episode is for you.

So, Charles, welcome to the podcast. It is so nice to be back. This is not the first time we're sitting in a situation like this. The last time was a bit more stressful; it was an interview at Uber. You were the hiring manager and I think you were the final interview in a series of interviews, and that's how I joined. That's how we started to work together.

Yeah. And actually right before you joined, I don't know if you remember that, but I called you to ask you if you could change stack. I think you were hired as a back end engineer and I asked you to do iOS because this is where we needed you, and you said, "Yeah, let's do it."

It was Android. It was a very interesting interview situation. I was doing iOS development back then. I did Windows phone before and I don't know if you know but my interview got messed up. I thought I would be doing an iOS interview and then they put me on a backend track. I did backend before, a few years before, so I was like, "Alright, let's do it." And apparently I did good enough on it, which was surprising. And then I was supposed to join as a Python engineer, and I hadn't done Python before, but obviously Python you can learn easily. I was like, "Okay, let me learn it." So I got a Python book, you know, Python 101, and then you emailed me saying, "Could you do Android?" What was typical of Uber back then to have this kind of hectic situation? What was the reason? It all felt very hectic when I joined.

Yeah, it was definitely very hectic. And it's interesting because when you're in those hectic times, you feel like this is not normal and I would much prefer a time where it's much more quiet. But then when it gets quieter, you miss the hectic time. So it was definitely that hectic. It's not the most surprising thing that happened when it comes to hiring. But it was a good hectic, right? You're growing super fast, so you have to ask people to be flexible. And I think that's what people look for in a startup, right? They want to see things that are very different, where they get exposed to a lot of different things. And that's definitely what you got with a panel for one stack, a background for another stack, and then you get hired and asked on your first day—I think it was close to your first day—to do Android. It was pretty hectic.

And at Uber, when did you join? You were very early.

Yeah. So I joined in 2012 thanks to the first driver operation person who I was working with before. And at the time, yeah, the team was about 20 engineers.

Like engineer number 20-ish.

Yeah, something like that. So definitely super interesting because there was no road map, very little process. So you're definitely discovering a lot of things as you go. And what better way to learn, right? When you are at a startup, things are very unstructured and you get exposed to a lot of problems and you make a lot of things up. You learn by doing, essentially. And it was a truly amazing experience.

And then you joined as a software engineer and then you made your way into engineering management, right?

Yes, that's right. One year later, Thuan Pham, who was the CTO at the time, asked me to step in. Best career decision ever. It was a really rewarding experience. And then since then, I've moved back and forth between software engineer and engineering manager. And I really appreciate doing both. I believe in really hands-on engineering managers.

And so by being that early, you must have worked with Travis Kalanick, the co-founder of Uber, right?

Yes. I actually still have some pretty good emails from when we broke features. What was interesting at Uber is that we built a product that helps people make a living. And so when there's an incident, it's not only a feature that is broken, it's also people who were not getting paid on time. And I remember this event where the payment process for drivers failed, and this was right before Christmas. So we got forwarded an email from a driver who couldn't purchase gifts for their kids. And I think that shows the responsibility that you have as a software engineer when you own things in production. It's not only features, it's not only code, it's also potentially people's livelihood.

I remember when I interviewed at Uber, you told me something similar, and that was the first time I kind of paused and thought, "Oh, I could actually work on something that has real-world implications in terms of people relying on it and it being important." And I do remember that whenever we did an incident review, it wasn't just about how many ad dollars we lost—which, at some places, that's what it is—but it was about how many people's rides could not happen. And behind that, there's a story, there's frustration. I've had it before when a driver cancels, for example, and I'm in a hurry or stuck in a traffic jam and I'm late, and that itself is frustrating enough.

Yeah. So right now I'm working at Cloud Kitchens, which is another startup that works in the physical world. And today we speak a lot about AI startups, and it's evidently a very important sector. But at the same time, all of those startups that actually change things in the physical world and have an impact on people's lives, I think people should also consider joining them because you can see the result of your work. So that's one. The physical world is a source of never-ending complexity and challenges. For instance, when it comes to delivering food, you have the optimization across space and time that is really interesting from a technical standpoint, from an algorithm standpoint, from a system standpoint, from a latency standpoint. You have a budget of time that is not infinite because the food needs to get delivered. So it's really interesting to get exposed to the physical world.

So at Cloud Kitchens, how are things similar or different to Uber?

You're right, the time of hyperscale startups is probably behind us. I think you did a lot of articles on this, the end of the zero interest rate policy. What this means is that the model of this hyperscale startup that grows super fast and hires extremely fast is probably behind us, and it's probably a good thing. I think the thing we do differently at Cloud Kitchens is we're much smaller in terms of team size and much more focused. And yeah, you're right, it's really interesting to see that there's a bit more humility, there's less money floating around, which leads to probably better decisions. I think at the time of Uber, there was probably a lot of waste in the way we hired, the way we built teams, the way we started projects.

I do think hyperscaling is behind us because there was this special time where there was so much money on the market and Uber was so good at raising money. You know, they raised billions for a physical product. For every single round, their valuations kept going higher and higher. We are seeing something similar right now with AI, but it's not in the physical world. Well, you can argue GPUs are, but I feel that's a little bit different.

Let's talk about what it was like to be inside, because I remember when I joined, it was crazy. And one of my first memories of how crazy it was is when I became an engineering manager, I asked you—I was an apprentice manager and you were my manager—and I asked you how headcount planning works. You told me it's like, headcount planning should work by us making a plan, making an ask, and doing it. But in reality, you keep hiring and when you hit your limit, you get more headcount because it just comes out of this black box and more headcount comes out of it. It's always been like this. And I was like... and you were like, "Don't worry, it doesn't make sense. Just keep hiring."

Yeah. Until it makes sense, right? And yeah, clearly there was probably a lack of financial professionalism and being diligent with the company's resources. But I think now, as you've described, with the end of zero interest rates, it's probably much more sound financial principles driving the headcount planning. But yeah, to come back to hyperscale, it's really chaos. That's what it is. But good chaos, right? Chaos where you get exposed to many problems. And if you're curious, which I think is a key quality to have if you want to grow, then you get exposed to so many problems at the same time, which means that you have to ramp up on those problems really, really fast. And you get to learn because the best way to learn is to try something, get feedback, and then keep on iterating with a continuous improvement mindset. So that was really fascinating.

Can you explain what you saw, how hyperscale actually meant in terms of the day-to-day, and then how it later eased into what a normal thing is? Because I feel a lot of people have not seen it. They will probably not see it. So it's good for us to describe what we saw inside.

I would say you remember it, right? It's I don't want to say a never-ending stream of incidents, but a lot of incidents, right? A lot of incidents where you feel like you're covering the immediate action items that result from the incident, but you don't have time to actually fix the fundamental architectural root cause complexity or some of the things. I would say yeah, incidents definitely, and on-call was terrible.

Well, I remember that we had a lot of enthusiasm through the door. I still think back how we could manage with so much. Our on-call was terrible, like we were waking up in the middle of the night almost every night. And we just tried to fix it, but it was just patches. But I remember that because a lot of new people were joining and people were very excited because from the outside the company was very attractive. They brought so much energy, and that energy definitely lasted for a good four to six months. And we had a never-ending stream of people who, just as you were kind of getting tired, someone would be like, "Oh cool, let's do this, let's fix it." And there was a mentality of let's fix this.

Yeah, the incident is really the best way to learn about your architecture deficiencies, right? You can look at the architecture on paper and you can see, okay, this component is probably a problem. But the best thing really to know the bottlenecks is to see what breaks. And especially in 2012 and 2013, every Friday evening I would prepare to come back home and then something would break and then it would be firefighting mode. And this is how you learn that, oh, Redis breaks after that, and oh, Postgres breaks after that, and oh, we have this queue here that is not dimensioned correctly, or we have this instance... and at the time, yeah, autoscaling wasn't really a thing. So really interesting, a good way to learn essentially about your system's efficiency, still to this day, right? That's a great way.

I'm not sure if it had to do with hyperscaling or not, but we had this rule that when you are deploying a system that seems like a medium-sized change, go to the Slack channel or the chat channel and let people know that you're doing it. Because I think we were kind of used to this could cause issues, so it's just giving a heads up that if you see something going wrong, just let us know. Again, back then we didn't have that good observability. In fact, we had to build our own observability; there were no vendors that we could onboard.

Coming back to the incidents and the fact that you had to announce deployments, this is why I think the most fundamental way to improve this is to decouple the release from the deployment. First, deploy your code and it's behind a feature flag, so nothing happens. And then you turn on your feature flag just for your users so that you can test, and then you can roll out slowly. This very simple approach works wonders in terms of providing stability because then you get into the habit of making sure that a deploy never breaks anything. You first release and test with your users, and then because you're the one turning on the feature flag, you are presumably following the feature and monitoring it with the product manager, etc.

I remember that we had an incident where someone on social media posted that something is broken at Uber. And I remember you were upset, and you were upset that we didn't catch it, that our monitoring did not catch it, and that someone had to post on social media. I remember what you told us, you might not remember, but you said, "This is the ultimate failure of us needing to learn from a customer that our system is broken." And this still stuck with me, this lesson of since then I remember like, okay, how can we make sure that we will know and we will not have to have someone tell us this is broken.

Yeah, frankly to this day, I don't think even on my current team, we still have to work on those things. There are three metrics at Cloud Kitchens that we always look at for any incident: time to detection, time to mitigation, and time to resolution. And the most important part for me is time to mitigation. It means how much time does it take us to bring the system back to a state where the customer doesn't see the impact. Resolution goes after. And if you waste like one week of time because you did not detect the issues, there you go. This is what you need to fix. And at scale, especially at Uber scale, being able to detect is much easier because you have such a high volume that if something breaks, you must have some kind of monitoring that will detect it.

One thing with hyperscaling is things are so fast. You keep hiring people, but you're hiring people a lot slower than your growth is in terms of customers, features, all those things. Automation becomes really, really important. You've always been a huge fan of automation. I remember at Uber you kind of built your scripts to automate this or that. What have you learned about automation, how it works, how it doesn't work? Maybe some things are not as obvious about it.

The most recent thing I learned about this is this article I read about the ironies of automation, which is a pretty old article but it's fascinating. And it talks about two things that I've definitely seen and I've made a lot of mistakes in this area. One, sometime when we automate, we replace user error with automation designer error. We are lacking context, we don't understand how the software or the process is used, so when we automate, we actually put in the wrong automation. That's one. Two, because you cannot automate everything, you usually automate the simplest things and you leave the user in charge of the most complex stuff. So that's the irony here is like automation sometime fails. So what can you do to prevent that? The simplest thing is to observe and get a deep understanding of what the process is. You have to do it manually. We say startups should do things that don't scale first. So do the things manually first. This will give you the business context, product context, operational context to then automate correctly.

One interesting challenge during the hyperscaling time was hiring. And again, most companies will not be in hyperscale mode, but every now and then you need to hire really, really fast. This might be because you get a big funding round or for other reasons. What did you learn about what works when you need to hire efficiently and you need to hire a lot of people, often times experienced people?

You look at the whole process, right? You look at every single step of the process as a funnel, as a pipeline, and you see where you have a leaky bucket. So from a metric standpoint, it's very important to have that. The other thing is you build a partnership with a recruiter. So many people don't do that. You have to build this partnership so that you have a feedback loop where your recruiter knows what you're looking for and makes sure that those candidates are sourced, are front-screened from a recruiter standpoint, and then get through the process. And then the last thing I would say is like every single step of the process, you have a continuous improvement mindset. One thing you remember we did at Uber, we do it at Cloud Kitchens as well, is pairing interviewers. You have two interviewers, and this way they can give each other feedback. They can train each other. There is only so much you can do when it comes to interviewing training. And so to pair interviewers, you can share feedback between the two.

Yeah, I remember the two things that we did and I don't think many places still do it. First, we had a weekly catch-up with the recruiter. So engineering managers and recruiters, we would do it. It's a bit like a team retrospective. And they would talk about things like what are their sourcing strategies, what kind of cohorts are they targeting, or LinkedIn has this new feature. And they would share tips and we're like, "Oh, pretty cool." Or talking about what kind of companies they're thinking of targeting and we'd be like, "Oh, what about this one? I just heard that there's some negative news coming from there, maybe people will be interested in joining from there." And I've never seen that before and it was really good.

100%. This relationship between the recruiter and the engineering manager is critical. And actually something that came from Thuan Pham, which you should have on your podcast as well. He told me that every morning he would go and check with the recruiters and say, "Hey, what are some of the interesting profiles you saw this week?" And it's a great way to have this feedback loop. The recruiter brings a lot of value and has their specific skill set, and you know who you're looking for. You know who your team is missing. So if you don't have this tight feedback loop, it's going to take a lot of time to hire.

What can you do when you're feeling really overloaded? I remember, and you were my manager, but there was a time where you looked pretty burnt out. We had performance season and a manager left the company, and suddenly their reports also reported to you. You had suddenly 30 reports and you had to do 30 performance check-in conversations. I remember that when you did the last one, a few of us more experienced engineers, we knew, we saw your calendar. It was full. And when you came out, we were there kind of clapping that you're done because it took like two weeks. And I remember at the time, you had 30 reports. You were expected to still get stuff done. I think you were still doing some hands-on work. And you could tell that usually you're pretty chipper, but back then you were not chipper. What did you do to get out of this situation or to push through it?

Yeah, I would say there's two things. There's one, personal productivity. It's a topic I've always been super passionate about. So I've always invested a lot. The first book I read on this topic was Getting Things Done by David Allen, which is a critical skill, right? The earlier you invest in your personal productivity, the more dividend it's going to pay. And I really believe in compound interest. The second thing is a principle that works wonders everywhere: divide and conquer. So yeah, you're right. I remember that time. You have 30 reports, you cannot do everything. So now you're going to have to find people who can take over some of those topics from you. So I remember asking you to take lead on a project, for instance. And it's a great way because you give responsibilities to people, you help them stretch outside of their comfort zone, and you're saving time. So it's a win-win thing.

I remember that you did this, and what I liked about it is you were very explicit about the ask. You were like, "I need you to lead this project, and I don't have as much time to spend on it. I'm expecting to check in with you once a week. If you need something more before that, let me know." And then I think you also went more specific: "I'd like you to make a plan on how you're going to do this, let's check in on it." And it was a way for you to spend less time, give more responsibility, and actually people liked it because people were growing professionally and they also understood the reason why this added responsibility is going to them.

This divide and conquer is very similar to what we do with software, right? We create an abstraction, we give it responsibilities, we design an API that is clean. One of my managers told me one of the most powerful expressions you can tell somebody is "I need your help." This is very powerful, very simple, right? "I need your help with this." And the employee feels also empowered to make decisions.

You were an engineer yourself. You managed a lot of pretty good engineers. Can we talk about a story of a standout software engineer and what made this person stand out?

Yeah, there are so many good stories. I'm not sure I want to single out somebody in particular, but some of the traits of an archetypically good software engineer. I would say there's a couple things. There's one, shipping. I've written three or four career ladders. A lot of career ladders in the industry, after senior, they overfocus on meta work: reviewing RFCs, attending meetings, influencing that, strategizing this. It's really difficult to understand what's going on. So the first quality, and we try to put that in the Cloud Kitchens career ladder, is the focus on building, shipping value, being creative, being an expert in your programming language, in system architecture. So that's one, shipping is really key.

So even at the above senior, like the staff engineer level, you still place focus on, you still need to ship things, you need to get into production.

Absolutely. And we expect, for instance, staff engineers to really find creative ways to speed up execution or achieve a 10x improvement in quality. That's what we expect. Not only reviewing RFCs or attending meetings. So that's one, shipping. And lifting, lifting people around you. This is critical. We are knowledge workers essentially, right? So you have to train people around you, you have to give a hand, you have to help, you have to have a good attitude. The best engineers that I've worked with have this amazing attitude of ownership, of taking a problem and not stopping at team boundaries.

I feel there's starting to be no real excuses. Because before AI, you could say it is hard to onboard. It's hard to get to know it. It takes time and effort. And I mean, it still takes time and effort to become an expert, but as you say, to unblock yourself, at an age and time when a product manager can open a pull request that might be accepted, I think engineers are necessarily becoming full stack with an expertise in their home stack.

Yes. Ship and lift. Yes. And I will add one thing: humor. Taking yourself lightly. It's really important. I mean, evidently not limited to solid engineers, but I think humility and being able to self-deprecate is pretty important.

I feel that getting better at project management, managing your own project that you're working on, whether it's a one-person thing or a few people, it's a superpower as a software engineer. Because I think as software engineers, we think that our job is to write code, but actually our job is to solve problems and to ship solutions to problems that the business has, whether that be new features or products. And you become more experienced and frankly, higher paid when you can take on more complex things. And there's only so far you can go without project management.

Yeah, I would even go as far as saying that project management is one thing, but I would also recommend getting into product management. Because the irony of the software engineer's work is there's a lot of blame, a lot of things that can be blamed on the software engineer, right? Like a feature is buggy, hasn't been tested enough, there's an incident, a wrong architecture decision was made. And it's very objective how you measure. There is one thing that is relatively difficult to measure and that most companies I think don't do a great job at, which is making sure a product is doing the right thing and was the right strategy in the first place and is shipping the right value. So as a software engineer, if you don't do that, you risk wasting not days but weeks, maybe months of time working on a feature that is actually not going to move the needle for the customer.

What do you see? How is AI already changing software engineering, especially for the work that you're doing at Cloud Kitchens?

We did our own study. We found it was moderately useful. It is useful. I think it's particularly useful when you have a well-defined problem that you're trying to solve, like navigating the code. As a coach and as a trainer, as a manager, I use it a lot to review my documents. As a matter of principle, I never copy-paste text that was written by an AI. I want to make sure I stay in control of what I write.

Because yeah, writing is thinking, right?

Yeah, it's the same process. So I don't want to outsource my thinking to an AI. I would be really, really sad if that was the case. Every single time we have one of those revolutions, and AI might be the biggest yet, don't get me wrong, but every time we have those revolutions, the press and everyone speaks about replacing engineers. We're still there. You remember when machine learning really picked up, people were saying we won't need software engineers anymore. The truth is, it's the data scientist that was really a job role that was under a lot of challenges because of machine learning and ready-made models.

Your team is still hiring, correct?

Yeah, we're still hiring. I think it's going to enable engineers to do more, to focus on more interesting tasks. We talked about migration. I think migration is potentially a great use case.

We hate doing it.

Yeah. So migration is a great use case. For instance, at Cloud Kitchens, we had a migration and the team that was owning the internal platform put together a prompt that you could copy-paste to do the migration, and still review the code because there's always things that go wrong.

You've worked at a couple of fast-paced startups, Uber. You later worked at other startups. You're now at Cloud Kitchens. What have you seen people do to be successful at these startups?

I would say first, personal productivity helps a lot. So investing in this early in your career so that then you can hit the ground running really rapidly and you stay structured and methodical when you onboard at a new company.

For flashcards, what kind of things do you memorize?

So for instance, I memorized the whole Python standard library, not everything, but a lot of it, and it did help a lot in the beginning. And then you memorize like architecture patterns, data science methodologies, the standard library of your current language, or some issues you have, bash commands that you use consistently. Yeah.

What are some non-obvious recommendations to thrive inside a high growth startup?

Yeah. Not obvious. I talked about memorization. Using flashcards to memorize stuff. Very powerful. And second, I'm not sure if it's super non-obvious, but extreme ownership. So that means you feel like the team is yours, the project is fully yours, and really going out of your way to understand all of the dependencies and the overall context it fits in is really critical to be super successful. I talked about how the best engineers don't stop at team boundaries. They will keep going. Our industry, our work is mostly about building abstractions on top of abstractions on top of other abstractions. In reality, there's always leaky abstraction details, implementation details. So the more you understand about the stack you rest upon, I think the more effective you're going to be as an engineer. It also applies to your tools. It applies to your programming language. It applies to everything.

When it comes at working at high scale at fast-paced startups, a lot of people have imposter syndrome. Did you have this and did you see people have it and how did you work around it?

Yeah, you always have this. I think to this day, it's a good thing to have it, actually. We often talk about the imposter syndrome as a bad thing, but I think actually it's the engine that drives your curiosity and is what's going to move you toward essentially improving yourself and having this continuous improvement mindset. Dan Heller, which we both know, has a really good way to phrase this thing. He said, "I think imposter syndrome is underrated. A lot of talk goes into overcoming imposter syndrome. I say embrace self-skepticism and doubt yourself every day." What else to add? It's a good drive. It's a good engine.

What are some non-obvious reading recommendations you might have for engineers who want to get better?

Yeah. So, I would say read the fundamentals, not necessarily only the most recent technical books. For instance, one of the non-obvious books I would say is The Linux Programming Interface, which goes into what is the API that is exposed by the kernel. It's fascinating because not only does it explain the kernel and it's super useful because most of our stack is built on top of Linux. And two, it also goes into the historical technical decisions. There are some really interesting algorithms as well, like why did the kernel choose a red-black tree for instance instead of a hashmap for certain data structures. So it's really interesting, really fascinating.

I've also been reading a lot more philosophy lately. I mean, evidently software engineering is a very scientific endeavor, but there's a lot to be said also about the fact that we handle concepts. So it's a very conceptual work that is very similar to philosophy. And so for instance, I listened to your interview with Dr. Ousterhout about the philosophy of software design, right? And he mentioned that at the core of software design, you have this decomposition of problems. Well, it's a core of human thought. It's the core of reason, right? Reason is about distinguishing between different concepts.

Yeah. In fact, when it comes to, for example, designing a new system, you kind of need to go down a path. You can argue if it's a philosophy or not, if you're doing microservices or monoliths, but the argument can turn into kind of philosophical. There is no one right or wrong. And I guess at some point you need to go with one of them. It's fascinating how there are some parallels.

Yeah, when you're writing an RFC, you're making a case. So then there is a big component of logic. How sound are your arguments? So for instance, at Cloud Kitchens, in the competencies, we have a big section about truth, and we have one competency which I like pretty much, which is called QED, right? Quod Erat Demonstrandum, what needed to be demonstrated. And it focuses on how well do you make your case?