Netflix needs no introduction, but its scale can still surprise many people. But what is it like to work at a streaming company as a software engineer? I sat down with Netflix CTO Elizabeth Stone to get more details.

In today's conversation, we cover the unique engineering challenges at Netflix, including the learnings from 3 years of Netflix live. Netflix's engineering principles and why Elizabeth's favorite is yearn to learn. How Netflix has no performance reviews and what they do instead. How the company uses AI tools and why anime detection and analysis is a great use case that they found for them. And many more details.

If you're interested in understanding more about how Netflix works as a software engineer and what it takes to do well in the kind of environment they operate in, then this episode is for you.

So, Elizabeth, welcome to the podcast.

Thank you. Thank you for having me and welcome to Netflix.

It is so nice to be here at Netflix. I'm sitting in a director chair. It has the Netflix on it. So, it's inside the Netflix offices, which truly feels special. And it just reminds me that Netflix is an entertainment company at its core.

Yeah. A lot of magic happens here. Even in the tech teams, we take things like video very seriously.

A lot of people know Netflix, of course, from the video offerings, the films, the movies. Behind the scenes, what is the scale of the company from an engineer's perspective? How can I make sense of how large this operation is?

It's probably larger than people realize. Very often I'll get questions from people in my personal life saying well how many engineers can it really take to build the Netflix product? So first of all it takes quite a few when you think about how do you make the tech work so well that it's basically seamless in some ways ideally invisible because members just get to enjoy their experience.

But then we also as a tech team build tools and products for studio productions, our advertising tech stack. We build a lot of the developer platform capabilities and launch capabilities for games. Think about anything related to commerce plans, pricing, payments, partnerships. Those are all things that are supported by the tech team. So when you add that up, we have more than a trillion events that we're capturing every day between consumer interactions, things that are happening across products and services that support decision making. So it's quite a global enterprise at this point.

Some of the things that you mention I feel are kind of somewhat typical at large other large companies. For example, building payments, building ads, building some of the things. You mentioned things that I haven't heard in any other company which is for example building custom software or your production studio.

Yeah, it's actually very much part of our superpower that we've been able to bring technology to entertainment. We're one of the biggest studios in the world and we have an advantage in thinking about what are some of the problems we could uniquely solve for those productions. So, good examples would be things like our media production suite, which took something that was a fairly antiquated, slow, and expensive way for media files to travel across creative teams around the world and really modernized what that looks like.

So that if you have a production that's shooting somewhere in Europe and you have someone sitting in Los Angeles who's going to review the daily clips and footage, they're able to have those media files travel around the world, provide notes or input on that, have those notes travel back and be ready for another day of production just the next day. So things like that media production suite or other tools that allow us to monitor how are we progressing on some of the productions is something that's very novel from Netflix.

We also have a big presence through Scanline and Eyline, which is a visual effects studio that was an acquisition a few years ago that does really cutting edge research and technology for things that affect how we do data capture, visual effects, different ways to think about strategies that bring life to productions that wouldn't be easy just based on standard camera technology.

In terms of the engineering work behind that, what are some of the challenges here? Because what I'm hearing is you're saying you know like movie files etc. To me what the rings bell is like it'll be like large amounts of data probably I assume latency might be interesting challenges.

It's unbelievable scale when you think about hundreds or thousands of productions that are in progress at any given moment of time. So across all those productions you have media files which are also especially large complex and difficult to move. So think about scale, think about cost of both storage, compute and travel of that data. Latency in some cases, you know, it depends on the use case. For some cases, it's capturing media that is going to be reviewed in a way that can be the next day, but for other things, we've got media traveling for things like live productions that has to be essentially instantaneous.

The other thing I would say about some of the unique challenges in that space is the level of quality that we're trying to bring. So when you think about some of the challenges to think about how do you create very high quality images, videos, whether that's the content itself or the promotion of that content on the service. There's a lot of engineering challenges that come to meeting that type of bar.

The other thing that's familiar externally which I'm sure you've heard about is open connector or content delivery network. Which is extremely unique that Netflix has that. It was a big bet that we made more than 10 years ago to build our own content delivery network and the scale of that often surprises people. So 6,000 locations around the world, more than 175 countries and that actually allows us to place local files for film, TV, games that you're going to play so that there can be very low latency and very high quality for members no matter where they click play.

Basically these are server locations at 6,000 different location inside cities. whatnot where you have it's like it's like your edge network, right?

That's right. And we integrate with internet service providers to actually get the content to when someone clicks play on their phone, on their TV, on their laptop, that actually gets the content through that last mile to the member or the consumer.

And then when you're inside Netflix, you get to be exposed to some level of this detail, which I guess most engineers at other places wouldn't be. You would use a CDM provider and it will be a black box. But you're building this thing, right?

we're building this thing. And it's been an incredible head start as we think about new content types. So, when we started to go into live into games, especially cloud or streaming games, as we think about just the breadth of our film and TV offering, Open Connect has been a huge strategic advantage and we're extending that to be able to deliver different types of content.

The other thing that's unique is open connect as a content delivery or or edge network is sort of the end of a very long integrated life cycle that content moves. So I mentioned that media production suite that's on a studio production files are being transferred for review quality, making sure that we're aligning with the creative vision. Once a title is ready to launch that flows through other pipelines that would think about do we have the promotional assets? Are we ready to give great recommendations to the right audiences? How do we encode those files so that they're actually ready to be transmitted through open connect as the content delivery network?

When you think about that, sometimes we lightly call it pitchtoplay, there's an element of engineering all the way along that life cycle, which is unusual because at many other companies, they haven't built that endto-end pipeline themselves as Netflix has over time.

And what is pitch to play?

So think about from the moment a title is pitched that someone in the content team greenlights yes we're going to develop and produce this title. There's data science teams, there's engineering teams that helps to support those decisions on programming. Then there's tech teams that help support the creation of that content, the promotion of that content, the recommendations and ultimately the delivery of that. So tech basically underlies that whole life cycle.

Wow. So it's this sounds like a lot more workflows. Usually when I hear of of the work of pipeline, you know, in engineering, we would think of a CI/CD pipeline and I think we're very familiar with that. You know, you have your code reviews, test run, and it kind of goes on. But what I understand is just a lot bigger.

Imagine that CI/CD pipeline times thousands because it's actually going to follow an entire bring content to life for members cycle.

And now my first association seeing this this longer pipeline is it sounds like it would be rigid but of course Netflix is moving really fast. What does it look like from a software engineers perspective from from an engineering team's perspective getting a project done? How are these these typically done? Is is it based on some sort of following you know the schedule or is it a lot more elastic?

Oh this is probably where the the uniqueness of Netflix's culture comes into play. So a lot of the way that our engineering systems, products and tools were built was highly driven by individual contributors thinking about how to build those systems. So innovation is really driven from within the teams rather than top down. There's a lot of autonomy and local judgment in how we build things that has allowed us to build this endto-end view of how to deliver content in a way that we think delivers the best quality most efficiently and allows us to play with the puzzle pieces of that as we have new needs that come up.

So like I mentioned, we had many of those things in place for video on demand film and TV. When we went into live, engineers needed to rethink how are we going to have to change how we think about how we're delivering content given the requirements of live. They were able to start with what we've already built, but also have a lot of their own decision-making on the how to evolve all of our systems and products to be able to deliver new content types.

So over time the way Netflix has been built has been very driven by engineers within the teams rather than some you know top down overarching let me draw the architecture for you and now let's build it in that direction which has both advantages but also things that we've had to evolve towards over time because as the company becomes much bigger scale becomes more of a challenge we want to make sure that we're building things in a way that support that so it's not like it's static and that elasticity to use your word is something that has allowed us to actually engineer for what Netflix requires today versus what it required 10 years ago.

Can we talk about specifically live because Live was a big launch last year. I remember one of the big the the boxing match between Jake Paul Mike Tyson was a huge event. Can you give us a little bit of insight of of how that project started? How engineering teams got involved? Like like was it a small team? Was it multiple teams? I'm assuming there must have been multiple teams working together. And like what was your process of of getting this to to launch again? Like was it like overly planned out? Was it was it just yolo? Something in between?

Yeah, yolo. [laughter] Uh not quite yolo. Our first live title was a Chris Rock special. I believe in March 2023. That was our first time bringing live to Netflix members and started what was a very intense period of if I take through to that Paul Tyson match that was November 2024. So you think about that as basically 18 months from our first ever outing on live to the largest streamed event ever which is what Paul Tyson ended up being.

The way that that came to life was with urgency, a lot of scrainess, and like I mentioned, engineers making it happen. So, typically we would set a goal saying, "Okay, so we've got Paul Tyson scheduled, originally it was scheduled for July 2024. It was rescheduled because of Tyson's health to November 2024, which gave us a few more months.

But picture teams from open connect in coding, our content production and promotion teams, our discovery teams thinking who are the right people to lean in here and help bring this to life. But they self-organize. They develop their own road maps. They think about who needs to be on point for what things. What are some of the systems that we need to make sure are actually resilient enough for live. It was an incredibly tight timeline end to end. Not to mention that we had a lot of learnings from Paul Tyson because it was such a large event.

Yeah. I I've often world's largest, right?

65 million concurrent streams. Watching that tick up. I think one of our biggest ever days of signups. So, we were watching the signups, you know, go through the roof that day. Then watching I think by the time we were in some of the first couple of undercard fights, we'd already exceeded our expectations for how big the fight would be.

The energy in the launch room here in Loscatos was like palpable that you could feel, you know, excitement, nervousness, like very real time problem solving by engineers because we'd never seen no one had ever seen scale like that. So you get like your real time figuring out how to deliver something in real time. I've never been prouder of the team for figuring out like what are the levers we need to pull to keep this as stable as possible.

With those learnings in November 2024, we had about five weeks to be ready for two American NFL football games on Christmas Day where the bar is very high to deliver well for members and for fans. And so the team immediately took the learnings from Paul Tyson to say, how do we build greater resilience? How do we think about how we're going to direct content if we end up bandwidth constrained in some markets? How can we really optimize by using some of our quality levers for what that experience is going to be? And those NFL games ended up being flawless.

That from Chris Rock to a Love is blind failure to Paul Tyson with lots of learnings at that scale to NFL and now weekly WWE, not to mention lots of other big events. That was all driven by teams on the ground being relentless in saying, "How do we do this well? What are the problems we need to solve?" and the accountability for learn fast doesn't mean we're not going to have failures, but when we have failures, learn fast, iterate, and deliver ever better. That's really where the beauty of the Netflix culture comes into play.

You mentioned the the control room, but I think most people have not had this experience. It's a live event, of course, you can you can tweak things. Can you explain to to me what was the control room like? I assume it must have been a bunch of dashboards on all sorts of metrics, right? Was it a little bit like that?

Yeah. So, even our dashboards were brand new. The data science and engineering team collectively put together a set of dashboards that would give us visibility into some core quality of experience metrics. So things like time to render, app start timing, rebuffer rates. It was the rebuffers that we started to see amp up during Paul Tyson.

There were probably a hundred people on site. I was sitting in a room with maybe 30 or 40 both engineers and data scientists. We had our laptops and makeshift screens sitting there. Everyone was hardlined into internet so that we weren't, you know, risking anything with the Wi-Fi. We had VPN backups if anything was to go wrong. We had a launch commander with a headset dialed in talking to people in the production truck, but it was new.

So, it it wasn't streamlined. It wasn't perfect. It wasn't like the fancy launch rooms I imagine a lot of live productions typically have. And so we're watching metrics, you know, some things start flashing in red, you know, causing you to draw attention to it. We would create makeshift Google Meet rooms, so small groups of people could triage. We had people who were on the hook for being the informed captains or decision makers.

So if we have an issue with open connect, if we have an issue with playback, if we have an issue with discovery, are people actually able to find a title? There were people that we had in basically the launch plan. Imagine I think the document ended up being 40 or 50 pages of if then statements. If this thing happens, then what? You know, it was new for us.

When I think about where we were for Paul Tyson, I would I I joke with people. I feel like I lost 10 years of my life in that one night because it was so stressful and there's so little like there's no hands on keyboard thing I can do to help. I'm there to support the team to trust the team in making decisions.

When I look now at how we're doing NFL games or the Canelo Crawford fight or a WWE, it's much more sophisticated in the sense of the resiliency we've built, the metrics and dashboards, the visibility we have into what's happening. That was very human-driven when we were first coming out of the gate, which was a lot of where the learnings came from. But I say to the team, how often do you get to work at a mature company, but build something truly from scratch like this and think about all the things that might go wrong and how to be prepared for them and then stay very calm and cool under pressure if something happens.

you mentioned that the team had done a bunch of preparation. you mentioned 40 or 50 pages if then else like which which sounds way more detailed than I've seen most launches be you usually have a launch plan but again it was a complex launch so you prepared even so there were there were hiccups both with love is blind both with the Jake Paul match can you tell me how the team handled the the aftermath uh again it's pretty common to have blameless postmortems but I'm I'm more curious on how formal this process is how less formal driven by by people stepping up or again. Do you have like more rigid process around this or or it's just getting together and and people, you know, do go and fix things?

Like you said, it's not uncommon to have a blameless post-mortem or retro. So, we we have that for sure. It's much more interesting to talk about the learnings from something than to overly focus on who did what thing wrong. There's not much that we actually gain from that. It's not a rigid process. It happens very organically and it's led by the people who are close to the work itself and feel tons of accountability for doing reflections on both what went well and what can we do better.

If I take all Tyson as a good example, it was a pretty complicated set of emotions and couple days afterwards. You know, we're celebrating the biggest ever live stream event. Way bigger than we ever could have hoped for. We're celebrating that we work at a company that takes such a big swing. We're celebrating that we didn't collapse when we got to 20 million, 30 million, 40 million concurrent streams. If someone said to me, you're going to have 65 million concurrent streams, I would have said, "This is not going to go well." And yet, yes, we have hiccups. We always want to deliver a great member experience.

I would like to say I woke up the next morning. I was awake the whole night thinking about what do we do next and what are like what you know we only have five weeks till NFL. I woke up to a set of memos where the team had already written down. Here's what we observed. Here's what we think we can improve. Here's some of the things we should immediately prioritize.

So some of that was how do we direct traffic when we get congested? What were our algorithms doing to think about directing traffic in that moment versus what do we want them to do if we end up congested and thinking about what are ways that we can gracefully fall back or degrade when we're under that type of duress. There was nothing in that event that we could have created before seeing what happens to the systems live.

And so just knowing, you know, I woke up thinking all my like, you know, we're going to have to do this, we're going to have to do that. It was already there in a document. the team feels such accountability to how do we do this better that we don't really require a process to say now we should do a postmortem now we should develop reflections on this the team owns that very directly we have language in our culture memo about being unusually responsible that's really the talent on the team it comes with high talent density it comes with treating people like adults where they get a lot of autonomy in making decisions and then they own the outcomes.

When it comes to engineering culture at a lot of companies, you know, you go into a company as a new engineer and you ask around saying, "Hey, what are the processes I need to follow?" Because at a lot of companies, there's a mandatory code review. If you launch a feature, you might need to use a feature flag. Let's say on the mobile app, you you need to on the code review might need to have certain signoffs from people. So, there's a bunch CI/CD always needs to pass. You cannot override it. In the Netflix engineering team, how much of these things are kind of put down at a global level? lever needs to follow it. Teams can decide and do decide versus just based on the judgment of the the engineering team or or the engineer themselves.

Yeah, a lot is left to the engineering team and engineer themselves. So even for new or early career talent, one of the things we've evolved towards, so you know, even if I think going back however many years when Netflix introduced Chaos Monkey as a as a concept, the idea of an individual engineer having responsibility to understand how and when their system will break and how they're going to be resilient, detect that and recover quickly was just a core part of the culture and something that we continued to maintain.

Pre-live, there were a lot of ways to take smart risks with video on demand because we had many years under our belt of understanding when something breaks, how are we going to fix that? And it was left to teams to think about the extent of testing and resilience that they needed to have. and great on call teams and support teams for when something does go wrong. When we introduced live, there's a different threshold because you have to watch it live. There's no such thing as, you know, Netflix is going to be down temporarily while we address something where we were taking a smart risk.

as we started to introduce what are the guardrails we need to have to do this safely, it was things like introducing especially for tier zero or tier one applications that were in the critical path for live a higher threshold for what's the testing that you're doing. to make sure that your system is ready for the duress it may be under in a live event. We will share those guidelines. It comes from our central engineering team and it gives people an opportunity to have less process because they're able to say if I pass these guidelines if I've done this testing I don't need to be in a quiet period for example during a live event.

or we've done endto-end testing so we know those system dependencies very deeply and we're able to prepare for the whatifs something goes wrong but that's not a very structured process like a code review or you must check these boxes or some type of gating function for code being actually deployed. We do have quiet periods during the end of year holidays. I think that's pretty common. We'll have some rules of the road around live events to make sure we don't take unnecessary risk, but we are constantly finding ways to like how do we reduce the quiet periods? How do we leave a lot of judgment to the teams and then they're accountable for anything that goes wrong with their service?

Now, when I was a manager, one thing that did distract me, every six months on the dock, guess what? Performance review season. And as we just called it Perf, it was one month of my life. Uh, or especially at the end of the year, even longer. How do you go about this necessary evil of of or just necessary process of performance management because I understand it's very different and related to this uh it's very public publicly uh you know like known the keeper test which you which Netflix shares on on the website as well. How does this play into it if it does at all?

Yeah, we don't have formal performance reviews, which is probably the first unusual thing. So, when you think about other companies spending that time to talk through each person or assign a rating for whether they meet exceed, you know, I've seen that at other companies, too. Uh we don't do it that way, but we do carefully think about feedback, performance, expectations, all the things that would feed into keeper test, which I'm happy to talk through.

So, the way we approach it at Netflix is first trying to get to something that looks like continuous, timely, candid feedback. Easier said than done. It requires trust. It requires deep relationships to be able to give someone in the moment very candid feedback. It could be here's the thing you did great. It's not always a negative or a constructive thing. and to be able to receive that type of feedback. If we're living the Netflix culture well, that's something that would be familiar and comfortable every day of the year.

So, you're not having to wait for a certain performance review or feedback cycle in order to hear how you're doing or be able to provide that type of input to others. We do as kind of a safety net have an annual 360 process where I would request feedback from a a bunch of people I work with. I get requests from a bunch of people, but that's something that you're having a direct conversation with the individual about feedback. It's something I would review with my manager to say, "Here are some of the themes that I heard, some of the things I'm going to work on." So, there's an opportunity to think about what is my performance? How are people perceiving our working relationship and my contributions? But it's not structured as an evaluation. It's structured in the context of feedback that helps people improve.

And then separately once a year we go through both compensation review which is a reflection in ways of what's my level of impact what skills have I gained what are my contributions to the company. So you naturally talk about performance as part of thinking about someone's personal top of market which is our compensation philosophy. So it comes up as a conversation there where managers really think about for each person on their team how do I think about the compensation that reflects this person's value to Netflix and value in the market. So that has a performance flavor to it but is not a performance review.

And then a couple years times a year we evaluate promotions. So in that case for a group of people who might be up for promotion from you know level five to a level six we would collect feedback that helps us make that decision. So if I look across the feedback continuous in the 360 cycle compensation review and promotion evaluations, we get quite a few touch points where people are hearing how they're doing, but it feels more constructive and actionable than the performance review structure that other companies have.

So one of the very exciting things these days of course is AI and AI tools both building with them but also using as engineers. In your experience inside Netflix, how are the engineering teams using these AI tools for their own work? How are they experimenting with them? What is working? What is maybe not a great fit?

Yep. It's a huge area of focus for us right now, but with a lot of intention and pragmatism of where these tools are actually helpful versus they're not. The hope is that we identify those places where we actually get higher quality, more impact for the business versus things that are lower quality or just about cost reduction. That's really not interesting to us.

So for engineers we are experimenting with a set of coding assistants. The way we approach it is to provide a lot of different tools to the teams so they are able to explore experiment decide which tools meet their needs start to learn what works better for some use cases or some applications than others. We're trying to create space so that people actually have the time to do that.

We're doing things like both enabling the tools. We have some weeks where we let people just be focused on let me try a new project. Let me experiment with something new that gives people a little bit of space. And then we're collecting tons of feedback from across the team around which tools are actually useful, which do we want to graduate to paved paths, where are the areas where we actually see the most impact.

You said you're getting a lot of feedback already. You know, people, teams, organizations are sharing what's working, what's not. Are you seeing some areas where where these tools specifically AI coding assistance agentic tools are maybe a little bit more more helpful? May may that be green field things, migrations, prototyping or some other areas.

You named a couple of them very well. So maybe starting at the end there prototyping is a lot faster and actually that's a place where when you think about the cross functional teams across engineers, data scientists, product managers, designers, we're hoping that we can actually bootstrap things very quickly. I have an idea. Let's visualize that idea. Let's like quickly throw together a set of code that would help to bring this to life. That's not necessarily something we would productize or consider production ready code, but that's okay because it helps teams advance and innovate and kind of workshop ideas very quickly.

And there's also things around detecting issues. So anomaly detection, response, being able to do deep dives of issues. We're finding that there's a lot of promise for Genaii tools in that space which helps us with some of our resilience and just general best practices and health as an engineering organization.

If we're able to use Genai tools in those spaces, prototyping, documentation, migrations, detection and response, it leaves a lot of time for the more innovative work. So, how do we think about architectures and systems and products that we're building to deliver business impact? So then hopefully we can actually get more impact for engineers because they're able to actually leverage some of the tools or agentic experiences to minimize the time spent on some of the less impactful activities.

One of the most surprising things I've learned about Netflix just very recently is how much you invest in open source. This might sound a bit silly because we know Chaos Monkey is is very famous. In fact, everyone that's Netflix is known known for that one. But again, in a recent report, it looked at all the the companies and what percentage of engineers end up working on open source. And Netflix again was at the very the highest bar. This publication estimated that about one in five engineers work on open source projects. Sure enough, I go to your open source page. It's just so so much open source. Can you tell me why and how and since when is Netflix doing so much open source and why do we not know about this? This was new to me.

Oh yeah, perhaps we should be talking about it more. So this is a good opportunity to start. You know, we were talking earlier about the engineering culture and the sense of it, you know, talent density, which often comes for a passion to contribute to the broader technical community.

So, the people at Netflix care deeply about the quality of their work and advancing innovation more generally. For some things it it's Netflix specific innovation and it's important we keep that IP as a competitive advantage but for many it's something that helps to actually drive broader industry innovation which also benefits Netflix over time.

So if I can give you one example among the list of places where we've been very involved both internally and externally it's in the encoding space and driving a ton of innovation right video encoding. We've now won, I believe, nine Emmys for these contributions. I always used to associate Emmys, you know, just with, you know, the TV and the, you know, the red carpet, but we've won a lot of technical and engineering Emmys at this point, specifically on video encoding work.

So, as one example, that helps to contribute incredibly to quality and efficiency of our ability to encode our titles and deliver them. So Netflix gets an immediate benefit by improving the technology in that space. But we are also a founding member of the open media alliance which is an industry community that pushes for open advancement of encoding technology. If we're able to inspire that work, Netflix actually also benefits because the whole industry uplevels and we think about integrations with different technologies that we might do over time with everyone helping to push the bar.

A statistic I like to cite is when you look at the catalog now of Netflix content, think about how much bigger the catalog is than when we were first starting with originals. I believe we now require 60% less bandwidth, 60% fewer bit rates for same or better quality with a much bigger catalog. That comes from our media encoding innovation. and having a whole industry that's pushing that benefits anyone who's in the entertainment space and definitely benefits consumers and our members.

So as as closing, Netflix sounds like a very different and and special place compared even even across all of the the larger tech companies or even the innovative companies. What would your advice be for a new start software engineer starting at Netflix? How can they succeed in this environment and how can they grow up to to the expectations at a place like this?

Curiosity. Curiosity. Curiosity. to when people ask me like what's the Netflix value that most resonates with me and I most love to see across the team it's curiosity asking questions questioning whether we're solving the right problems in the right way just because you're new to Netflix or earlier in the career doesn't mean you're not going to be the source of innovation if anything it's great ideas come from everywhere and that starts with just being curious open-minded experiment explore take smart risks.

Try to reduce that that kind of voice in your head that is fearful of exploring something new or taking that risk. And I think when people join Netflix and they approach it with that type of curious mindset, they're already set up for success. I would also say lean on other people. So, we have great talent at Netflix and they are all more than happy to help other people be successful. So don't shy away from finding a mentor, asking somebody, why does this work this way? Can you give me more of the history of this? Can you help me understand which business problem we're solving and why? It's another flavor of curiosity, but it's also about the broader community and really leveraging that at Netflix.

Well, Elizabeth, thank you. This was very, very interesting and I've learned a lot.

Great. Really happy to be here and I was happy it worked out to have this conversation.