[Host]: AWS S3 is the world's largest cloud storage service, but just how big is it and how is it engineered to be as reliable as it is at such a massive scale? Milon is the VP of data and analytics at AWS and has been running S3 for 13 years. Today we discuss the sheer scale of S3 in the data stored and the number of servers it runs on. How seemingly overnight AWS went from an eventually consistent data store to a strongly consistent one and the massive injury and complexity behind this move. What is correlated failure, crash consistency, and failure allowances, and why engineers on S3 live and breathe these concepts, the importance of formal methods to ensure correctness at S3 scale, and many more. A lot of these topics are ones that AWS engineering rarely talks about in public. I hope you enjoy these rare details shared. If you're interested in how one of the largest systems in the world is built and keeps evolving, this episode is for you. This episode is presented by Statsig, the Unified platform for flags, analytics, experiments, and more. So, Milan, welcome to the podcast.

Milan: Thanks for having me.

[Host]: To kick things off, can you tell me the scale of S3 today?

Milan: Well, if you want to take a step back and just think about S3, it is a place where you put an incredible amount of data. And so, right now, S3 holds over 500 trillion objects. We have hundreds of exabytes of data. And we serve hundreds of millions of transactions per second worldwide. And if you want another fun stat, we process over a quadrillion requests every single year. And what's under the hood of all that is also pretty amazing scale. If you think about, you know, what's underneath the hood of S3 at the fundamentally were we're discs and servers which sit in racks and those sit in buildings. And if you try to think about all of the scale of what is under the hood, we manage tens of millions of hard drives across millions of servers. And that is in 120 availability zones across 38 regions, which is pretty amazing if you think about it.

[Host]: So deep down it it all starts with hard drives sitting inside servers, sitting inside racks, and then you have a bunch of these racks and then rows of them, buildings of them, right? And that's what you said. So there's tens of millions of hard drives deep down in in in the bottom of this.

Milan: That's right. In fact, if you think about the scale of this, if you imagine stacking all of our drives one on top of another, it would go all the way to the International Space Station and just about back. And so like that, I mean, it's kind of a fun visual to have for us who work on the service, but you know, kind of fundamentally, it's it's really hard to get your brain around the scale of S3. And so a lot of our customers they they don't they they assume the scale is there. They assume that you know all of the drives are always there and they just focus on what S3 is to them which is it just works. It just works for any type of data and all of your data.

[Host]: Yeah. Even I mean even for me for the scale when you talk about exabytes I actually had to look up exabytes because I know of pabytes which is already massive. If if a company has like one or two or three pabytes of data it's it's tons. And exabyte is it is is it a yes it's a thousand pabytes is an exabyte and and you told me that you're you're you're thinking in that level. It's just hard hard to hard to fathom.

Milan: Yeah, we I mean we have individual customers that have exabytes of data. Individual customers who have exabytes of data and what they call a data lake. Although last week I heard a great term. We had the um Sony group CEO talk about what Sony is doing with data and they refer to it as a data ocean and not a data lake but a data ocean and so like if you have exabytes of data in your data lake it is in fact a data ocean and that ocean is is kind of fundamentally S3.

[Host]: Can you tell me how S3 started? I I did some research and there was a story about a distinguished engineer sitting in a pub in Seattle. who knows it was true or not but I read that this this was a story that he was a bit frustrated with engineers at Amazon building a lot of infrastructure again and again.

Milan: Yeah. If you think back into you know S3 development really started in 2005 and we launched as the first AWS service in 2006 and if you think about the technical problems of 2006 you know a lot of customers were building things like like e-commerce websites right like Amazon.com and so the engineers at Amazon knew that they had a lot of data that at the time was very unstructured data it was PDFs it was images it was backups and they needed wanted a place where they could store that at an economic price point that let them not think about the growth of storage. And so they built S3 and they really built it for a certain type of storage. And so the original design of S3 in 2006 was really anchored around eventual consistency. And the idea of eventual consistency is that when you put data in storage for S3, you know, we're not going to give you an act back on your put unless we actually have your data. So, we have your data, but the eventual consistency part is that if you were to list your data, it might not show up because it's being eventually consistent. It's there, but it might not show up on a list. And so, we did that at the time that consistency model at the time, uh, we built that because, you know, we were really optimizing for things like durability and availability. And it worked like a champ for, you know, e-commerce sites and things like that because, you know, when a human was interacting with an e-commerce site and an image happened to not show up exactly at the moment where you put the data into storage, it was okay because a human would just refresh. And so when we launched in 2006, here's a a fun fact for you. 2006 is actually when Apache Hadoop first began as a community as well. And so we had a set of what I think of as frontier data customers like Netflix and uh Pinterest who took a look at things like Hadoop and they put it together with the economics and the attributes of S3 which is you know unlimited storage with pretty good performance at a great price point. And they um they decided to build their you know what we first began to call data lakes at the time. they decided to build to extend the idea of unstructured storage and include things like tabular data. And so the first wave of frontier data customers were adopting quote unquote data links um in about 2013 to 2015. Those were the frontier data customers born in the cloud. And around 2015 to I would say 2020, we started to see all the enterprises take that same data pattern of how can I use S3 the home of all the unstructured data you know on the planet and extend it to tabular data and that's when about five years ago 2020 I started to see a ton of exabytes of you know basically parquet files and you know I I have worked on S3 for a minute I started working on S3 in 20 I guess it was 2013. I'd been at AWS since 2010, so kind of a while. And the rise of parquet was really interesting because what people did is they said, "Oh, okay. I like the traits and the attributes of S3 and I want to apply it to a table." And so I am going to run my own parquet data in S3. And then you know around 20 I would say 19 2020 we started to see basically the rise of iceberg and iceberg at the time you know is incredibly popular and it gives the table attributes to the underlying parquet data and customers started to do it in you know many of my largest data links across different industries and different customers and so one of the things that we did in 2024 is we introduced S3 tables.

[Host]: just for those who don't know what iceberg is. So, it's it's an open-source data format for like massive analytic workflows. Right.

Milan: That's right. If I ask our customers of these data oceans why they care so much about iceberg, it's because they want to be able to have what a lot of customers are calling this decentralized analytics architecture where, you know, they can have lines of businesses or different teams within their company that pick what type of analytics to use as long as it's Icebro compliant. And so if iceberg is the common metaphor for data for tabular data then you have choice you have flexibility and choice for what type of analytics engines you use in a decentralized analytics architecture and so I think that's one of the reasons why iceberg has just taken off is that it makes it easy to use data at scale but it also gives a business owner this you know the chief data officers or the CTOs of the world it gives them future proofing for analytics they can replace their analytics they can change it out. They can adopt new types of analytics and AI because you have this iceberg at the bottom turtle of S3. We lost S3 tables in 2000 um in December 2024. This year we've had over 15 um new features that we've added to S3 tables. Um and uh and then this year of course we launched the preview of S3 vectors in July and then last week we were generally available and so you know the story of S3 it's like a story that our customers have written for data but it's been super fun to work on all these different evolving attributes.

[Host]: as an engineer. What is the kind of basic architecture and the basic terminology I should know about when I'm starting to work with S3?

Milan: When we first launched in 2006, the whole goal for SRE is to provide a very simple developer experience and we've really tried to stick with that. In fact, when the engineers and you know when we're sitting around and we're talking about what do we build next, we always go back to that idea of how do you make things really simple to use to use S3. And so fundamentally S3 we have a lot of different capabilities now, but it's really about the put and the get. the put of the storage in in and the get of the storage out and where we can do that really well at scale that that is kind of the heart of S3. Now we have a ton of extra capabilities that we've launched over time but you know fundamentally when customers think about using S3 they think about the put and the get.

[Host]: Yeah. So like put data get data and I guess some of the other like operations it's a bit like HTTP right? There's also delete, list, copy, a few kind of other like I guess primitives.

Milan: there is and you know if I think about where we have gone over time we've added capabilities on top of that just based on what developers are trying to do. Okay, let's just take put. Okay, um we recently added a set of conditionals to the put capability and like last year we did put if absent or put if match. Um this year we did a copy if absent or a put if match and we did delete if match. And the the core thing about for for us with conditionals is that we can give developers the capabilities of doing things like the put but to do it based on the behaviors of their application.

[Host]: Outside of the the get and put the basic operations I guess the base terminology that you should just know about is the buckets, objects and keys, right? That's how we think about our data.

Milan: Yeah. And now it's not just objects. If you think about um the two latest um primitives or building blocks we've introduced as as native to S3, one of them is the iceberg table with our S3 tables and the other one is is vectors. And you know under the hood of an S3 table is a set of parquet files that we're managing on your behalf. But that's not the case for vectors. A vector is just basically a long string of numbers. And that is a new data structure for us and it's sitting um in S3 just like your objects.

[Host]: So I I I'd like to still go back to the beginning of of S3. When it was launched, it was pretty shocking for the broader community because S3 launched with a pricing of 15 cents per gigabyte per month, which was about a third to fifth cheaper than anything else. The going rate at the time was something like 50 cents or 75 cents. And on the first day, I I read that like 12,000 developers signed up immediately. A lot of companies immediately or very quickly moved over and then the surprising thing was that S3 kept cutting prices. It was unheard of before. You were there in the 2010s when some large price cuts happen. Can you tell me what was the thinking inside the S3 team on the this unusual pricing it seemed customers would have been willing to pay more and also the the cutting of prices continuously even today? I think today it's something like 2 cents or 2.3 cents, something like that for the same storage as as as it was 15 cents on launch.

Milan: Yeah. You know, I think part of this goes back to what the goal is for S3. Okay. And so the mission of S3 is to provide the best storage service on the planet. Okay. And our goal too is that if you think about the growth of data, IDC says that data is growing at a rate of 27% year-over-year. But I have to tell you, we have so many customers that are growing so much faster than that. ... And in order to have all that data and to grow it you have to be able to grow it economically. You have to be able to grow it at a price point where you don't really think okay what data am I going to delete now because I'm running out of space. You don't have that conversation with S3 customers because of of two things. One is, you know, we do lower the price of either storage or the capabilities of what we're doing. ... We give you the ability to do something called intelligent taring, which is if you don't touch your data for a month, we'll give you an automatic discount on that data because we're watching your storage and you don't touch it for much, we'll give you up to 40% discount on that storage. And it's like dynamic discounting so you don't even have to think about it. And so our whole goal is that you can grow the data that you need to grow because we know that's being used to pre-train models. We know it's being used to fine-tune and do any type of post-raining of AI. ... whatever you need the data should be there and you should be able to grow it and keep it and use it any way you want.

[Host]: Yeah, I I did want to ask you about this part. So there's intelligent taring which was launched in 2018. So like 12 years after S3 was launched. One thing that really got my attention Amazon Glacier which is was launched in 2012. So a long time ago and it's you can store data that you don't need immediate access to. You're okay waiting for some time to uh to get access to it. I think maybe even hours. when it launched it was only one cent per gigabyte per month which was again this was something back then the going rate for storage was like 15 cents so almost like almost 10 times cheaper. How do you do that? Like what what is the architecture and thinking behind how you're able to have this trade-off of like look if you don't need your data quickly we can do it a lot cheaper. How h how could I imagine the kind of trade-offs that that you and the engineering team were were were thinking of making?

Milan: Well, you know, I mean, a as you know, I mean, you're an engineer yourself and you know, as you know, a lot of engineering is about constraints, right? And that is the fun part about working on on S3 is that when you think about constraints, you think about constraints that we have for availability, you consider you think about constraints that we have around, you know, the cost of storage, we start to get really really creative. Okay? And in S3, because you know we build all the way down to the metal of S, you know, of the drives and the capabilities that we have in our hardware, we're able to drive, you know, efficiencies at every single part of our stack. Okay? And so our engineers when they get together and they and they talk about the constraints, they talk about the design goals, we'll do something like we'll set a a a target for you know the cost of a bite and we'll drive for that and we'll drive for it at every single part of the process. And the part of the process that we are also including is is you know it includes a data center. How do our data center tech uh technicians be able to operate the the service of S3 from a hardware and a data center perspective like the physical buildings just like we do the same thing for the software and the layers of S3 itself and when you have that when you have that ability to to run across the whole stack all the way down to the physical buildings and we're thinking about so deeply about the cost and the lifetime of every bite it you're able to do things like like Glacier.

[Host]: You mentioned something really interesting that when S3 started it was eventually consistent which means that you know data eventually arrives it it might not be there and you might be behind and there's a lot of things that you can do with this and and it gives you some constraints but you mentioned that the reason that the team launched this because durability and availability was more important and I I assume of course cost as well but during those initial phases while S3 was eventually consistent what what kind of benefits does it give to have eventual consistency? Is it a cost uh constraint? Is it just easier to do high availability systems from from an engineering perspective?

Milan: Well, I mean from an engineering perspective, the the main optimization was it was availability. ... So if you think about the indexing subsystem in S3, that holds all of your object metadata. And so that's like its name, its tags, its creation time, and the index, our index is accessed on every single get or put or list or head or delete, any API call like that. ... And so the data is basically in our in our in our index system is stored across a set of replicas and it uses something called you know it's it's basically a quorum based algorithm. Okay. And a quorum based algorithm tends to be very forgiving to to failures. ... And so when we have quorum at the index storage layer, we can see reads and writes overlap, but in in the cache, they don't because we're optimizing for availability. ... And so the engineers had to come up with a new data structure. ... we had to build this replicated journal. Okay. And the replicated journal is basically a distributed data structure where we're chaining nodes together so that when this write is coming into the system it's flowing through the nodes sequentially. ... And therefore on a subsequent read like through our our cache, the sequence number can be retrieved and stored. And so now you have this strongly consistent and highly available capability in S3. And the heart of that is actually this replicated journey.

[Host]: Okay. But what's what's the catch on one on one end because there's always always something with trade-offs. You always have something. So on one end you obviously have more complicated business logic. And then I guess the second obvious question is what about failures? Because in the case of eventual consistency, you don't worry too much about one failure. Clearly in this case uh what if a node in the sequence fails either at the time at the first time or or later or how does the system monitor this recover because that that's I guess that's going to be the tricky part right.

Milan: there's another piece to this puzzle that we implemented which is um you know it's it's basically a cash coherency protocol and the idea is that um this is where we built what we think of as a failure allowance where in in this mode, we uh needed to retain the property that like multiple servers can receive requests and some are allowed to fail. And so it's kind of this combination of this replicated journal as a as a new data structure plus we implemented this new cache coherency protocol that gave us a failure allowance and those two things working in concert gave us this uh strong consistency. I will say too this does come at some um actual cost. ... We said that when we launch this, we should launch strong consistency. We should make it free of charge to customers and it should just work for any request that comes into S3. We shouldn't sort of say it's only available on this bucket type or what have you. This should be true for every request made to S3.

[Host]: How do you know that you're strongly consistent?

Milan: And that is why we used automated reasoning. ... automated reasoning is a specialized form of computer science. Okay. And girly, if you if you kind of think about if computer science and math got married and had kids, right, it would be automated research.

[Host]: is it formal formal methods or based on formal methods?

Milan: That is right. And we use formal methods in many different places in S3. But one of the first places that we adopted was for us to feel good that we actually had delivered strong consistency across every request. So what we did is we proofed it, right? We basically built a proof for it and then we incorporated our proof on check-ins into this index area that I talked about... And so when somebody anybody is working on our index subsystem now and they're checking in code into the code paths that that are being used um for uh consistency we are proofing through formal methods that we haven't regressed our consistency model. ... at a certain scale math has to save you right because at a certain scale you can't do all the combinatorics of every single edge case but math math can save you and help you on this uh at S3 scale.

[Host]: Amazon S3 has very very like high durability promises. I think it's 11 9 which I I had to like do a double check on because in in uh backend systems whenever you say three nines it's like h when you say four 9ines of availability we're not talking dur availability four 9s is already hard to achieve and beyond that it just gets very expensive and I have never heard of 11 9 of durability... How can you prove that not just in a formal way but you're now storing as you said 500 trillion uh objects... do you actually like validate it on the actual data as well on outside of the proof?

Milan: ...we really think about it in the storage layer. And so if you think about it in the storage layer, you have this design, this promise of you know the design here and underneath that is a combination of things. It's software but it's also the physical layout of where our data is across everything that we have in S3. ... I think the most important thing for us is our auditors. ... Collection of systems, which are over 200 microservices now, that all sit behind one S3 regional endpoint. And a fair number of those subsystems, those microservices are all dedicated to the notion of durability. ... And you know, part of our design is that at any given moment in this conversation that you and I have had just just today, we're we're having servers fail because servers fail. And so what we are building and what we've built in S3 is an assumption that servers fail.

[Host]: I understand in your business or or when you work at S3 scale, this is just every day. ... and what is a correlated failure?

Milan: Okay. So that's super interesting. So if you think about what I talked about with, you know, eventual consistency, we talked about quorum. Okay? And quorum is okay for one node to fail, but if all of the nodes go south, for example, and they're in the same availability zone or on the same rack, then you're really going to be messing with your availability of the underlying storage, okay? ... So when you upload an object to S3 with a put, we replicate that object. Okay? We don't just store one copy of it. We store it many times. And that replication is important. It's important for durability. But what's interesting about it, it's also important for availability because if any of those correlated failure domains fail, like if a whole a fails, there's still a copy somewhere else... The whole idea of crash consistency is that a system any system that you build it should always return to a consistent state after a a fail stop failure.

[Host]: How do you think of this this beast this this really complex system hundreds of microservices data that is hard to fathom... how do you engineers kind of wrangle this because it does feel a bit intimidating I'm not going to lie.

Milan: well I think so much of this just comes back to the culture and the commitment on the team... we have this principle engineering tenant called respect what came before and that's an Amazon engineering tenant which is if it has worked for many many years you have to respect that but then there's this also this tenant these two tenants are a little bit in tension with each other which is kind of what makes it so fun Amazon engineering tenant is called um be technically fearless.

[Host]: Now going back to the evolution of of S3 from unstructured to structured data. ... is it done Is is it finished or or is it still evolving? Because there is this notion that S3 can store anything already, right? Like any any object, any blob? What what new thing is there?

Milan: ...I actually think that the world's data for tabular data is going to live in the future in S3. ... SQL as we know is a lingua frana of data and the world's LLMs have all been trained on decades of SQL and therefore... You just need to know SQL. ... With vector we built a new data structure a new data type... we were getting about um 100 milliseconds or less for a warm query to our vector space, which is actually pretty fast. ... We have S3 service tenants as well. And one of the tenants and one phrase that I use all the time and our engineers do too is scale is to your advantage. ... The bigger S3 gets, the more decorrelated the workloads are that run in S3. That is a great example of scale is to your advantage.

[Host]: It it kind of sounds like you have all these microservices. It's kind of evolving almost like a plant or a living organism.

Milan: No. Yes, I uh I am in fact a former peacecore volunteer from forestry... yeah I mean S3 is this living breathing repository of data that lets people do things with data that they never thought possible. ... There's a a strong value and relentless curiosity. Okay. And you know, I talked a little bit about coloring within the lines and how when you work on S3 or a large scale distributive system which continues to reinvent what storage means, you're not really coloring within the lines. you're just kind of looking, you're taking a step back and you're saying, you know, I I will draw what the lines are today and I will know that I might have to rub those out and draw new lines in the future for wherever things go.

[Host]: Well, Miline, thank you very much. This was fascinating and and very interesting to get a peak into this massive world of scale of data and and respecting the bite and and treating it and making sure that it's durable.

Milan: It was great talking to you and thank you to both yourself. I know you're a fan of S3 and to all of your listeners who use S3. Uh we quite literally wouldn't be able to do what we do without the feedback and the encouragement from everybody who uses S3 today. So thank you for that.