在 AI 时代既要快速迭代又要保持稳定：PlanetScale 数据库高可用架构与 Agent 演进实践

Hello everyone, how are you? I hope you are having a great last day of the AI ​​ Engineer conference. Today I'm going to talk about how to move fast without breaking anything. So, the phrase "move fast and break everything," you've probably heard it before. This is a saying from the formative years of the engineering culture of early Facebook, right? The idea was to move and release products quickly, without being limited by database crashes or broken features, because the high value of a quick release was a priority. That was 20 years ago, right? Even before AI agents appeared that wrote code for us, released and deployed solutions for us. So, more than ever before, this idea is becoming relevant.

But I think we can actually achieve both goals, right? I want to release products quickly, build great features, deliver them to customers, but at the same time not break the databases, infrastructure, and application servers that power my work. We've all probably seen something similar to this, and I'm not pointing to any specific company. There are many such companies, right? Which, due to the crazy scale that we have been observing for the last six months, a year or two, have encountered problems. Where the number of daily active users has doubled, tripled, 10 or 100 times. This is largely driven by the demand for AI agents. The infrastructure simply cannot keep up with this development.

Part of that is because it's a very difficult task, right? This is not an attempt to single out or blame anyone. It is difficult to scale systems to tens or hundreds of millions of users. But we want to have great products that scale well for many users, and for our status pages to look something like this, right? Three or four nines of availability. Maybe five-ninths of availability, where our databases are up and running, our applications are working, and our users are ultimately happy. This is the main idea of my talk, which is a little different from most of the speeches you have attended. Many of the talks are devoted to inference or specific ways of working with agents, prompting, and other things. This is more of a report on infrastructure: how to work quickly, taking into account the architecture and systems we run, which power the work of these applications. And we provide support for their databases. So this is something we as a company take very seriously. How do we scale and support companies that have millions of users on their platforms? Part of what I'm talking about is taken from an article we published last year about our database philosophy. I would advise you to find it on Google and read it.

But let's look at a few of these principles. So, the first and one of the most important is insulation, right? When you manage infrastructure, it is important to make sure that you have configured the system so that a failure in one thing does not "bring down" the entire system, that is, each component must be isolated. So what I'm showing here is the data plane. This effectively means: where are your databases located? Where does your most important data live, which is needed for users to log in and interact with your app? But there are many other components, right? There are your layers of governance, your oversight loops, perhaps your analytics frameworks, your applications, your MCPs, and all of these components can potentially fail. In fact, if you use a good database platform, the database is unlikely to fail, but you update these other things more often, so they are more likely to fail. So we want to make sure that we design everything so that if the management layer fails to update, it doesn't affect the rest of the system and users can still access the database, right? And to everything at the data level. So, this is one of those basic principles that we take very seriously.

And another point is redundancy. And if any of you have worked with databases before, you've probably seen a diagram that looks something like this. Although in the case of application servers, redundancy is usually a little easier. There are many autoscaling tools, such as Cloudflare Workers, Vercel functions, and AWS Lambda, which are very easy to scale and easily provide redundancy because they are stateless. But in the database world, state needs to be stored reliably, you can't lose a single byte of data, so how do you do that? A very common approach, which is more expensive and complex, but which can scale reliably, is to have a primary node and multiple replicas. So, in the case of a database, you write to the primary node, you can write all of your data to the primary node, and then it is replicated to each replica. That way, if a server crashes, and you see that red "X" next to the primary node, the system can automatically promote one of the replicas to primary and your users can continue reading and writing data without any problems. And this is a very standard way to scale databases. It's not the cheapest, it's not the easiest, but it's a very standard way. And I would say that even if you don't have thousands or tens of thousands of users. You might have 100 users, and in that case it's still worth having redundancy, if you're serious about your business, right? You can usually easily get through 100,000 requests per day, maybe even a million requests per day. But there comes a point when other technologies have to be implemented for effective and efficient scaling.

So let's take a few minutes to talk about how we scale these systems, and probably what you're most interested in— AI agents, which we'll get to at the end. So one of the key things, and this is something we do a lot at Planet Scale, is sharding. Oh, I just realized my mouse cursor is on the screen. Let me remove it from the screen. Let's see if it disappears later.

So, database sharding, what is it? Instead of relying on a single node that works well when you have 10GB of data, but not so well when you have 10TB, 100TB, or a petabyte of data. So what should you start doing? You need to distribute your data and queries across many servers, which is what all those shards below are for. Usually you set up a proxy, and we'll talk about what VTGate is, which intelligently distributes all user requests or database queries between all these servers. And that's exactly what we often do for some of our larger clients at Planet Scale. We perform sharding through Vitess for MySQL and Nicky for Postgres. This is a very fundamental way of sharding, but the thing is, you don't use sharding as a stand-alone solution. If we look here and see, for example, shard A in this block, you actually need a backup system. So, you should have one master server and multiple replicas. Then, if one of the servers fails, that individual shard will be able to recover on its own and get back up and running very quickly in the event of a problem. This is one way we see it, and probably the most common and efficient way to scale databases, but the principle of “distributing queries and data across many servers, nodes, and disks” is applicable to many other parts of your stack.

Back pressure. This is another aspect that is a combination of scalability and reliability. We mean that there are many systems, like MySQL, Postgres, and SQLite. These are very popular databases that people use, but they don't handle overload situations well. So if you send a lot of queries to them, their performance can degrade quickly, and they can basically crash. And that's usually where people's applications start having problems. So you want to design a system with the concept of backpressure. Back pressure is the design of a system with the ability to resist; when I find that too many requests are coming in, I want to be able to tell the client, "Hey, you are sending too many requests, slow down, stop doing that, reduce the number of requests." This is a very, very common and very important principle for scaling systems.

Now let's move on to the next one: compute partitioning. So we've talked about data partitioning, right? But what about compute? There's this idea that you have a single node running your app, but with compute partitioning, you break down different parts of your application and deploy them to different servers so that no one has to deal with the struggle for resources. So if you suddenly have a spike in job queues and you need to scale the queues, but nothing else needs to be scaled, it wouldn't be a bad idea to independently scale the compute resources for each part of our backend that powers your favorite application. So that's the idea of ​​compute partitioning.

And finally, something that's really, really, really important, and something that we think a lot about at Planet Scale, is schema changes. And that's something we'll talk about. So this is not a complete list. What have we already covered? We discussed sharding, backpressure, and now we're talking about partitioning.

There are many other aspects of scalability, and again, you can spend your entire career becoming an expert in this area. We work reliably, providing fault tolerance and high scalability, but today in my talk I wanted to focus on how we can combine this with AI agents, right? Because they are the ones writing code for us, but at the same time they are interacting with the lowest levels of our infrastructure. Databases, queues, things that power every other level of your stack. And that can be a little scary, because it's practically become the standard these days, right? Write code using AI. In some cases, people don't even write a single line of code themselves anymore, right? The AI ​​actually writes all your code, tests it, and you just direct these agents to perform tasks. But it’s a completely different thing to say, “Now I’m also going to give my agents the ability to deploy changes to the database, change its configuration, or perform sharding, right?” Because if you make a mistake here, it's usually impossible to just hit the undo button, right? This will disable your entire application, and every user will notice it. So we want to do it efficiently and safely. So I'll go back to a few concepts I talked about earlier and consider how to use agents to solve this problem.

So, let's get back to sharding. I will tell you more about Vitess— it is an open source project. We manage Vitess databases for our clients, and it works with MySQL to implement sharding. So, as you can see here, all these shards down there are MySQL databases, and VTGate is a proxy, an intelligent proxy, that takes MySQL requests, parses them, determines which shard they should go to, and routes all these requests in a very sophisticated way. The software is quite complex, but the interface we provide for defining how you want all your queries and data to be distributed across shards is actually quite simple. This is what we call a VSchema, and it's literally a JSON file where you specify, "Here's the column I want to shard by, and here's the way I want to do it." And you can specify which tables should be placed on which shards, having full control over where and how everything is configured. And the great thing is, as we all know, AI agents work great with things like text files and JSON files. So you can give your agent a task and say, "Hey, I'm having problems with the scalability of my database. Here's my schema. I want you to help me prototype a good schema, okay?" And once you've designed and engineered something, thought through and decided on the most efficient ways to shard on a platform like PlanetScale, you literally plug it in, deploy it, and we'll give you an interface to create new sharded systems and scale your database. A lot of this can happen either through AI or by allowing AI to take control of these processes through our API and the command line interface that we run. So, this is one of those things, right? There's actually a very complex infrastructure under the hood that scales your systems, but you provide the right interfaces for it, like simple configuration files, and these things become much more accessible to any company trying to scale.

Um, one more thing, this relates to the back pressure concept we talked about earlier, right? So when the database is overloaded, and as you can imagine, this probably happens all the time at some of these companies, right? I think there was a tweet from GitHub a few months ago that said their traffic had increased two, three, or four times from what was already an extremely high load, and almost all of that was driven by AI agents, right? So, they already had a gigantic infrastructure because they are the most popular place in the world to develop, and then they got a multi-fold increase in traffic, especially during the holidays, right? Because of these overloaded systems.

So, how do we deal with this? One of our solutions, and the way we like to think about it, is that we provide users with a system called "traffic control" that allows you to essentially classify and label all the traffic coming into your database, and then you create resource budgets to say, "Hey, this segment of my traffic can't exceed this amount of CPU or server process resource usage." And again, we perform a smooth performance reduction when this happens. So what I'm showing here is a budget schedule that I've set up to say, "Don't exceed these resource amounts and warn me if I start to exceed these resources." So it shows me the number of requests that exceed your budget over time. And we, we, we send a warning to the client, like, “Hey, I’m overloaded. You need to slow down." And we can make it even stricter, where we actually kill those queries that are running, right? And this is one of those things where, well, it's not ideal to reject requests, but it's much better than having your entire application crash and become inaccessible to all your users.

Uh, and finally, one more thing I want to talk about, and I know I'm showing the PlanetScale interface here, but it's not even about PlanetScale itself, it's about the philosophy of how we do these things with agents, which is— most developers are used to using a Git-like process during development. And that also applies to your agents, right? You have a master branch, you create branches, make changes to the code, merge them, and deploy it dynamically to where your application is deployed. What we did, and this was long before agents, was we wanted to take that same UX and replicate it for your database, because often when you build new features in an application, you also make changes to the schema. You add tables, columns, delete items, add indexes, you do all of this to your database to keep it in sync with the codebase.

So what we give you, using the same Vitess that we mentioned earlier, is the ability to create branches of a database schema, make changes to that schema in an isolated environment, and then merge that back into the production environment using what we call a deployment request, all without downtime. And not only that, but also the ability, you can see it down here, the ability, if you've deployed a schema change and you see a problem, we give you a one-click ability to undo it and go back to the previous state of your database without losing any data. Uh, and so that's something that's been great for people, but we also have an API and a CLI to allow agents to automate this. So when they do things, automate code, create branches, do merge requests, and check in code, there can be an actual mirrored state of the database that works in sync with all of these things, right? And it can be either fully automated or semi-automated. Obviously, many of us are operating in a “human in the loop” mode, where humans are still checking all these changes that AI agents are making, and that’s a good thing. But we already have ready-made primitives for building these systems.

There are still a lot of things I didn't have time to talk about, but the main summary I want to make is: we are a company, and I know many are trying to focus on the developer experience right now. And when I say “developer experience,” I don't just mean dark or light theme toggles, convenient keys, or anything like that, but a real-world experience: are we giving the developer responsible for the company’s database or infrastructure the tools to avoid mistakes, deploy systems safely, speed up the database, and improve its performance? We've been building these things for years, and it turns out that when you focus on the developer experience, you end up with great primitives that allow AI to work with the database safely and reliably and help it scale. Since we provide access to many of these tools through command-line interfaces, there are of course debates about which is better for working with AI, but by opening these capabilities to both sides, it provides a great experience for managing the database and interacting with AI agents. So, that's all I wanted to say. Thank you to everyone who came. I'm Ben, and I wish you a wonderful rest of the conference.