从单体巨型提示词到文件系统架构：Vercel 如何重构 AI Agent 开发范式

Hey everyone, thanks for coming. I'm Andrew. I'm the chief of software at Vercel and I'm here to talk to you about how we solved agent building at Vercel. I'm the chief of software. So I work on a mix of internal engineering, external experimentation, and generally being at the frontier and building new libraries, frameworks, and technologies.

For those of you that don't know Vercel, Vercel builds agentic infrastructure so people can build what's next. We got started in the web world helping people ship websites and web apps without having to worry about the infrastructure that doesn't make their app any better. It can scale to a million and scale down to zero effortlessly. But we're seeing a change in what people want to build. You know, people started by building pages, but now we see them want to build agents. And we've been embarking on a similar journey to make it easy for people to build agents and agentic applications easier.

We built this thing called the AI SDK. So instead of needing to switch out 300-400 lines of provider-specific code, you're just going to switch out one line of code and we have the same model interface underlying for all these different providers. We built a lot of other tools to make it easier to have model fallbacks, secure code execution, better pricing when it's inactive and waiting for responses, as well as for durability and resumability.

And I'm here to talk to you about how I went on this crazy experiment roughly a year ago that led to an agentic explosion at Vercel and led to a really cool thing that we built recently.

This is 1980. Bill Gates, before my time, had this quote saying he imagined there would be a computer on every desk and in every home. You know, that was probably pretty contrarian then and today it seems like very normal to have that happen. And me and the CTO had this thought: instead of a computer on every desk, could we potentially have an agent on every desk? Today we only really use agents for coding and technical workloads, but we're starting to see expansion into things like design, product management, and other verticals.

And this was maybe about a year ago, so I would say I'm pretty early to this, but that was when it was like Sonnet 3.5 and things weren't as sophisticated as they were today. And I tried to actually explore this out, see what we could do about it. I went around to various job functions at Vercel—marketing, sales, finance, legal—and I asked them, "What do you hate most about your job?"

And the most compelling use case I heard was that the data team, they were growing. They were a very lean team, but Vercel was growing faster. You know, they had so much more data from customers, analytics, metrics, sales. They just had to keep on aggregating and keep on making available for themselves to use. And at this time, if you think about what the data science people ever have to do: whenever someone from marketing or sales has a question about a customer or product, the data science team has to drop everything they're doing, write the query, process it, do an analysis, and come back with some recommendation on what to do. And this was really killer to productivity. You know, the data team did not want to drop everything and just write queries all day.

And so I worked with our VP of Data to try to build a better way for them to operate this way. And so if you think about the very first thing you would ever do if you want to try to use AI to solve a problem: you may just build like a huge mega-prompt. You have a question, you pass it into an LLM, you have it respond, and that's it. This was how the first version really looked. Honestly, I asked them for a dump of the Snowflake schema. I pasted it into a system prompt with a question, and then when it generated SQL, I actually copy and pasted that in and just ran it myself. I just wanted to see: are the models good enough today in order to write valid SQL given some decent structure?

And I would say this gave us a little bit of confidence that, you know, models today aren't that good, but maybe we can harness engineer or make the context around it a little better and give us some more guardrails to operate a little better.

And so if you actually think about what a data scientist actually needs to do when they get a question: they have to process the question, they may have to explore the semantic layer and actually figure out what the join patterns are, they will actually go and execute the SQL, they may go back and do that again if the SQL did not execute or was too expensive, and they'll eventually report on it, including visualize the data, maybe write some paragraphs, maybe do a retro, maybe do some other stuff.

And so if you think about those different phases, me and the VP of Data tried to sit down and map those out into specific agent workloads. And so the second version of this data science agent, called D0—I'm going to reference D0 from now on—is you ask a question, we have a query agent that passes on a query to the planning agent that will then have an execution agent, etc.

And if you chain all of these together, you actually get something that looks like this where each agent has a very dedicated system prompt focused to what that does with tools scoped to exactly that function. So example here, you can see that for the first one, the planning agent has a read entity YAML and a search schemas tool. And so it will only use those capabilities until it has an answer to pass on to the planning agent, and then to the SQL agent, and then to reporting.

And this was getting better. You know, we were able to get away from having to copy and paste SQL and have to come back and report on it. It was now actually doing like the end-to-end loop from question to answer. But we started hitting some walls with this architecture.

And around this time we came to the conclusion that what you actually need is one agent with all the mega context within it and for it to sort of manage its own memory. You know, this was around the time when we realized that you want to actually have the agent be able to look back on what it's done, sort of reflect and figure out the steps that got to get here. And with the previous model, you may have noticed that the only thing that the next agent gets is a summary and a small snippet of the previous thing that was done.

Now this way, you can imagine that you have one mega agent and internally it manages its own state. At some points it's planning, some points it's building, some points it's executing, and some points it's reporting. And this is sort of what it looked like: you have one big AI call, maybe max steps 100, and you give it the ability to manage its own state based on where it's at inside of its execution journey. And so you can see similar tools, you can see a similar shape, but the best part about this is if it ever ran into an error when executing or joining, it could go back and explore more, or it could go and read more and figure out what it was doing wrong.

And it was very good at this point. We were pretty confident in the actual system at hand and we actually spread it to a few trusted members. You know, this is a very powerful tool and we didn't really want to put it in the hands of the wrong people or people that were using very critical workloads. So we got it into a few people's hands and the immediate response was it was awful. You know, we thought we were cooking. We thought this was nailing 30% of our evals, but we couldn't have anticipated some of the questions that were being asked. And for us to spend more time manually mapping out some of these scenarios, it didn't seem like a very scalable way to do this.

And then Claude Code and Opus 4.5 came out. Well, more like Opus 4.5 came out in tangent with Claude Code, which is so powerful. They sort of unlocked the concept of a file system agent, and we on the side were like, "Wow, Claude Code and Opus 4.5 is basically AGI compared to what we had before." It would answer most of our questions without even missing a beat compared to the hand-grown agent we had.

And when we tried to step back and wonder what we were doing wrong and why this was so much better, we realized that the big unlock was that it was just a file system. It had a very minimal set of tools: list file, read file, run bash, and we gave a few more here for our own data agent use case. But the biggest thing was it was able to use the tools that agents are well trained on and was able to explore and write work where it needs to. Claude Code was not giving it a very prescriptive set of tools. It was sort of just letting it go wild and explore emergent behavior.

And so from this, we learned that you can really just use a file system. We saw the learnings from Claude Code and how powerful it was given that it just executes locally. And we tried to rebuild it in a way that was very Claude-Code-esque. It was now going to run in a sandbox. That sandbox would dump the whole semantic layer into it. The agent would be able to grep, bash, read file, write file all around to figure out what it needs, and we would just sprinkle a few tools on top to make sure it could do everything that is Vercel-specific.

And this was actually the biggest unlock ever. The leap from single agent to Claude Code SDK, and then from Claude Code SDK to file system agent in general, fine-tuned or purpose-built for our use case, was an amazing leap. At this point, the eval score basically doubled.

It's very simple: you just give it a bash tool—we have a nice helper called bash-tool on npm—and you attach it to a sandbox, and you can attach files to the sandbox for it to read, write, and execute.

And after this revelation and after I saw that we were passing so many of the questions that we failed to do before, I wrote this banger blog post. The week that I wrote this, it was responsible for 70% of our vercel.com traffic.

And after that, the next logical step was that we want to figure out the common use cases we had. By then, we were getting thousands of queries a day from people wanting everything from customer metrics, sales metrics, number metrics, npm downloads. And it turns out that a lot of these queries are actually the same in shape: there's only so many ways you can do an aggregation, only so many ways you can look up a product, only so many ways you can do billing info.

And so we actually have a recurring job that takes the most recent queries and tries to distill them into a skill. And right now we have roughly 100 skills that do a mix of aggregation all the way through looking up specific data about certain people. And we found this very effective because every new agent run sort of just starts from nothing—there's really no pre-established context besides the semantic layer and the system prompt. But with a skill, it already starts off with a lot of contextual knowledge that has otherwise already been done.

This is roughly how it looks: the inclusion of a skills folder is actually very powerful. We also built this tool at Vercel called Skillsh—it's the most popular way to find agent skills and run them yourself.

And I'm saying all this because this journey is something that most of you may hit: where you start from something simple, you gradually add complexity, and you eventually hit a system in which you can ship to prod.

And I'm telling you this because at every step along building this agent, someone at Vercel was agent-curious and they tried to fork off of my DZero agent and build their own. And at every step, we sort of had a better way to do something that was not previously known. And we were wondering: what if people today could start from the very last insight and not have to ever start from just a simple prompt or from reinventing best principles from first principles?

And so we actually thought: what if we built the Next.js for agents? For those that don't know, Next.js is a popular web framework that Vercel built that invented this thing of file system framework-defined infrastructure. You don't have to worry about where things go; you just have to write files in the right conventions and it automatically declares where they should go. Your pages go to the CDN, your serverless functions go there, your caching goes in the middle.

And we thought building agents should be this simple. You should only have to create a skills folder, a tools folder, a channels folder, and you should be able to just declare these very easily, and the framework should know exactly how to make an agent out of it.

And that's why two weeks ago we released Eve. Eve is an agent framework like the Next.js for agents where it's very easy: from just starting with a sample template to having a fully ready agent, being able to add in your own custom knowledge, your own custom tools, and even integrate it into the channels that you are familiar with.

This is roughly what we think an agent actually looks like: an agent has a runtime and it has channels. And in that runtime, you're going to have durability, you're going to want to run things in an isolated environment, you're going to want to call into different models, and you're going to want to have connections.

And we built this with open source in mind. We built Eve so you can plug in your own open source adapters for Postgres, OpenAI's responses API, Docker, and other connectors. But we also made it incredibly easy to deploy in Vercel: Vercel Workflows for durability, Sandbox for secure execution, and Vercel Connect (something we just released to make it easy to generate short-lived OIDC tokens for connections).

And we actually rewrote the whole DZero agent in Eve as we were building Eve. The file system looks very simple: you have a bunch of system instructions, a couple skills, a couple tools, and it's very easy to compose this into a real agent and very easy to iterate on.

We gave this out to a few beta customers before we fully released it at our London event. One company that partners closely with us, Aura, rebuilt their agent (a mini-cloud testing agent that goes to websites, installs them, and tests services) and saw incredible success from the ground up using Eve compared to using an off-the-shelf cloud agent: fewer steps, higher success rates, and better insights.

When you deploy Eve to Vercel, you get observability out of the box: you see all agent runs, tool calls, each step it takes, estimated costs, and potential optimizations. You can get started today at eve.dev: clone it, start from a template, deploy easily, or self-host if needed.

And the reason why I bring this up is because I hope that there will be more and more business-specific use case agents. Before we built DZero, we actually battle-tested a lot of the industry well-funded startups that were doing these vertical agents dedicated to taking your Snowflake instance and running queries against it.

But we found out that what really makes this agent good is it has a lot of very specific company knowledge. The way that Vercel is a web-based company: we have a lot of customers that have websites and web properties. That goes a lot deeper into when you should query for what and what things link to what. And so a lot of these off-the-shelf agents, they're great to try, but if you really want to get the most juice out of a squeeze, you should really try to build your own agent and add in as much company-specific knowledge as you can.

Today, we've had roughly 20 decently PMF agents at Vercel that range from marketing retros to figure out who to reach out to, to the first-ever redline of a contract when legal sees a new negotiation, all the way to my data science agent helping with data queries. And that goes to show that we have been very agent-filled. All of this stuff is actually saving us a lot of time.

The data team has never been more productive. They have more time to go and improve the performance of Snowflake, to add new data sources that were missing, to fill in the gaps that they previously did not have time to because they were so busy writing queries.

And I think it's never been easier for you at your big, small, medium-sized company to sort of automate away some of the things that you do not want to do or some of the things that you're spending too much time doing. A lot of HR, finance, and sales can be somewhat automated with agents. And I think Eve is the best way to build said agents today.

These are my socials. Thank you all for coming and listening. I'm Andrew and I'll be around if you want to chat outside.