Harvey的Agent平台实践：坏掉的不是Agent本身，而是它周围的工程基建

And honestly, the agents were the easy part.

After 10,000 PRs, the biggest lesson I can share is this. The thing that broke was an agent. It was everything around them.

10,000 PRs that an agent wrote. And it isn't just Tavac team playing with a toy. Last month, about 460 different people at Harvey use it. And over the summer, it stopped being only about code. People started asking a data question in Slack and getting answers back. My favorite recently someone on our marketing team dropped a prototype and a talk track into Slack and ask for a story book for a product video.

Once the harness exists, people bring the work that you never planned for.

This is talk about hardness, odd things around the model, checking out the ripple, holding the key, opening the PR, getting the merged.

When we first wrote about Spectre back in April, we put it like this. At some point, a harness become the product.

Now, whenever I tell people this, I get the same question. Why on earth would you build this when clock or cursor already exist?

Pass that you need a cloud. And cloud needs our secrets, our real stack, our alerts and tickets, one set of rules and enough capacity that nobody gets starved. Those single two owns that for us. So we build layer and that layer exactly what broke. So I want to tell three stories today. How we stopped competing on writing code. how we let any agent drive and were broke once they did.

More of our engineers were using cursor and they ask how do I get our secret there? How do I test against our real stack? We could have rebuilt all that inside cursor and then again for next to and the one after that.

Instead, the answer became one sentence. Just ask cursor to use vector.

And that's really the whole idea. People will always have a favorite agent and the favorite keeps changing. So pick yours. Cloud code, codeex, cursor, trag your own hands.

Start on your laptop, push the session to the cloud, put it back. That's how Spectre sits underneath all of that.

If you only build one of those stages that I described, build a follow-up loop. We switched on for everyone in early October and that months. Four out of five agent PRs merged into our backend PR had been through it.

That's the difference between agent that write code and an agent that finished the job.

So we gave it a hope projects people write down what matters. Adrian write down what they learn. Team pings the skills you trust and there's a record of who rent what.

Any agent working in the project with the same page. My personal view whichever agent you pick the audit trail and the checks should be the same.

So, we made a rule. If a person can click it, an agent can call it.

Start a task. Turn on the follow-up loop on automerge, publish a page, all from CLI,

which means your laptop agent can call it. A agent inside the sandbox can also call it. Even CI can call it too. And then people ran with it. By October, the command line was starting about 6,500 ts a month. The web app didn't shrink. Everyone else just grew around it.

Task started from the terminal were merging mops and task started in browser and the new kind of users showed up. People who start hundreds of tasks a month and never open the app at all.

It split the tickets into 10 different milestones. K one for me and the rest to Spectre with hundreds of tasks. Each has its own sandbox to validate end to end.

Then at the end of the October, our it give everyone trag now watch Spectre for me. It list the question Spectre is waiting on draft my answers and once I say yes, send them back.

I actually DM to my friend like that. I just use cursor clock code or whatever to spam spectre. Now,

well, that's the dream, but it's also the problem.

The fix keep a copy of every ripple on every machine and let each task borrow it.

Someone has set up workflow that tag our agent on every alert and every tag started task. On normal day those task started about 50 task. That day in about four hours they started around 900.

900 agents are chasing the same bug until that wasn't room left for a single note.

The fix was almost boring. A repeat alert now joins the investigation that's already open.

About 950 tasks from the command line. In about 3 hours, nothing in our platform can work. The cluster filled up and everyone else work sat in line behind one laptop. That's not a user problem. That's our problem.

4 days later, we had a real fix. A fair launch queue. Now starting a task gives you a place in line and the dispatcher started when there's room. People sitting at keyboard goes first. Everyone gets a fair share and can borrow more than the cluster is quite.

And whenever you start a task, you can see a placing line. When we hit it with a 300 task burst in low test, the task lit went from almost 2 seconds to 70 milliseconds.

When it's the busiest day of every week back in August, two certain task and about one in 10 didn't finish.

Then in November 4th, more than twice that and much better in the finish line, but we still have some room to grow.

I will be modest, we change several things at once, so I cannot tell you which fix mattered the most, but the birds kept getting bigger and the platform kept up.

At the end of the October, our dev platform team raised an alarm. PRS across the company had jumped around 60% in a single month. A merge that used to take 15 to 30 minutes was taking over an hour now. And merge Q kept having to be switched off. So now a PR rebase onto main rerun CI and merge if it's green. If main breaks everyone CI breaks with it. Alterately we haven't cracked this one yet.

But before I go there's one more thing. While we were busy fixing all it something else was quite growing. Most of what I've been showing you start with a person asking.

The fastest growing source of the work on Spectre doesn't these are automations agent that wake up on schedule on an event and just do the work tri alert read the PR write the post.

Last month about 110 people had automation running and we learned to give them a guardrail on day one. If one fails five times in a row, it switch itself off and it can carry a budget or expiry date because when no one's watching, an agent can start thousands of tasks, too.

What if every Slack channel had its own agent that never locks off? One that follows competition and start work on its own. What if every pure prove it works before a human even look at it? Today only one merge in 20 carries that kind of proof.

And how do you merge when agents never stop opening PRs?

Ask me again 6 months. I will tell you what broke. If you're building something like this when you work, here's three things I do again. Oh, what happens after code is written. Make Eric action something an agent can call and cap what an agent, human or alert can start before they find out what they can do with it.

And one last thought, that's my own view. Engineers can write code with whichever agent they like. The work that's lost is the harness that runs it, checks it, and get it merged.

Thank you so much for watching. Come find me on linking if you are building similar things.