谁来验证软件工厂：Anthropic 与 Sonar 谈 AI 代码规模化验证

Almost everyone, thanks. And keep your hands raised up if you have merged pull requests and reviewed every line of code that was merged.

Oh, I do see a few people. That's excellent.

So, what I'm going to talk about is the gap between the uh two uh camps, uh merging requests without reviewing manually, and uh you know, merging requests uh after doing some reviews.

So, my name is Manish Kapoor, and I lead the product and solution uh developer relations uh team at Sonar, and I'm excited to be here uh with my guest speaker, Addy Osmani from Anthropic. Uh Addy has been uh had been at Google for about 14 years. Most of you probably know him. And he's written lots and lots of content and articles. Uh I learn a lot from Addy, so I'm uh really fortunate to have him here, stand next to me, and uh share his wisdom with all of us here today.

Uh he recently joined Anthropic, and Addy, so congratulations on your uh role at uh Anthropic. Would you like to tell the team what you're working on at Anthropic, what your role is?

All right. Uh so, let's start with some problem and context setting slide. There are two numbers on the slide. CircleCI analyzed 28 million CI workflows this year. And they noticed that the throughput jumped by about 59%. But yet three in 10 attempts to merge that code into production are failing. And this is a stat from one independent stat from CircleCI. Uh And I think I'm seeing the same trend from most of our customers that I speak to every day.

Uh this is a quote from Adi. A software factory is harnessing loops at scale. This resonated quite a bit for me. And if you have not read that article, you can look it up. It's pretty educational article and talking about future how the future will look like in terms of soft software factories. Uh so, Adi, would you like to expand on what you were thinking when you wrote this article about soft loops and harnessing the loops at scale?

But, that's sort of what I was thinking. I think a lot of teams are somewhere on this spectrum and are on this journey to figure out like how how do I get further and further into automation in a way that I can trust it.

So, uh AI coding is now basically a solved problem. We all know agents are there. 85% of the organizations are already using coding agents. It will go to 95% and very soon to 100%. Uh 75% of the uh 76% of the developers that generate AI code, uh they are saying that it it's being code is being produced at a speed that is impossible for them to verify it. So, they are spending more and more time reviewing the code. So, what do you think is the most pressing question that engineering leaders have right now?

Like if you are hoping to get the best quality output from your agents, you need to give your agents a way to verify that the work is correct. If you've got that in place and it you can make sure that the code is both correct and complete, that can easily two to three by like the quality of your outcomes.

Um and even for our recent refresh of Claude.ai, we put out a case study saying that we kind of three-by'd the performance of it. That was accomplished by merging about 3,000 pull requests in a very short space of time with engineers, product designers, a number of different functions contributing to it. So, our workflow for being able to get verification and code review in place is that we first of all have Claude do a first-pass review. What's very important in there is making sure that it has a way to actually try the experience out similar to a real user. So, that can mean end-to-end tasks, you know, opening up the experience, checking it in a browser, running any tools, doing any linting, doing any other checks that are necessary to build up that confidence. Um people usually ask, "Is it okay, well, what happens if, you know, you discover a bug or a gap in functionality?" Well, we have hooks that will then go and run on any edits. They will study the diffs and allow us to get this loop in place where we're correcting the PR as quickly as possible. So, we have uh Claude Code doing the review. We have a um code review product uh that you can also check out. It's pretty good. And the idea there is we're trying to automate code review um as much as possible. This is something that we use ourselves. Um and increasingly what's important is for your teams to be thinking about how can you cut We talk about taste and judgment so much these days, right? It's important for you to be trying to codify what you consider best practice and taste in your team's shared like Claude.md, agents.md, anywhere where the agent can then consult during code review. And then people are responsible for checking on the agent's evidence.

So, how do we know? Some If I'm building out um a piece of UI that's going to be, you know, used by a user in the browser, I want to make sure that it's tested in the browser, maybe generated me a video of that experience where it's been clicking around and gives me confidence. Well, okay, well, this works. If I care about the mobile or desktop experience, maybe it generates me screenshots or videos of that. But basically, you want to get to a place where the agent's giving you as much verification evidence as possible.

Uh we surveyed about a thousand more than a thousand developers and we found some interesting data which will give us insights as to what's happening in the industry today and where we are headed.

So, 84% say AI features uh uh gets gets features and fixes put to production faster.

The pull requests are getting larger and more complex. Uh and you know, yet uh only uh 31% more code is reaching reaching the production without review.

So, that's a little little bit of a scary fact particularly for uh production uh applications in the regulated industries in the mission-critical applications where you absolutely make need to make sure uh that you know, uh you have to verify that code and you're confident in the code that is being shipped. So, uh what are the where are the next gains going to come from, Addy?

Yeah, that's a good question. How do you define a software factory? Would you like to take that question?

The more that the agents are able to generate verification and give you confidence, the more that you can get to that place where you go dark. And I think that yeah, most people are still somewhere in that spectrum and are not quite as this data shows at that dark point just yet.

And automating them at scale where there's pretty much in a dark factory, there would be no lights on. It's autonomously working, fully autonomously working. A light factory has uh is where people are involved in the process at every step or most steps. Uh whereas a dark factory, the goal would be to turn off the lights and let the factory run independently 24/7. Uh all right. So, you talked about verification.

Uh I'll talk a little bit about what we think at Sonar about verification. If you saw our keynote earlier today from our CEO, he talked about verification should be multi-layered. Uh we absolutely 100% believe that each company should adopt a zero trust multi-layered verification approach. And verification should be done in every loop. Whether you're operating inside the agentic loop or whether you're working inside the CI verification loop or whether you're doing code maintenance. In each and every loop, it is important to verify the output of the factories or the agents that code that the agents are producing.

So, we look at it in terms of layers and think of it like, you know, the there's first layer which is a deterministic layer which catches uh bugs, security issues, things like SQL injection, buffer overflow bugs, and all of those are caught. And then there's a second layer where agentic review happens where agents are used to like Adi was talking about, agents review review that code. And in that agentic review, they catch logic and intent issues which the deterministic layer is not well suited to catch. So, we we use each layer is targeted at catching a specific types of issues. And then the third layer could be a software supply chain security layer where you look at all the CVEs, all your malware detection happens, all your malicious package detection happens. All of that happens in the third layer. So, it's a layered approach where each layer is tuned and optimized to catch a certain type of issue.

So, we absolutely recommend doing a multi-layered verification. Oops.

And 5% are operating software factories autonomously. What are your thoughts there, Adi? Uh in terms of

All right. Uh here's my last slide and then we'll wrap it up. Uh so what separates uh teams that are shipping uh fast but not shipping blind? I think that three key takeaways for you. Uh writing code is the fastest part and that is solved. Strong review and verification for our teams is what will will help you maintain that speed. Fast to write is not fast to ship. Uh you have to basically count what is getting merged, looked at what is getting merged, and make sure it doesn't escape the sandbox and the things like that before it gets into the production.

Verify everything. And autonomy is earned. Uh you We will get to software factories very soon if not next year. Uh you need to put in independent automated verification inside each loop of your factory. Make sure that you can trust the output that the factory is producing.

Uh, with that, I'll wrap it up. Thanks for attending our talk today, uh, a conversation with Adi. Uh, again, Adi, I appreciate you taking the time and joining us here in this conversation. Uh, visit us at the big red booth, uh, the sonar booth in the expo hall if you have any further questions or would like to, uh, talk about it. Thank you again.