AI 生成代码激增下的代码审查瓶颈与 SRE 运维挑战

For the last year or so, on top of the humans that we've been supporting for the past almost 13 years, uh we've also been starting to build support for agents, and they've been our primary customers. Uh for those agents, uh all this code base tooling, code search, code graph, code navigation has become a code base context engine.

Uh today, I do want to talk a little bit less about how Sourcegraph directly supports the hands-on keyboard work of developers building those code bases, and focus more on how AI-driven changes in software development have impacted frontline support for those services, and where Sourcegraph can help.

Um engineering teams increasingly rely on AI and agentic tools to support their work. These tools inevitably lead to significant increases in developer velocity, and code bases inevitably start ballooning.

Um organizations dedicate huge budgets to AI tooling and expect developers to use them heavily and just to justify those purchases that they're making. Uh and for the the past couple of years, the first question on basically any AI roadmap was how many heads can this save us,

right? Um in Sourcegraph's most recent customer advisory board though, that question for maybe the first time ever just didn't come up. So, the mindset is clearly shifting.

We have started to hearing this time and time again from so many of our customers. Between the increase in developer velocity, the pressure to use tools purchased by the business and the expectation that product releases should increase proportionally to the use of these AI tools, the developer priorities have never inevitably started shifting. Where the bottleneck used to be software development and in software development used to be developer bandwidth and how quickly a human can just type code, that bottleneck is now very clearly with how quickly humans can review the code that the AI is now writing.

AI isn't just writing code faster, it's writing so much more of it. PR counts have increased as PR sizes have ballooned. Uh and whether or not developers actually trust the code that they're pushing now matters a whole lot less than just getting ahead of an ever-growing list of PRs that they need to review.

AI-written PRs often lacking the organizational and in larger code base context needed to actually be production-quality gets merged into production without a human ever seeing it more often than not now.

It matters it's only a matter of time before breaking changes are missed and merged by code reviewers. And AI as you AI usage increases, so do AI-caused incidents.

Uh the context were required to resolve these incidents is lacking again because a human has never seen it and it's even harder to track down than it was before. These incidents are becoming more frequent. They take longer to resolve. They pop up on a recurring basis. They're costlier and the burden just continues to fall to our front-line support to resolve them.

Leadership has started looking to hiring more SREs as as a band-aid solution, but the role is increasingly in more and more high demand. It's getting more expensive and the rise of incidents is quickly starting to outpace the ability to hire people to resolve them.

Again, this is something that we have started hearing time and time again from our customers. Uh raise raise your hand if if this is something you guys are feeling. Yeah.

That being said, I mean of course AI still hallucinates. It's getting a whole lot better. We all love it. That's why we're here, but it's not perfect. And um this is particularly true in really large, really complex organizations maybe with a whole lot of of legacy code where the AI just doesn't have all of the context needed to actually build the product in the way that your organization might need it to. If you can't clone the entire codebase locally, the AI just needs to make judgment calls and make guesses. Sometimes they're informed, but often it just means they uh only have a partial view of the environment that they're building in.

As a result, incidents increasingly have started tracing back to unreviewed AI code.

And as a result of that, remediation just simply cannot keep pace anymore. A fix found in one repo often applies to hundreds if not thousands of others. It If disparate teams are relied upon to apply those fixes within the repos, the fix can't go out quick enough more often than not um and things start getting missed.

All of this is even more true in the industries that you all are working in. Um when you're working in a financial institution, the stakes are just a whole lot higher and you have the responsibility for meeting all of those regulatory and compliance requirements. Uh I don't want to spend too much time on these next numbers that I have on the next few slides, but just to kind of show you how some of our our industry peers are thinking about this and some of the analytics that they've they've done. Um I'll pull up a few.

Um as part of that, the code review challenge that I mentioned earlier, um so some organizations are estimating that almost twice the number of PRs are being merged now with agentic coding. The size of those PRs are is more than double the size of just a human-written PR um and that means developers are starting to spend almost twice as much time just doing code review.

Trust in AI code is still lagging tremendously. Almost every developer will say that they don't trust everything that AI is writing for them. And surprisingly at the same time, almost half of developers will say they don't always verify the code that AI is writing for them. Kind of scary for an enterprise, right?

As a result, quality and security decline, like I mentioned. Um more incidents per PR, more incidents being traced back just to AI-generated PRs as well. And stability declines as those incidents rise. Um more deployment-related issues are being traced back to AI-generated code. And there's been an increase in things like rollbacks, hot fixes, customer-impacting incidents as a result of it.

All of that at the same time, Anthropic has started reporting that because developers are becoming more and more reliant on AI to write the code for them, their comprehension of that code is also at a decline.

Um I don't want to be up here just doom and gloom. I I know I've spent a whole lot of time on on negatives here. Um but yeah, at least as of today, where AI is actually winning, it's not necessarily on this like end-all be-all broad automations of developing code, right? If it were, then we would all be out of jobs. Um AI is winning a whole lot more though when you can narrow its focus to well-bounded domains with all of the necessary context to meet your organizational and compliance requirements. Things like vulnerability remediation, security scanning, large-scale migrations, non-developers prototyping their their own tools, things like that with really bounded domains and bounded ideas um are where it starts to succeed. And that's where I think SREs and platform teams come in a whole lot more.

And because of that, they're now both kind of the barometers for how effective AI coding is and how trustworthy it is within your organization by how many incidents that AI code is producing. Um but they're also the teams that can see some of the highest gains uh from effective AI usage. Uh they already work with automation automated remediation tooling, semi-autonomous incident response runbooks, CVE monitoring and remediation.

Um so, those teams build the processes and tooling to prevent AI-initiated incidents from becoming rampant, but ironically, they're also the teams that can leverage AI within their semi-autonomous existing processes to see real productivity and safety gains.

That same tooling is also really, really effective for issue identification and resolution as well. And in the context of SREs and in platform team, that's where I want to focus for the rest of this talk.

Just to give you a high-level overview of kind of what Sourcegraph is, some of our features. Um at our core, we have a full code graph, a complete index of of your organization's entire codebase uh that really powers kind of all of our features and everything that we do. The next level up from there, as I've mentioned a few times, is code search and code intelligence. These tools that we've spent over a decade building and tuning to give human and agents the ability to navigate, especially really large and and complex codebases extremely quickly and efficiently.

Uh the graph, search, and intelligence form kind of the context engine that I mentioned for agents specifically. Uh they're accessible via our LMT P, and they really power at our core all of our own AI and agentic-enabled features uh that ensure humans and agents have the information and the context they need to build code better.

On the investigation front, with Sourcegraph, we have the tooling to find the root cause of cause of issues in minutes instead of days. Uh these issues that are being created by uh AI-written code uh are much easier to investigate, find. Uh our classic deterministic search allows you to search across every repo repo, every branch, every commit with just a single query. And then deep search, our identic search feature, allows you to easily ask what's broken, get cited answers, and be confident in the responses that that you're receiving.

On the prevention front um it's obviously great to be able to find the root cause of a lot of these issues really quickly, but it's even better to prevent them from occurring to begin with. Um at Sourcegraph, we have built-in monitoring capabilities to alert when risky code is committed, when certain patterns are matched that may be non-compliant, things of of that nature. And through these monitors, you can ensure that AI-written code follows all of those patterns and requirements that you and your organization have in production.

And then lastly, remediation. Once you've even uh identified a break or a risky pattern, how do you make sure that that fix is applied quickly and to all impacted areas of your code? Uh Sourcegraph tooling allows you to find all those impacted areas like I've mentioned, but we also have the ability to make bulk changes across all of your repos within your entire organization to remediate them and make sure that they don't occur across other services with other teams, whatnot, very, very quickly.

Um our security team built this sort of workflow organically really a while ago. Um we've been using HackerOne for a while now, even pre-ChatGPT. Uh but the volume of the reports that our security team has been receiving has been skyrocketing, frankly. Uh and they require real-time judgment to review, digest, and decide, like, is this something that the team actually needs to spend much time on? Uh so they set up uh from HackerOne webhooks that reach out to our agentic searching capabilities, and then make judgments on whether or not our team needs to dive deeper in it, maybe propose a change or remediation, if it just needs to be reviewed more deeply by our security team, or if it's something that we can ignore completely.

Um how they did that when there's a service record downtime or significant spike in their error rates, the internal root cause analysis agent that they've built within their organization automatically spins up, takes all the logs and traces, and connects those back to the source of the fault. First, you have to identify what branch or tag was deployed in production because it's rarely main, of course. And then using a agentic search over the Git history to find the specific commit and author that triggered the incident. The agent then has the ability to revert and kick off a redeployment or simply triage and share the investigation with the SRE team SRE team.

Um their eval suite is made up of hundreds of historical branches of that code base. And so their agents needed the ability to search across all of those branches um instantly, affordably, at scale without relying on the non-determinism of some of the the other agentic tools available to them. Um so they used our read-only code finder agent to make that possible. And when the evals pass, they push and publish processes are are are kicked off.

Yeah, so all in all, um if you or your SRE team or if you your SRE teams, your platform teams are really starting to see a lot of the burdens of incidents and the rise of incidents caused by AI-generated code, um if your teams don't really have a clear way and effective way for investigating uh what caused those incidents and then coming up with a very quick remediation plan to address the increasing volume of those incidents within your code base or really if your SREs, platform teams, software engineers uh are looking for better improved uh context engine code understanding platform that they use or they would like to use with all of their agentic tooling please feel free to come talk to us at our booth.

Thanks, y'all.