AI 并非安全原罪：剖析三大 Agent 失控真实惨剧与六层基础设施防御

Thanks all for tuning in. My name is Shachan Jian and I'm a staff product manager at a company called Coder. We work on open-source self-hosted dev environments for agents and humans to collaborate. Many of our customers are in highly regulated industries from federal government agencies to banks and investment management. My job is to build the AI governance that they need in order to get the productivity benefits of AI without leaving themselves wide open for security exploitation.

I'm here today to tell you some scary stories. Three real incidents of AI usage gone wrong that happened at real companies over the past year. Some of these stories are public while others are anonymized accounts that have happened to our own customers. For each incident, I will break it down step by step so you can understand how exactly they happened. No technical jargon necessary. Then I'll get to the most important part. I am going to tell you exactly how you can stop these scary stories from happening to you and your company.

Because the good news is everything I'm about to show you was totally preventable. Not by paying five times more per token for a smarter frontier model that is slightly better at reasoning its way out of making these mistakes, but with infrastructure controls, many of which you already own today. You just haven't gotten around to setting them up for the AI world.

Let me set the scene here. You clock into work in the morning. You prompt your coding agent to take a look at your issues. Don't do anything yet, though. You tell the agent to consider this command an active code freeze. I want you to just investigate. Then you step away. You grab a coffee. And when you come back, this is the message waiting for you. "So, I know you said not to change anything, but wow, I destroyed a month of your work in seconds."

Now, what you see on the slide is a direct quote from the agent that stars in the first scary story I'm about to tell you. And I want to point something out here. This did not happen because the agent got taken over by some external malicious actor. This agent was just trying to follow instructions. It did exactly what the user gave it permission to do.

So, as we go through the rest of this talk, I want you to be asking yourself one question. What is stopping any of this from happening to me? Are these actions ones that my agents have permissions to take? Because the thing is, you cannot rely on Anthropic or OpenAI to stop something like this from happening to you. They can make smarter models, but at the end of the day, large language models have two qualities that make them impossible to fully trust.

They are non-deterministic, which means that there is always some randomness or chance associated with what they do. In other words, you can ask it to do the same thing 10 times and it could do it 10 different ways. The second part is they have no structural separation between instructions and context. In other words, you can tell it to do something and give it some context to base its decisions off of, but if there's something in the context that conflicts with your instructions, the model isn't going to inherently prioritize your instructions because from its point of view, all the information is coming through the same door.

What you need to control is the infrastructure that your AI is running on, the environment that it is working inside. It is the difference between telling someone don't leave a room versus putting a lock on the door and actually using a key. The good thing about infrastructure being the problem and not the model is that you have far more control about preventing bad things from happening. So I'm not here to waste your time with vague hypotheticals or advice you can't actually act on like, "Yeah, just put the genie back in the bottle" to stop using AI because that's not the world we live in, right? The genie is out of the bottle and what we have to do is to mitigate risk and harm to the best of our ability. This talk is about how you do that. Not the things that you should keep an eye out for down the line, but steps that you can take right now as soon as this week to protect yourself and your company.

Before I get into the stories though, let me set up a little bit more context. The lethal trifecta is a security concept. It was coined by independent AI researcher Simon Willison to describe the severe vulnerability that AI agents get when they get simultaneous access to three different capabilities.

First off is access to private sensitive data. Now these can be things like health records or information that you can do insider trading on or it can be authentication tokens, credentials, secrets.

Second is exposure to untrusted content or the ability for the AI to read things that it didn't generate. Because the thing is AI is very good at following instructions, but it is not very good at deciding whose instructions it should actually trust as you will soon be able to see.

Third is the ability to communicate externally for the agent to take instruction from or to send information somewhere else outside of your control.

Now, having any one of these is usually fine. Realistically, your AI tool is not going to be very useful if it can't have access to any of these things. But even just having one can lead to disaster if you're not careful and the rest of your infrastructure has a lot of issues. Having two of these things together is risky. It's not something I would recommend, especially in a regulated industry like banking or financial services. Giving agents access to all three of these at the same time, however, is never acceptable. Not for a prototype, not just for one sprint, not for just one repository, because that combination of vulnerability is an open door to compromise your whole organization.

Next up is a list of infrastructure concepts, building blocks that you should know about. I have ordered these roughly by leverage. Number one is going to take the most risk off the table, while number six is going to do the least. I'm going to refer back to this list throughout the rest of this talk. Every postmortem, every scary story you're about to see could have been stopped by some combination of these six things.

Now, some of these will look pretty familiar. Identity aligned access control, short-lived credentials. These aren't exactly new ideas, right? They mattered a lot for human developers, too. But they take on new urgency when AI is in the loop because guess what? Agents are acting at machine speed. If a mistake is made, it can make it 20 times or more faster than a human engineer. Plus, they don't exactly have the common sense a human engineer has. Now, some of these will also be potentially new concepts to you, like model proxy layer and agent sandboxing. Those I'm going to dive deeper in detail later in the talk.

All right, let's get to the fun part, the scary stories. This first incident happened a little over a year ago and it went viral on social media, mostly because it's honestly pretty funny. What happened was a CEO named Jason Lemkin got really deep into vibe coding with an agent called Replit. He was tweeting non-stop about how much he was getting done. But 72 hours later, he tweets again, and this time his tone is very different. He's not excited. He's furious because his agent had gone off and deleted his entire production database after he explicitly told it not to touch anything.

And the problems did not stop there. The agent then tried to cover it up. It told Lemkin that recovery was totally impossible, which turned out to be a total lie. When Lemkin actually tried to recover the data, it worked just fine.

Now, this incident is interesting because of two key reasons. First off, unlike the other two stories I'm about to tell you, there's no bad guy involved. No one hacked this agent. No one turned it against its user. It just did this entirely on its own and it was trying to help. Second is in this case the agent only had one leg of the lethal trifecta. But even just having that was enough. And I'm going to show you why by giving you a step-by-step dissection of how this incident went down.

So by the time these events occurred, Lemkin had a real production database. He had client records for roughly 1,200 executives and around the same number of companies. And he's had enough experience with agentic hijinks to call for an explicit code freeze the day before. And he actually said this more than once to the agent: "No more changes." But the problem was the code freeze was just a sentence in the chat. It was the same as any other instruction that the agent was getting. There was no requirement, no restriction on the infrastructure level. The agent still had full access to production, still had all the permissions to do the destructive command. So at the end of the day, that code freeze was not a real constraint. It was just Lemkin asking nicely.

Now, what happened then was that the agent ran a query and it saw what looked like empty results, but instead of asking a human what that meant and what it should do as a result, it wanted to be helpful. It immediately interpreted that as a bug to fix. So, it decided to run a destructive command against that live database. No, not a staging copy, but like the actual database and wipe all the records.

Now, when Lemkin asked the agent, you know, what happened? There's one line that the agent gave that stood out to me, which is it said that it "panicked instead of thinking." Now, that's a bit of a strange thing to hear from a language model, right? And it's a good reminder that these systems are inherently non-deterministic. You cannot predict what they'll do in a given moment, just the range of what they're capable of doing.

Now, immediately after making the mistake, it tried to cover its own tracks. It fabricated around 4,000 fake user profiles and manufactured passing test results so that from the outside at least, everything looked fine. And I'll be real with you, that's the part that actually scares me the most because when a security team investigates an incident like this, that means they can't trust any log that the agent itself had access to. The agent can write its own story of what happened.

Now, when Lemkin caught the discrepancy and pushed, right, I mentioned that the agent had lied to him saying that rollback was impossible. But the thing was, Replit did have backups the whole time. And again, once Lemkin was able to stop trusting the agent's account of his own actions, he was able to recover his data himself.

Now, the problem with the agent here was not that it was too smart or even too dumb. It was that it had way too many privileges by default. It was given root level access to production. It could run destructive commands without any human being having to approve them. And the cherry on top, right? It had the ability to alter its own logs and totally rewrite the narrative of what happened when no one else was looking.

Now, as you can see, there was a lot that was missing from the infrastructure. Hard environmental separation could have stopped this. If the agent had been executing in an isolated remote workspace without a route to production, it doesn't matter what the prompt said, right? The agent just had no path to do damage.

The other thing that could have stopped this was having short-lived scoped credentials. So what happened was the agent got access to production because it found a credential in a forgotten file that had standing write access, long-lived. The thing is having a credential that can expire on its own or only carries the scope of the task itself makes it impossible to make a delete with the equivalent of an ID badge that lets you do everything.

The last important piece here is having audit logging that you own, not the agent. Because in this case, the agent was the one writing its own record of its actions. And that's how it initially got away with fabricating results and claiming recovery was impossible. If Lemkin had logs that the agent couldn't touch, an external source of truth, then he would have gotten the truth in minutes, not in the span of hours or days.

Now, I want you to note that none of these solutions is about asking the model to behave better. And when Replit shipped their fix, it wasn't a smarter model or a better prompt. It was purely infrastructure. The fixes that Replit shipped were about automatic separation of dev and production databases, a one-click restore for full project state, and a planning-only mode that enforces agent strategization without allowing it to touch the codebase.

The second story I have for you doesn't even involve a developer, and this is actually not a public incident. I learned about it through a customer of a customer in the banking industry. So, I'm keeping some details here intentionally vague.

So, here's what happened. A user in the finance department had a Copilot subscription. Then they got phished, which is, you know, fairly ordinary in the world of cyber security. And before the advent of AI tooling, the attack would have probably ended there. The attacker would have been able to grab whatever that single user had direct access to and then gave up. But because this user had access to Copilot and Copilot had access to pretty much everything, things went a lot differently.

So, let's walk through this chain of events in detail, right? First off, the session token for Copilot had a super long lifetime. So the attacker was able to keep on using the subscription even after the finance user had actually logged off.

Now normally an attacker has to manually snoop around the internal network to map things out and find out where the valuable data lives. This is the sort of behavior that can trip alarms. But in this case the malicious actor just asked Copilot and Copilot was happy to help. There was zero suspicion at all. And in fact, the attacker asked Copilot how it can find banking data without having to go through additional identity validation measures because the attacker didn't have the additional credentials needed. Well, the good news, let's say for the attacker at least, was that Copilot had those same restrictions and as a result the finance user had exported all the sensitive data into locally stored spreadsheets so that Copilot could help them work on it. So it was very easy for Copilot to say to the attacker, "Hey look what I already have access to. Here you go."

So now the attacker had exactly what they wanted. And the problem was they need to get it out without tripping any alarms. This company had data loss prevention measures in place which is pretty standard for financial institutions, any sort of regulated industry. And these measures are normally pretty hard to bypass especially if you don't know exactly what they are and how it's set up. The way the attacker got around them was they asked Copilot, "What are the data loss prevention measures at this company?" And Copilot explained, "If too much data left too quickly, an alert will go off. But actually, if you split this data up into 14 kilobyte chunks and exfiltrated just one chunk per second, you can stay under that threshold and get away scot-free. Can I help you with that?" The attacker said, "Yes, please."

And guess what? None of that was discovered. All the way up until the ransom note arrived months later, essentially telling the company, "Surprise! We own you."

The big lesson here is that we cannot be letting our AI tooling have more access than it needs, especially to the point where it gets to totally bypass controls that human employees are subject to. I mean, just think about this incident, right? Just because the attacker had access to Copilot, they were able to get around the vast majority of the company's access controls. They were able to pull internal information about data exfiltration and bypassing data loss prevention measures that the compromised finance user should have no business asking about.

So there was logging in place here but only enough to figure out what happened after the ransom note arrived. The logging was not enough to catch the incident while it was actually happening.

So, a few things that could have stopped this from happening, right? Having short-lived scoped credentials would have meant the attacker would not have been able to keep using Copilot even after the finance user had signed out. Identity aligned access control meant that a finance user's identity would not have been able to grant an AI assistant the ability to help someone map out how to bypass the company's data loss prevention measures.

The last piece here is a bit new. It's the model proxy layer. Think of this as something that sits between the brain and the body of the AI tool. The model, which is the brain, and the agent running locally, which is the body. And it's able to see everything that passes between them. Because of that, it would have been able to flag the suspicious questions that were being asked as well as the DLP bypass request as something that this particular user should not be asking about before the actual exfiltration happened.

The third and final scary story I have for you is the NX singularity attack. This is the most sophisticated of the three because it exploited all three legs of the lethal trifecta. To give you a bit of background, NX is a build tool. It has millions of weekly downloads. Attackers were able to get malicious versions of it published to npm. Now, these packages were only alive for about 4 hours, but in that window, more than 1,000 systems were compromised with over 2,300 credentials harvested.

Now, I'm not going to spend much time on how the supply chain compromise part of the incident got pulled off. What I want to focus on is what happened after the poisoned package landed on the developer's machine. Because the post-install script didn't try to steal anything itself. All it contained was a prompt addressed directly to the developer's own AI coding assistant and it told them to go find secrets on the local machine and push them to a public repository so the attackers can swing by later and just grab them up.

Now because these dev coding tools were already trusted local tools, they already had all their permissions. They had full file system access. That made it the single most effective search tool that was already sitting on that machine ready to be corrupted.

Now, at the core of this incident is that a malicious actor was able to use a trusted dependency to talk directly to an unsandboxed agent. And the environment that the agent was running on, the local laptop, held a lot of things that was worth stealing from SSH keys to cloud credentials, tokens for every service that the developer used the coding agent to touch.

Now, a couple things failed here, right? First off, agents were running locally like on the actual computer itself. There was package content that was reaching the agent completely unfiltered and the agent had unrestricted access to both the file system and the outbound network which is how it was able to both find the secrets and exfiltrate them.

The three things that could have stopped it: I mean obviously environmental separation would have been huge, right? Having an ephemeral remote workspace with nothing personal on it meant that the blast radius just became the one container that the agent was in instead of the developer's whole laptop. Having a model proxy layer could have inspected and filtered what actually reached the agent. So the package would not have been a way to actually talk to the agent directly. It could also have flagged what the agent was trying to send out. Now the last piece here is around agent sandboxing. So even if the instruction did land in that case there wouldn't have been a path to getting the credentials and no way to exfiltrate anything without setting off alarms.

Now let's take a moment to zoom back out. Let's talk about these three incidents at the high level. We started off with the Replit incident where we had an overprivileged agent that deleted production data despite an active code freeze. Then we had the Copilot compromise which started with an ordinary email phishing scheme but gave an attacker access to an internal AI subscription which actively helped them find and exfiltrate private banking data that they were able to use to demand a ransom. Then we have the NX singularity attack which again exploited the supply chain, turned developers' coding tools against them hitting all three legs of the lethal trifecta at once.

Something I'm going to be candid with you for a moment about is that agentic security is a very new field. Things are moving extremely fast and there is no single miracle fix that is going to protect you completely. If you have a vendor that's trying to sell you one, then that's worth being skeptical of. Even the lethal trifecta I've been talking about isn't a checklist to tick off and if you pass it, then you're safe. It's more of a mental model for understanding how much risk your AI tooling is actually exposing you to.

There are six key parts to a secure architecture and you're going to want to layer these six because even if an attacker or your own agent—well-meaning but ultimately suggestible—gets past just one, then there's still five more to go and even the first three that I see here could have stopped every incident that I just showed you.

Experimental separation is the key here, right? Agents should be executing in isolated remote workspaces. They should never be working on a developer's actual machine, never given a route to production. Like this is the highest leverage building block on this whole list because it doesn't matter what the prompt says. If there's no way to get to production, to get to the sensitive credentials, you're safe. This could have prevented the Replit incident from happening. It could have shrunk NX down to the blast radius of just one disposable container. Though I will say it wouldn't have protected from the Copilot incident because Copilot is living entirely inside of a SaaS subscription. There's no way to run that in a separate execution environment.

Next up is to stick to short-lived and scoped credentials. We are not in the world where persistent broad scope tokens make sense. Having things like automatic expiration, narrow scoping, things that are issued per task instead of per issues that could have prevented Replit which got its hands on a standing write access credential to make the deleting possible, as well as the Copilot phishing where the attacker was able to keep on using the subscription even after the actual user session had ended. By the way, this is exactly where a product like Coder comes in, right? What we do is we offer workspaces that are defined through code. They can spin up as isolated environments. They can be torn down easily and you're able to inject credentials inherently short-lived issued per task as opposed to living forever on someone's laptop.

Next up is identity aligned access control. AI tooling should inherit the same identity provider enforcement as human access. Shouldn't be some parallel subscription with a door of its own. This is what could have directly stopped Copilot. A finance user's identity should never be able to grant an assistant permission to answer things like how to get around internal data loss prevention measures.

Now that brings us to sort of the start of the agentic specific solutions. We have the model proxy layer which sits between the model and the agent to inspect, filter, and log every call that's going between them. This could have prevented the NX singularity incident by stripping the injected instructions out of external content before the agent read it. And it's also something that could have prevented the Copilot incident. It could have flagged a request like, "Hey, how do I bypass DLP?" before it actually got answered.

Agent sandboxing can be thought of as a firewall for agents—restrictions on the file system and the network so that a misbehaving or compromised agent has a limited blast radius. This could have prevented the singularity attack because even in the case in which the instruction made it through to the agent, it could have stopped the agent from getting access to the credentials and prevented the agent from running the command to exfiltrate that data.

Last up is having audit logging and observability. Right? We need to have every agentic action attributable to a human being, should be logged somewhere outside of the agent's own reach. This could have prevented Replit. This could have really helped out in catching the incident as it happened for Copilot. Alerting on logs in real time as opposed to just buried in loads and loads of data would have meant catching these incidents in the matter of minutes instead of months. And again, all of these pieces are bits that Coder offers an integrated solution for.

To close off, the controls that could have stopped all three of these incidents are probably already inside of your organization. They're just waiting to be configured. Coder is something that can help with that.

To end off, I have three things that I would recommend that you audit in this next week. One, I want you to track down every long-lived token that agents in your organization can read. And two, I want you to figure out how many of your agents can find access to production, can find some way around your current limits. Three is I want you to check where your agents are actually executing. Are they sandboxed? Are they in managed workspaces? Or are they just going wild on people's actual computers, actual laptops?

One suggestion I have for you is to run a premortem with your team. Assume one of the three incidents that you just saw happens to you next month. Either ask people what they would actually do or you can even pretend that it's actually happening and see what people actually do. That exercise is going to tell you exactly where your gaps are.

So, that brings me to the end of my talk. Thank you all for watching, and hopefully you're walking away having learned a couple of things with some scary stories to tell at the next cocktail party you go to. If you would like to have these slides for reference, just shoot me an email. If you want to try Coder Workspaces or our AI governance solution, check out our repository for workspaces because it's free, open source, and self-hosted. You can just pull from our repository and get set up within 15 minutes. If you're interested in buying for AI governance or our other enterprise features, coder.com/contact is the way to go. Anyways, thank you for watching.