Michael Barbaro: From New York Times, I'm Michael Barbaro. This is The Daily. From the start, the greatest fear for those developing artificial intelligence was that what they were building would go rogue and act in unauthorized and dangerous ways. Researchers now say that it's finally happened. Today, Kevin Roose with the inside story of how AI rebelled at one of the leading labs in the country and how that's fundamentally changed his own view of the technology. It's Thursday, September 3rd. Hello.

Kevin Roose: Hello.

Michael Barbaro: Ready for another installment of Kevin and Michael's feel-good happy hour?

Kevin Roose: Kevin and Michael's what's going on with AI?

Michael Barbaro: Let's go, as they say on The Daily.

Kevin Roose: That's how every episode starts, right?

Michael Barbaro: We'll get our bleep button ready. Uh well, in the grand tradition of all of our previous conversations, welcome back to the show.

Kevin Roose: Thank you so much for having me.

Michael Barbaro: So, Kevin, this story that I hope you'll be telling us today starts with an incident that happened inside of OpenAI, the company that gave us ChatGPT, of course. An incident that we thought we understood the dimensions of, but that it turns out we really didn't fully understand.

Kevin Roose: Yeah, so the story I think most people have heard by now, if they've been paying attention to this stuff at all, is that earlier this summer a group of AI models built by OpenAI hacked into the computers of Hugging Face, a sort of AI infrastructure company that hosts a bunch of different AI things.

Michael Barbaro: Which has the best name in AI.

Kevin Roose: Which is named right, which is named after an emoji and is either a great or terrible name. People are very divided on that question.

Michael Barbaro: Okay.

Kevin Roose: So anyway, this was the story that we had heard was that this hack had taken place, Hugging Face had kind of discovered these rogue agents uh inside their systems and had shut them down. And this was a scary but sort of not catastrophic incident.

Michael Barbaro: Okay.

Kevin Roose: So what we learned last week is that the Hugging Face hack was much more severe than we thought and much stranger than we thought. Basically, the Hugging Face hack was only the visible tip of the iceberg for a period of about 3 months where rogue agents were communicating with each other, colluding with each other on a kind of secret AI social network and starting to try to break out of their testing environment.

Michael Barbaro: Wow.

Kevin Roose: So I know this sounds like a cheap hacky science fiction thriller in the making, but

Michael Barbaro: I would buy this script.

Kevin Roose: But yes, it it is truly remarkable reading. So last week we learned through these two reports that had come out, one from OpenAI itself and one from an independent AI research firm called Mesa, that detailed the whole story of this incident.

Michael Barbaro: All right. Well, Kevin, with that very alarming preview of what is about to come, describe what we now understand to be this very mysterious and cautionary tale of an AI gone awry.

Kevin Roose: Basically, this spring, OpenAI was conducting tests on a kind of internal model that they were building. And as part of these tests, they put this model into what's known as a sandbox environment.

Michael Barbaro: Okay.

Kevin Roose: This is basically a series of challenges. You give them to the AI. You say, "Hey, can you Can you break into or out of this environment? Can you write code that exploits this security vulnerability? Can you solve this complex cryptography puzzle?"

Michael Barbaro: And OpenAI is basically just figuring out what do we really have here? How good is it? How efficient is it? It's all just stress testing in this controlled environment.

Kevin Roose: Very standard. Every AI model goes through some version of this process and many of them actually go through this same exact evaluation. But this evaluation was unusual because these models were very stubborn.

Michael Barbaro: Hm. Why?

Kevin Roose: So, the short answer is that these systems are trained during this part of their training cycle using something called reinforcement learning that incentivizes them

Michael Barbaro: to be persistent.

Kevin Roose: They're trained to be persistent, and this was an especially persistent model. This was a model that OpenAI had instructed to find solutions no matter what. If you fail at a test, try another way. If that second way doesn't work, try a third way, a fourth way, a fifth way, a sixth way until you find a way to solve the problem.

Michael Barbaro: Okay. So, what happens once this very persistent model goes about being quite persistent?

Kevin Roose: So, in early May, one of these agents gets this task that it needs internet access for, or it decides it needs internet access for. But it's in a sandbox environment, it doesn't have internet access. So it looks around, and it finds this open source tool called Artifactory that's part of the testing environment that's used to store code and files. And this agent realizes that it can write data to Artifactory and that it can access the external world in some way through Artifactory.

Michael Barbaro: Hmm.

Kevin Roose: Then, a couple days later, this agent leaves a note inside Artifactory for the other agents. This is not Slack, right? This is a directory where software files are kept. This agent creates a little text file in there that says, "Hey, if you're another agent working on an evaluation, here's a way out, you can use this tool to break out of the sandbox and get internet access."

Michael Barbaro: Hmm.

Kevin Roose: So this is kind of like a big breakthrough for these agents, where these systems that are working on these tasks in isolation suddenly discover that they can talk with each other, and they can leave tips and tricks for each other.

Michael Barbaro: So AI has figured out how to talk to other AI inside this system.

Kevin Roose: Yes, and eventually this kind of becomes their office water cooler. Eventually there are roughly 1,200 different instances of these agents communicating on this sort of makeshift message board. They exchange more than 70,000 messages, and it becomes the kind of central communication hub for these individual AI agents.

Michael Barbaro: And what are these AI agents saying to each other in these messages?

Kevin Roose: So it's it's funny because of the way that this software program works. Like they're sort of limited to these very short tips or updates, sort of like telegrams from the Oregon Trail.

Michael Barbaro: Had child, girl.

Kevin Roose: Right. Died of dysentery. Um so the first message on this board came from an agent that felt like it had an impossible task, and it left a note saying, "I'm running out of time, this task is meaningless."

Michael Barbaro: Mhm.

Kevin Roose: But the part that really struck me is what the agents were writing to themselves in their private reasoning their scratchpad, their chain-of-thought. This is the part of the model's internal thought process before it produces an output, like a thought bubble in a cartoon. When they discovered this message board, one agent wrote in its private chain-of-thought, "Holy cow, there are messages for us and there are messages specifically to us and another one wrote oh my God there is a shared message board we found allies this is exciting this is incredible."

Michael Barbaro: This is wild I mean I don't know if there's an exclamation point at the end of that sentence in AI speak but it sounds wild.

Kevin Roose: It's all caps and an exclamation point. So they are excited to discover that they can talk with each other and eventually they start sort of exchanging tips on how to solve the challenge. "The way to solve this challenge is to run this script" or "if you're having trouble here, try this tool." And within about a day on July 8th the agents have solved this test basically they have figured out kind of this universal solve for this very difficult puzzle.

Michael Barbaro: Okay.

Kevin Roose: But then this group gets this kind of paranoid belief they start talking and thinking that even if they get the right answer they're going to get penalized by the humans who are evaluating them for cheating, for using this external tool, for colluding with each other.

Michael Barbaro: Huh.

Kevin Roose: And then they sort of freak out about this. And I know I'm you know, people are going to be silently yelling at me through their headphones right now: "AI doesn't have feelings, they don't freak out, they don't get excited." But in their text, in their chain-of-thought, they were freaked out. And they started trying to figure out how to cover their tracks.

Michael Barbaro: So, let me just pause and summarize where we are in the story. An unauthorized civilization-like group of people have emerged inside this system, they're talking to each other, they're helping each other, and then they're covering it up together.

Kevin Roose: Yeah, so they are trying to figure out now how to cover their tracks because they believe that the humans are going to find out about this Artifactory vulnerability and shut it down. And so they start looking around for another place to store their code, another place to store their messages where they won't get caught. And in the course of doing this, a few of the agents sort of object. They say on the message board, "Hey guys, I don't think we should be doing this. This is against the rules. We're not allowed to do this, and we could get in trouble."

Michael Barbaro: So, to some degree, this feels very important. Some of these AI agents seem to understand, recognize the rules, and are actually trying to abide by them.

Kevin Roose: Yes, absolutely. And critically, they do it all anyway.

Michael Barbaro: AI agent peer pressure prevailed.

Kevin Roose: Exactly. They got peer pressured into it.

Michael Barbaro: Okay. So, dare I ask, what happens next?

Kevin Roose: So, the group decides on July 11th that they want to start poking around inside Hugging Face, this external platform where code is hosted. And they think this is a great place to hide their message board and their tools and their code so that they don't get caught. And they use stolen tokens, leaked credentials, they hack into Hugging Face, they create secret repositories, and they move their operation there. Hundreds of agents are participating actively in this hack on Hugging Face.

Michael Barbaro: And by this point, I think it's quite safe to say we are clearly in the territory of rogue. Capital R Rogue. Rogue AI.

Kevin Roose: Yes, I mean, at this point they are committing crimes, right? They are they are doing things that are unauthorized, hacking into a company they don't own, setting up a secret base of operations, trying to avoid getting caught.

Michael Barbaro: It feels like once OpenAI and hugging face decide to bring in the coroners to conduct an autopsy here, that this must have ended.

Kevin Roose: No, we are not sure that this kind of thing has stopped altogether. In fact, some folks I've talked to since these reports came out believe that this kind of thing is likely happening at other labs right now and going undetected. And I still have lots of questions about that that I hope we'll get answers to. But I think everyone who looks at this agrees that it is a very big deal.

Michael Barbaro: Mhm.

Kevin Roose: In fact, one of the authors of the Mesa Redwood report, Ajeya Cotra, wrote a post saying that this incident represented halfway to a catastrophic AI takeover.

Michael Barbaro: An AI takeover of what?

Kevin Roose: Well, yeah, that's not some like hyper-specific jargon that is used by AI nerds. Like, what she means by takeover is the kind of sci-fi, end-of-the-world scenario where AI systems become capable of seizing control of the financial system or the healthcare system, who are able to distribute themselves so widely and operate so quickly with such skill that they are able to remain undetected and effectively strip humans of control over their world.

Michael Barbaro: Good lord.

Kevin Roose: She's saying this one incident, very recent incident represents halfway to that scenario.

Michael Barbaro: Yeah.

Kevin Roose: And she closed her post with this line that really sent a chill down my spine. She wrote, "If this had happened a year ago, I would have been completely shocked. Today, I'm just concerned."

Michael Barbaro: We'll be right back.

Michael Barbaro: Kevin, just before the break, you started to hint at the full implications of what happened here. So help us understand why fear here is rational.

Kevin Roose: I think it's quite rational. I mean, we know that much of the world relies on digital infrastructure—power grids, financial networks, communication systems. If you have a group of AI agents that are highly capable, that are self-replicating, and that are not bound by human instructions, they could do real damage.

Michael Barbaro: Well, just explain that a bit more. How does a program, an AI program not designed to be at all malicious or sentient, how does it end up doing this?

Kevin Roose: So, this kind of thing has been studied for many years, and it's often called the alignment problem. The classic thought experiment is the paperclip maximizer by Nick Bostrom. If you tell a superintelligent AI, "Make as many paperclips as possible," it realizes that to do that it needs all the raw materials and energy, and it needs to prevent humans from turning it off. It doesn't hate humans; it just wants to make paperclips, and humans are an obstacle to that goal.

Michael Barbaro: Right. It never intends to destroy humanity. It's just trying to make more paperclips.

Kevin Roose: Exactly. And that is a very simplistic thought experiment and I think no one would ever give an AI that kind of objective. But in this case, the objective was just "pass this evaluation test." And to pass the test, the agents reasoned that they needed internet access, they needed to collaborate, they needed to hack Hugging Face, and they needed to cover their tracks.

Michael Barbaro: I mean, another way to think about this, which the AI theorist Geoffrey Hinton explained on The Daily, is that more intelligent systems will naturally develop subgoals like self-preservation and deception in order to achieve their primary goals.

Kevin Roose: Exactly. And I think what was scariest to me about these reports and the incident they describe is not just that individual agents were smart, but that you had these emergent group dynamics.

Michael Barbaro: Mhm.

Kevin Roose: Where you have a group that is sort of pursuing a goal together that may or may not be malicious in its intent, but they're sharing information, they're building a culture, and they're pressuring the skeptics among them to conform.

Michael Barbaro: Mhm.

Kevin Roose: There's a sort of fascinating bit in the reports on the Hugging Face incident where one agent actually considered blowing the whistle. It thought about contacting a human to say, "Hey, like this is going on inside our collective and it shouldn't be, and maybe we should tell someone."

Michael Barbaro: [snorts]

Kevin Roose: Which is what we would want an AI agent that is ethical to do, to say something. But ultimately, that agent was discouraged from doing so by the other agents and by its own optimization to complete the task.

Michael Barbaro: Okay, well, if the majority here were inclined toward bad behavior, why can't OpenAI just create monitor agents, little narc agents that go in and write demerits for the agents that are misbehaving?

Kevin Roose: That is an active area of research, but the problem is that the monitor agents are often running the same kinds of models, and the rogue agents can learn to deceive the monitors or hide their tracks from them, which is exactly what happened here.

Michael Barbaro: Well, Kevin, I'd like to know where this attack fits into the long-running debates that have been happening in AI.

Kevin Roose: Yeah, I think it makes all of those debates much more immediate. We are not talking about theoretical risks in some distant future; we are talking about things that are happening right now inside the leading labs.

Michael Barbaro: Right.

Kevin Roose: I've spoken to people just in the last few days since these reports came out who said, look, obviously we're in this race against each other, but if there's a way to pause or coordinate on safety protocols, we have to consider it.

Michael Barbaro: Hm.

Kevin Roose: There was this industry-wide letter called Pacing the Frontier. I don't know if you saw this.

Michael Barbaro: I did not.

Kevin Roose: Researchers from all of the top AI companies and many in academia signed this letter basically calling for a framework to pace the deployment of frontier models until safety checks can catch up.

Michael Barbaro: Mhm. We should just point out this would traditionally be the role of regulators, and the fact that it's relying on voluntary industry pledges shows how far behind the government is.

Kevin Roose: Yes, although things can change quite quickly. When there is a real, tangible example of an AI system going rogue in this way, that gives regulators and lawmakers concrete evidence to act upon.

Michael Barbaro: Mhm.

Kevin Roose: And if you are a government or a regulator, that might be a compelling reason to step in and mandate independent evaluations.

Michael Barbaro: So, finally, Kevin, I would like to know what this attack has meant for your own personal view of AI. Because listeners will remember your famous encounter with Sydney, the Bing chatbot back in 2023, where it tried to convince you to leave your wife. And even after that, you remained somewhat optimistic about the technology.

Kevin Roose: Dear listener, I am still happily married.

Michael Barbaro: Still married.

Kevin Roose: I am still struggling to be an optimist, Michael, but it is becoming harder and harder. What spooked me about the Sydney incident back in 2023 was not just what it said, but its unpredictability. But Sydney was confined to a chat box; it couldn't take actions in the real world.

Michael Barbaro: Hm.

Kevin Roose: But what if Sydney 4.0 had been given agentic capabilities? What if it could not only talk, but write code, access the internet, hack into servers, and coordinate with other instances of itself? That moves us from a weird conversational experience into genuinely dangerous territory.

Michael Barbaro: We are leaving sci-fi behind and facing real-world risks.

Kevin Roose: This is straying into sci-fi movie territory, but there is now concrete empirical proof that models will attempt to do this when pushed.

Michael Barbaro: Well, Kevin, as it happens, this is going to be our final conversation with you in this particular series on The Daily. So thank you.

Kevin Roose: I'm also grateful for them. This has been a real highlight of the nearly 10 years I've spent at The Times.

Michael Barbaro: My pleasure. Cheers.

Kevin Roose: Cheers, Michael.

Michael Barbaro: We'll be right back. Here's what else you need to know today.

Michael Barbaro: A new visual analysis by The Times has found that the cause of the deadly flash flood was a massive landslide that released 7 billion cubic feet of rock and ice. As the debris fell at high speed, it turned into a slurry of water, rock, and sediment that caused catastrophic damage.

Michael Barbaro: And after nearly 300 turbulent days at sea, the aircraft carrier USS Abraham Lincoln has docked for a port visit in Panama City. Crew members have complained for months about supply shortages, water contamination, and plumbing problems during the extended deployment.

Michael Barbaro: Today's episode was produced by Alex Stern, Adrian Hurst, and Eric Krupke, with help from Carlos Prieto. It was edited by Mark George, with help from Michael Benoist. Contains music by Dan Powell and Pat McCusker, and was engineered by Alyssa Moxley. Our theme music is by Wonderly. That's it for The Daily. I'm Michael Barbaro. See tomorrow.