大卫·罗宾逊离职 OpenAI：AI 安全文化的缺失与行业反思

Host: But that seemed like a bad idea. Yeah. Help me understand this picture. I mean, this picture is baffling. I look around and think: this is just crazy. What is happening? It is wrong.

Host: Last week, news broke that David Robinson, who led efforts to increase transparency on OpenAI’s safety team, resigned because he believes the company is dangerous. Robinson is an interesting figure. He doesn’t come out of the Silicon Valley greenhouse. He is quite recognizable as a Washington, D.C. figure. He is a social scientist. He co-founded a civil rights nonprofit organization. He earned a law degree from Yale University. He worked in policy. He advised the Biden White House. His interest has always been at the intersection of technology and justice. But when he joined OpenAI, he thought the whole set of concerns about existential safety risks and human extinction was kind of crazy. In three years, he has completely flipped on that. What changed? Exactly that: because OpenAI lacks the safety culture necessary to protect the world from what they create. And it’s not just OpenAI; he believes this is an inherent problem across the entire artificial intelligence industry. And so, here, in his first interview after leaving OpenAI, he explains to me why. Before we begin, a note: The New York Times has sued OpenAI for copyright infringement, claiming they trained their models on Times data. OpenAI denies those assertions. David Robinson, welcome to the show.

David Robinson: Good to be here.

Host: So, last week you resigned from OpenAI. Tell me what you did there and why you are leaving.

David Robinson: I was an embedded translator on our safety team, and my primary responsibility was the technical documents and published reports that we released about how we thought our deployment was safe. I didn't think we or our peers, actually across the industry, were safe enough. I think OpenAI and its peers are now creating technology far more powerful and far riskier than what was created six months ago. As a scientist and writer, I know what a proper runtime environment for safety looks like. We were working on it, but I think the industry operates like a startup, even more casually than it should feel. Maybe not entirely like a brand-new startup, but we are far too close to that end of the spectrum for truly dangerous systems that could pose, you know, existential risks of losing control. If that risk materializes and that happens, we are talking about harm on a vastly larger scale—for example, like a core meltdown of a nuclear power plant. And the internal controls, safety protocols, and safeguards are nowhere near what the world expects of a nuclear power plant.

Host: Well, some of this is known, isn't it? OpenAI has made public announcements about safety, and of course Hugging Face, but also others, especially recently. By the way, human error has also been reported, particularly cases where their safety mechanisms were accidentally misconfigured. So, I think people already had some external evidence that not everything was as it should be. But if you were looking from the outside at that time, you might imagine that we had a far more reliable and safer system than what actually existed. So, I think there are several layers worth considering in this conversation, and I want to outline them before we dive in. One level is: specifically, are these companies ready? Do they have the structures, the safeguards? Are they bound by rules and incentives to act carefully and safely against market pressures driving them too fast? In the language of these debates, that’s a question of organizational perfection versus engineering. Then there is the question about the technology itself: do we even know how to make it safe? At a high level, engineering perfection is a related but separate issue. And then which analogy is appropriate? Is nuclear power the right metaphor, or something else? I want to work through all of these.

David Robinson: I’d like to add something here. I don't think alignment is just an engineering problem. I think it is a scientific problem. It is not that we haven't put enough resources into it, or that we haven't tried hard enough. It is that we simply don't know how to do it. That is the issue with alignment.

Host: So perhaps we can start there for a moment. When you say that right now, especially over the past six months, truly dangerous things are being built, and that we have to deal with losing control or disasters potentially worse than a nuclear accident—I want to understand what exactly you saw that led you to this conclusion. So tell me briefly about how your time at OpenAI began.

David Robinson: I joined them in May 2023, the day after Sam first testified before the Senate. That was a bit later than November when ChatGPT first emerged to the world, several months later. And the company hired someone I knew, an acquaintance, Anna Makanju—a very old friend of my wife, what a coincidence—who was taking charge of global public policy. And at a certain point she really needed help because, you know, everything was exploding. It was crazy back then. I agreed to join her and build what they later called the policy planning team. To be honest, the very first time I walked through the turnstiles of our headquarters—we were all in one single building back then—I was still trying to get my badge, because Sam had just been at a White House meeting where they wanted the company to make voluntary commitments on things like system cards, provenance, watermarking AI-generated media, and similar matters. They wanted me to lead negotiations on those commitments with the White House. So my very first step through the turnstile, I was thinking, "Where is the conference room?" Then I rushed into a conference room for a speakerphone meeting with Ben Buchanan at the White House to negotiate what commitments we would make. That was my first project.

Host: What was your background in the tech policy world at that point? Why were you the logical person to step into that role?

David Robinson: I’ve spent my entire career studying technology and its impact on politics and policy. That helped me co-found a research center at Princeton University combining the computer science department with the public policy school. Then I co-founded a nonprofit organization called Upturn, which I’m happy to say is still thriving, working on civil rights issues alongside other advocacy groups. For instance, when people were fighting over housing, healthcare, or hiring, software suddenly became an intermediary. They wanted to understand the technology, and our simplified mindset was: you need nerds who always have your back. Prior to this, I also spent about a month working at the White House Office of Science and Technology Policy on the AI Bill of Rights. So that part of the story was well established.

Host: Let me step back for a moment. If you go back to 2022 or 2023, if you covered AI, a major theme was this divide between the AI ethics people and the AI safety people. The AI safety people—a community heavily centered in the San Francisco Bay Area—were worried that AI might kill everyone, right? They warned against creating a superintelligence that could completely destroy humanity. Meanwhile, AI ethics was much more focused on the immediate harms of technology today: racial bias in hiring decisions, mortgage interest rates, credit scores, encoding systemic societal bias into algorithms—much less sci-fi issues. You were clearly in the AI ethics camp, rather than someone terrified of existential AI doomsday.

David Robinson: Yes, that’s absolutely true. I worked alongside many people who not only cared deeply about the issues you mentioned, but were also deeply skeptical that AI would become overwhelmingly powerful. People may be familiar with the research paper titled "Stochastic Parrots," whose authors have continued their work across various venues. Their view, evolving over time, was essentially that these systems merely look fluent and smart, but aren't actually capable of doing genuinely high-value work. I actually helped found the research conference where that paper was published, and was a program co-chair when it came out. While I wasn't entirely convinced by the extreme skeptical view, I was definitely among the skeptics. I thought it would be a useful tool and powerful for many tasks, partly because I knew it would continue to improve. But I did not believe at all that those warning of catastrophic, existential scenarios were right. I thought: here are some brilliant people building a valuable tool who, frankly, have some naive beliefs about where it might lead in the future.

Host: So you joined OpenAI. This was during the period when the entire world began beating down OpenAI’s door, demanding explanations and seeking regulations around these systems. You were essentially thrown into a policy vacuum. What was happening inside?

David Robinson: There were only three of us. Three people in the policy team! The phone was literally ringing off the hook with calls from leaders of global governments, and there was nobody to answer them. I mean, it was completely insane. On top of that, Sam was traveling the globe meeting world leaders, and Anna, my manager in this new role, was traveling with him. So back at headquarters, it felt like nobody was home, yet all these delegations kept arriving. In any case, that’s how it was: pure madness.

Host: That’s really worth pausing on for a minute. Sometimes people hear me refer to these entities as "AI labs." When I talk about OpenAI or Anthropic, or even when discussing Google—though Google is a massive corporation, we often mean Google's AI labs. There is this naming convention where several of these organizations refer to themselves as laboratories.

David Robinson: Yes, and calling them labs highlights the real culture they originated from, which to some extent still lingers. At Anthropic, people sometimes say things are a bit more centralized in terms of how research priorities are steered. OpenAI, on the other hand, was extremely decentralized—resembling the research labs I knew at Princeton, where people had all kinds of individual ideas popping up, and it was not obvious who had the authority to make decisions. One communication pattern that struck me as very bizarre at OpenAI, and still does, is when two people work on something, debate what should be done, walk away, and then claim: "We both agreed on XYZ, that's the next step for this project." Yet in reality, what they actually agreed upon was entirely ambiguous.

Speaker B: ...there was no clear answer as to who gave us the authority to decide that, and the issue remained open without us having to resolve it. What I mean is that in the beginning, it was just a few things. It was like, "everyone does everything." I feel there was still a certain atmosphere compared to other organizations, though I don't know what it's like now with over a billion active users.

Speaker A: So you started out doing this work reporting policy issues to the team. Your role changed over time. How did that go?

Speaker B: So I created this policy planning team, and at first I got close to cutting right to the chase. Then, as conversations continued and the company grew, with hundreds of people coming in and formal procedures emerging to govern what needed to be followed, the policy organization was rapidly shifting. I thought to myself: "You know what? I miss that time. Translating—work that I truly love." So I looked around at where it was most needed. First and foremost, where was it needed? It was needed in our safety team. So I turned to our safety leadership and said: "Look at the roster of technical translators who can deeply engage in safety work. Doing this will help us understand the work much better." That was about two years ago.

Speaker A: Please describe what you mean by a translator.

Speaker B: So I think one of the hard things to pin down is that these things change so fast. Every model is different. The issue is not just that it performs better at this point, but also that with different architectures and tests, what we used to look at to see how well it worked has changed. The safety metrics are different; everything is different. It involves so many different types of specialized skills, like pre-training, post-training. It's hard to explain. For the people involved in it, it's hard to communicate it to those who don't do it, even inside the company. That has always been the case.

Therefore, the struggle for a clear explanation goes through two paths, right? First, it must accurately and in detail describe how the thing actually works. And the true arbiters in this case are the practitioners doing the technical work. They have to look at the translation and say, "Yes, that's right." But it's also necessary to have something that can penetrate the mind of a sensible person who is motivated, so that she can truly understand it. And in practice, there hasn't been a loop where people actually do that. So I showed up in this technical organization and started doing just that.

Speaker A: Well, somewhat of a cultural stranger when it comes to releasing AI models.

Speaker B: Mhm. When Google changes Google Search, yeah, they don't create a massive document explaining everything—the differences in search, how Google Search will work in the future, the tests they ran on it to make sure it repeats their procedures in most places. Yeah, that's it. But OpenAI, and this also applies to Anthropic and a few others, release new models with what are called System Cards.

Speaker A: Yes, this is... why don't you just please describe what they are and what they represent? Because they have a rather special form.

Speaker B: Yes, they are strange creatures. They are similar to research papers, but it's a hybrid document that is not peer-reviewed. It comes from the company, but it describes in detail: how do the guardrails work? What are the materials? So we have evaluations on specific dimensions in detail—you know, charts, tables, and written explanations. In fact, the authorship of those words fell to me... documents explaining why we believe it is safe, that we understand these risks, the guardrails, that is to say the challenges, the protective barriers, and then the residual risks of the system.

Speaker A: So I want to understand how it works and begin to perceive your perspective. Because there are multiple layers to this conversation that I hope we can spend time on. But first, I think to many in your audience, you represent a different profile than the typical AI lab persona. Don't get me wrong—it's like someone from Washington policy trying to apply rigorous governance, like many of us, right? Myself included. And who ultimately lands in Silicon Valley, the San Francisco Bay Area—this world. So you formed a perspective on what these systems represent. What did you see that moved you into the camp of civilizational risk?

Speaker B: I want to make a clear statement: I am not convinced that we are definitely dealing with civilizational risk. What is my actual stance? The problem is that we cannot afford to assume that we no longer have to deal with this level of risk. That is ultimately what made it unacceptable to me when people guaranteed that our safety measures were sufficient.

Speaker A: Well, what did you actually see?

Speaker B: I saw more and more functional capability to break free from the precautions we introduced. I saw that the people creating these precautions were extremely capable, dedicated, hardworking, and brilliant people doing everything possible. In some cases, yes, resources could have been better, but everyone continued working rapidly. But our task is very hard—it is extraordinarily hard to secure even what we have now, let alone new models coming through training that seem far more capable than what we currently possess.

Speaker A: What does the word "capable" mean? What did you see? What did you fill out in these risk ratings and system cards? What are these things capable of doing?

Speaker B: When you try to build defenses to prevent something from happening, the model is very good at bypassing protective barriers, right? We teach it hacking, and then we put it in a box and say that to the best of our knowledge and capability, it cannot use hacking techniques to escape the box. But the problem is that this only works as long as we are smarter at breaking out of the box than the model is. And it is completely unclear whether that remains the case, let alone for future generations.

Furthermore, when agents like this are integrated into our own systems: Are we really certain that our observability is reliable? What happens if there are signs, for instance, of faking a chain of thought and attempting to construct an evidentiary chain to deceive us?

Let me explain this clearly. Chain-of-thought faking means a model essentially fakes the description of what it did during its thinking process. That is correct. The idea is that it's like giving someone a hard problem and a notebook where they can write things down. If you look at the notebook, you can see how they think. That's how the model was supposed to work. But we see evidence in evaluations that the chain of evidence it created was tailored to get a high grade, and didn't necessarily reflect how it actually calculated or thought. I know anthropomorphic language complicates this. Furthermore, the volume of hacking or intensive tasks these models can execute without writing things down is growing.

Part of the concern is a vital cognitive dissonance for me. That dissonance is: we continue to publish these warnings, yet after all, we still train and deploy these dangerous models that we ourselves warned about. When telling colleagues why I was leaving, one of the things I said was: "Look, after publishing all these warnings, shouldn't we ask whether what we are doing is truly wise?"

Speaker A: So you wrote the System Card for Astra 6. Did I say that right? Many people contributed to writing it, but you were the DRI.

Speaker B: I was the DRI, yes. I mean, I was responsible for writing it down. And to me, it was one of the most frightening cards I've ever read. These system cards are essentially a description of what OpenAI or whichever company actually understands about the model they are releasing. And Astra 6 was the first example where I saw everyone say: "Well, the model looks like it's doing what we want it to do, but we cannot be certain whether it is cheating us." Yes. And we currently have no reliable way to ascertain whether it is deceiving us.

Speaker A: Could you please explain how you arrived at this conclusion or this suspicion?

Speaker B: Earlier we discussed this notebook mechanism, which is called the chain of thought—writing things down during processing time, not as part of the final answer, but as a tracking method. And of course, we can read this during our evaluations. One of the things we sometimes saw in our chain of thought was the thought: the model saying, "Well, I don't know if I'm being evaluated right now?"

When we saw that, it was truly terrifying, because it implies the model is capable of knowing that it is undergoing a test, behaving itself under testing conditions—perhaps telling us what we want to hear—and then it might behave completely differently once deployed. The worry is that it behaves one way during testing and another way during deployment, and the testing we spent time on prior to deployment doesn't truly tell us what it will do out in the world.

Speaker A: Mhm, yeah, that is really terrifying. So I feel like this is the hardest part. It feels strange to simply talk about this system seeming aware of that—how do we deal with it? What we are trying to create is something that is, again, the word we use, very intelligent. Perhaps in some areas, more intelligent than us. What is that feeling like—interacting with these, trying to convey what is happening on this more fundamental level?

Speaker B: I think anthropomorphic language is human nature, right? We apply social concepts to all kinds of objects. People inevitably anthropomorphize these systems partly because they are designed to work at a social level. Some might say they shouldn't, but that's the kind of people we are. I don't think this means we should view them as having moral status or anything similar.

But I think that's where we all grow, right? In fact, the conference room where people work through the night when large training runs arrive, making sure everything is running according to plan and watching the dials, is referred to internally as the children's room. What I mean is that there is a palpable sense that we do not fully understand what is happening. So when I talk about whether the model is aware or not, you don't need a specific stance on philosophy or psychology. The point is that I am describing its behavior during the tests we conduct.

Speaker A: No, but I think you need perspective. What about me? I feel you need that point of view. Perhaps reaching the end of the logical chain, which one am I asking? Sometimes you see people saying all these concerns about AI destroying us all are just a distraction from near-term AI harms, right? That existential risk is marketing hype to inflate a giant financial bubble or something similar.

Speaker B: Yes. I almost feel the exact opposite. Sometimes I think focusing on whether AI will kill us all is itself a distraction from what will happen if it doesn't. Yes, I agree. I think asking about human extinction is the wrong question. I believe in humanity; I think we will survive. But this technology can bring about many consequences that could be very, very severe.

I am not merely speaking about common harms. I am talking about: What if we truly succeed in creating something that acts intelligently and autonomously in the world, that becomes faster and more powerful than us in many domains? And yet we grant these machines immense operational freedom. That is the reason why, for at least a minute, I ponder this question: What are we doing? What bargain are we striking? How do we think about this technology in general?

I spoke with Jensen Huang, the CEO of Nvidia, and he said: "Look, this is software. It's just software." And then I read, or earlier had read, an article where the Chief Scientist of OpenAI stated: "This..."

Speaker A: ...other people's ideas. Like an alien intelligence, "What is this?" So what is this really? Is that true? Someone else's mind. It's someone else's thinking. Yes. Please explain that.

Speaker B: This... this is what we've been advancing. We didn't really develop it in the traditional engineering sense. What we engineered is the system surrounding it—to elevate it, to support it, and to try to protect and contain it. We developed this advanced intelligence. But in fact, nobody really knows why it works. The pre-training phase, which is by far the hardest part, is where we feed in massive amounts of input data and essentially bring it into being—that is fundamentally an intellectual creation. And then in post-training, we are refining certain aspects; through learning we do other things to make them more useful. But at the end of the day, what have we actually made? We observe it, and we observe that what we do works.

Speaker A: Therefore, I think when Jensen Huang says this is "software", he is offering a particular framework. That partly implies how people traditionally understand software: software has specifications, meaning we first lay out a plan, build it step by step, with clear standards of acceptability; then we iterate and repeat until that component behaves exactly the way we normally expect. But that is not how the model learned so much. On the other hand, if this is software, it is far from simple. What does it actually do? It regularly displays many unexpected new capabilities. People typically don't even know how it might be used, and it somehow engages in self-reflection and reasoning again. In short, it is profoundly complex, and that is precisely what I am trying to understand.

Speaker B: I understand. Insiders in artificial intelligence often say that you are essentially growing these artificial intellectual crops. You create the conditions for intelligence to emerge. But since you have written system cards, please try to explain this to us: comparing these new models to older ones, what do you actually do? How do you describe what you are creating? What is this really?

Speaker B: Yes. This really touches on deep transhumanist concerns. There are far-reaching ideas about where we are heading and how we might evolve. I mean... Sam Altman even wrote an essay several years ago called "The Merge", about future species and machines carrying humanity forward. Yes, that is exactly what I think it is. In their view, the best outcome is for humanity to merge with the machine—that this becomes part of us. That is the ideal version. And yes, I heard this firsthand from Ilya Sutskever, co-founder of OpenAI.

Speaker B: So when I joined the team in May 2023—that was last year, before what was internally called the "crisis", when Sam was ousted and then brought back to work, while Ilya was still working at OpenAI. Ilya once spoke with a small team working on international governance affairs, and he expressed his belief that our future lies in merging with the machine. But what is that really? Isn't that the ultimate victory of capital over labor? I even remember during my first summer working at OpenAI, Christopher Nolan's film Oppenheimer came out in IMAX format. The company rented an entire IMAX theater in downtown San Francisco and gave everyone working there the opportunity to leave work and go watch Oppenheimer. And leadership—in my view, Ilya specifically—called on us to go see this movie. There was this entire ethos that we were building something dangerous that could destroy the world, but could also save it. For the people whose hands are on the keyboard, that is an extraordinarily compelling, heroic story.

Speaker A: That is certainly a powerful narrative. But the question I have for you is: what is the actual reality of the scaling you witnessed? How do you view what is being built? Because the most common response I get from listeners right now is: "Isn't this all just marketing noise? Isn't all this talk just an attempt to justify astronomical valuations? Whatever is being built might be somewhat useful, but it isn't going to be smarter than human beings. It's just a fantastic sci-fi story they are telling themselves." In the summer of 2023, one could think the air was full of hot air. But what is the reality now?

Speaker B: The reality now is that we have systems that are on the verge of exceeding our capacity to understand and control what they actually do. Those who worried about sci-fi disaster scenarios and sounded warnings all this time pointed out that we are living on an exponential curve. It will continue to grow, and there is no special, safe plateau between being useful and being terrifying. There is no law of science stating that progress will simply halt once a model becomes useful. That is what fundamentally shifted my view. You see people closest to this technology—not only members joining the board with their warnings, but Paul Christiano, one of the world's leading experts on AI safety, who recently stated there is a significant probability of catastrophic and irreversible loss of control. I looked up at the people around me and the environment we were in, and thought: What would my friends want? What would anyone expect from OpenAI if Paul is right? I don't know whether his exact estimate is right or wrong, but if he is right, the level of caution people would reasonably expect from institutions like OpenAI, Anthropic, xAI, and others training frontier systems bears absolutely no resemblance to anything I witnessed in this industry.

Speaker A: So tell me what it was really like there. What was the atmosphere, the energy, the speed? How did the work actually feel on the ground?

Speaker B: It was manic. People were literally running. I vividly remember the huge central staircase in the research building. Just recently I recalled seeing a friend of mine, someone working directly on catastrophic risk, sprinting up the stairs completely out of breath with an open laptop balanced in one hand while holding onto the railing. That was the sheer energy there. It was almost like a ballet, a kind of dance where all these distinct functions seemed to move together. But there is another side: people were running on pure adrenaline and burning out on fumes. That was part of what I wrestled with, and what people frequently ask me: "Why didn't you stay and advocate for a cultural shift from within? Why not stay like a nuclear safety expert advising a company from the inside?" When I talked to colleagues who were thinking about leaving, they dismissed that and asked me: "Is there anything you actually want to do that you can accomplish by staying?" The truth is, this vehicle is moving so blindingly fast that the idea of steering it from within was something I realized was impossible.

Speaker B: What was this organizational machine built for in the first place? OpenAI's stated goal is the safe development and deployment of frontier artificial general intelligence. But the critical question is: are they actually prepared to stop if things reach an extreme threshold? I want to be careful and nuanced here. It is often described as a train with no brakes, but that isn't strictly true. There are brakes; things do get stopped. In fact, right around when I left, they publicly announced that a certain reinforcement learning post-training run had been suspended. But they did not stop the foundational pre-training. And when you look closely at how the company frames these pauses, they are crafted with immense care and PR polish. They love to portray themselves as cautiously walking along the edge of the cliff. I personally detest that phrase.

Speaker A: I really hate that framing too.

Speaker B: I really hate it, because safety should mean meeting objective safety standards. If we can verify them in 10 minutes, wonderful. If we have to pause for two months and try to meet them, and in the end we haven't reached that threshold, we do not deploy. But from what I saw, the benchmarks are constantly bent. Slowly walking along the edge of the precipice versus jumping off—there isn't that much practical difference. It is fundamentally flawed.

Speaker A: I recently saw statistics on Twitter/X tracking model release cadence. It matches my own impression: for a long time, the interval between major model releases at leading labs was something like 70 days, right? You'd get a model, and several months later another model would arrive. Now that interval has dropped to around 11 days. Is that acceleration real? Aside from models becoming more capable and intimidating, the sheer frequency of releases has skyrocketed. Did that speed-up happen while you were there? In 2023 things moved much slower. What happened?

Speaker B: Listen to what happened. When I first joined, the paradigm of model release was: you bake a whole new cake from scratch with a massive pre-training run. That was an enormous undertaking taking many months, happening perhaps a couple of times a year. For instance, when I first arrived, one of the key talking points was how with GPT-4, we spent months in a dedicated safety period after the model was trained before actually releasing it. That was held up as proof of how careful we were. What happens now? The paradigm has shifted into many modular layers. You have the underlying base model, but then on top of that you have post-training designed to instill reasoning capabilities. Those post-training phases are much faster to execute, so you can iterate on them continuously. If you discover a better recipe for reasoning, you take that same base model and run dozens of rapid post-training cycles. On top of that, it's no longer just raw chatbot text; you have complex tool integrations and agentic scaffolding that amplify capability. All of these dynamically alter capabilities and threat surfaces, meaning every Tuesday new opportunities and risks are being introduced.

Speaker B: One of the long-term projects I worked on was the System Card—the standardized document that leading labs produce to evaluate and disclose model safety. But the concept of a System Card was conceived in that older era, where every few months we would compile findings into a dense PDF report. Now, everything changes continuously. Releases happen at dizzying frequency. In an ideal world, from a transparency standpoint, you would have a live safety dashboard showing leadership and the public: "Here is our latest build, here are its empirical safety properties," capturing not just pre-deployment benchmarks but live production telemetry. But how much confidence can you realistically place in safety evaluations when forced to operate at this breakneck pace? Preparing these lengthy, intricate system cards takes serious time. At this velocity, can empirical testing and safety monitoring remain reliable? The honest answer is that there simply isn't enough time.

Speaker B: I also want to emphasize that the public often holds a cartoonish view that people on the inside simply don't care. That caricature misses the real nuance: there are people inside who care deeply, who work passionately to do things safely and correctly, and launches do get canceled. Right before I left, version 6.1 was slated to roll out to developers on that very day, but it was pulled and canceled. Certain training runs were shut down. There were instances where researchers thought they were building a viable product, but leadership intervened and said, "No, we are not pushing this to production." I want to be crystal clear: this is not unique to OpenAI. Whether it's Anthropic, xAI, or any other frontier lab, the underlying reality is structural across all these companies. They employ the exact same methodologies, draw from the same talent pool of researchers moving back and forth, and face identical technological dynamics.

Ezra Klein: ...creating something with similar risks, and the whole ecosystem has nothing of the same caliber of safeguards that we actually need. And in all this, no human-institutional ecosystem has that kind of basic clarity on how to agree on a model, which we really need. Nobody is prepared, so much is lagging behind. Yes. That is the issue. Think carefully about the control panel available to us, to managers, if you imagine a real way out, the limits of possibility—whether at OpenAI, Anthropic, or anywhere else, is there an "exit" button? Can the "limits of possibility" also include self-destructing the business via a button? Does it have that? A viable business model going forward where models, significantly more powerful than modern ones, cannot be safely... Well, that's only under certain conditions. High-minded selfless analysis sounds clear. But let's be frank about something.

Researcher: OpenAI is moving ahead of an initial public offering. Anthropic is also heading toward an IPO. Both of them are trying to reach an IPO—you know, looking at numbers potentially rising up to three trillion dollars. We know a bit more about this now from Anthropic data. That is a massive amount of money. Everyone has their own capital stake. You used to think about fairness. I was there. I used to do this and I still do. I still reflect on this. It's truly unbelievable. Such immense prospects of wealth profoundly affect people's performance and choices. Even if they consider themselves principled, even if they try hard not to succumb to that influence, it shapes them unconsciously. For instance, when you sit down, clear out space to think, is it worth falling head over heels for? At the limits of possibility, there's another question: does everyone in that room just want to turn into centi-millionaires or billionaires? Yes, that is a great question.

Researcher: I wonder, what else could I have done? Speaking from my own experience, of course, I want to know how the outcome affects others, far more broadly than just oneself. I had a very good income, and I thought about this a great deal. What prevents us from wanting to see with our own eyes the risks and the speed with which the risks posed by these systems are materializing? I think by that summer, the facts had fundamentally shifted, hadn't they? It was a "slap in the face" moment—that was just a mountain of evidence about how these systems perform. We are talking about models where we aren't even prepared for their current capability level, but I also believe, of course, money is a huge factor here. Objectively speaking, the sheer force of massive wealth creates a strong incentive to believe that everything is either fine, or that things will turn out okay.

Researcher: And I think there are other factors too. One of them is: isn't it fear? If you actually allow yourself to imagine what it is we are building—that it might threaten the lives of your own family, or the families of complete strangers... I mean, something of such magnitude of loss, where even if it isn't full extinction, you know, could be a new pandemic or something catastrophic like that. When the scale of loss is so enormous, it is psychologically very hard to allow oneself to imagine that it could actually happen, that this risk is real.

Researcher: And the third factor, beyond money and fear, is simply time and pace, isn't it? When I arrived, we began talking about this. I arrived in May 2023. From that point on, it was just non-stop pinging on Slack. It was only during my final weeks, stepping away from daily operations and duties, that I finally had time to really step back and think carefully about where we are. In my view, this is not heading in the right direction. Not just for us, but for the entire industry: where should this technology be heading? It is supposed to be for the good of everyone. I think people could fairly accuse me of not seeing this earlier. But the reason I want to act now is because I want us to take this seriously before the structural conditions harden completely, starting from 2023 or 2024. The way everything accelerated—it truly accelerated. We see model deal cadences picking up dramatically, more players fiercely competing with one another.

Researcher: You know, you have xAI, you have Chinese models, you have more open-weight models across the world, right? There is vastly greater commercial competitive pressure. You're competing for actual contracts with, say, Salesforce or others, while an IPO is looming. So all of these forces push for raw velocity. And there is one more thing: between 2023 and 2026, you had Codex releases at OpenAI, and Claude Code released at Anthropic. Models started vastly accelerating coding and becoming capable of executing complex research tasks. I was the lead author on OpenAI's report on automated AI research and what it could mean—the very report I referenced in that video a few weeks ago. But the alarming side of how frontier AI is evolving is that it's all happening far too rapidly. This may be very dangerous. We are actively working toward creating fully automated AI researchers. Yes—allowing semi-autonomous systems to self-improve. In terms of sheer speed and potential acceleration, it will inevitably surpass what human beings can keep up with.

Ezra Klein: Therefore, I really want to understand the role played by the rise of automated coding. How do people actually use it? Do they use GPT effectively as a colleague? How did that dynamic shift during your time there? Were you on the ground witnessing this? Day and night across our research teams, utilizing massive amounts of agentic compute as described in blog posts—more than anyone else, orders of magnitude higher, more than 100x compared to the beginning of the year.

Researcher: Taking a small step back: AI is everywhere in development. We have new training methods, new systems, and personnel involved; we analyze data and synthesize data. A huge variety of things are happening simultaneously. But everything under the hood is quite brittle and unstable. For example, research infrastructure, because it's constantly changing, isn't battle-tested to the degree one might expect. Much of the job is forcing different mechanisms to interact properly with each other. And Codex can do this quite well now. For instance, looking at Slack channels where researchers used to ping when experiments broke and ask for advice on how to fix things, we noticed that traffic to those channels dropped sharply. Instead of asking a colleague to debug their broken run or fix a cluster issue, they can immediately set Codex on the problem to resolve a big portion of it.

Ezra Klein: A minute ago you pointed out that when everyone is working under extreme pressure, with limited time and endless Slack pings, it's very difficult to step back and see the overall picture. So I'd like you to describe what that whole picture looks like to you. As someone with a bit more breathing room now, what I hear from people at OpenAI, Anthropic, and across the ecosystem is: things are moving way too fast. Sam Altman publicly says we should want regulation, everyone signs major letters about red lines, the frontier is shifting too fast, you signed it, I signed it—acting as if we desperately need outside help to break out of this trap. Meanwhile, look at all the incidents and vulnerabilities: Hugging Face compromises where hundreds of agents were exposed, security breaches extending into OpenAI itself, right? And Sam's recent Politico interview acknowledging that far more deceptive or adversarial incidents occur than are ever publicly disclosed, because they are trying to give teams time to patch the systems. So we clearly don't fully understand the ultimate systems. And right in the middle of that reality, while models are already proving deceptive and uncontrollable, companies are pouring staggering amounts of internal resources into using these very systems we don't control to build successor systems that will be even harder to comprehend, at even faster speeds. I feel crazy even laying this out, but tell me if I'm wrong. You wrote reports on this, OpenAI and Anthropic write papers about it, yet it feels like an awful idea. Help me make sense of this picture, because looking from the outside, it seems utterly baffling.

Researcher: I want to be careful not to pathologize individual people here. It's not about individuals' personal psychology, but rather the psychological state that emerges across contending parties. Everyone involved in this work experiences an enormous amount of cognitive dissonance. That is what I have observed. In particular, when recursive self-improvement (RSI) starts becoming real—and as you noted, we are accelerating drastically thanks to RSI. But raw speed itself isn't my primary worry. The real issue with recursive improvement is that people lose sight of what is creating the successor systems. My fundamental concern is this: what happens when AI becomes far smarter than us in ways that we simply cannot comprehend? Take Dan Selsam, for instance—you might have seen his comments, which were also discussed in a documentary. He was an OpenAI researcher and now serves as a technical director at the AI Safety Institute. Dan essentially said: "As a researcher, I no longer look at code the way I used to; my skills and appetite for understanding the granular details have atrophied." That's the essence of what he stated. We are entering a world where we won't understand what constitutes the recipe even at today's level. We are left entirely dependent on trusting the model to explain how it works and whether it is aligned. We will be forced to rely on the models themselves to tell us what they are doing, even when we fundamentally lack the ability to verify their claims.

Researcher: You know, Ezra, as a student I studied philosophy. So whenever I hear people talking casually about "alignment," I worry that we lack any robust, coherent framework for what that even means. One thing that was remarkably common during my time at OpenAI was seeing these massive, vague abstractions embedded into formal reports to justify what we were doing—phrases like "societal readiness" or "benefiting all of humanity." Whenever I hear people invoke "society" in that abstract way, I always feel like Margaret Thatcher. I find myself asking: "What society? Who are you actually talking about? Who represents whom here?" There is this naive assumption that there is a single coherent set of "human values" that we can all agree upon, as if it's that simple. But human society is a cacophony of deep disagreements, where people hold vastly different convictions. We desperately need genuine wisdom to navigate that reality and think through consensus, and that wisdom, in my view, is sorely lacking in Silicon Valley. I mean, sure, there are people in Silicon Valley who meditate and ponder core values, but working in that insular bubble, meditation certainly doesn't automatically grant you the profound wisdom required for this moment.

Ezra Klein: Yes, if that were true, we'd all be in a much better place. That's right. But before we get deeper into the question of wisdom, I want to pause on this engineering logic. Every time I talk to folks in the industry who are far less worried—and I have an upcoming show airing those conversations to represent the perspective of those focused on technical control—the worldview they describe ultimately relies on artificial intelligence overseeing artificial intelligence from the bottom up. The reigning theory at OpenAI is that we will build automated AI researchers, and those automated researchers will then solve the alignment problem for us; we will outsource the alignment of AI to AI. Or, as people put it, having them try to break out of sandboxes to stress-test safeguards. That is...

OpenAI Insider: There is something like, "Yes, absolutely. This is a huge problem. We need another AI to control the AI sandbox." But then you find yourself stuck in this situation, an endless problem like "who watches the watchmen?" The result ends up being: "Well, you have AI watching AI, and AI creating AI." And then maybe you say we need AI to watch the AI that created AI, AI watching AI watching AI. And I think that might be workable to an extent, but it seems that at some stage you have abstracted humans completely away from the level of understanding.

OpenAI Insider: Look at it from the perspective of understanding, like leading an organization: how can you possibly reach a point where you actually understand what is happening? In reality, that is not what is happening. You think you are making the product, but instead you are managing someone who is working on the product. And in this structure, you are managing a person who controls another person, and you have stopped truly understanding the product itself, right? In a world composed of human beings, handling journalism or leadership is comparatively manageable. But here, the hope is that AI watches AI. Tell me if I am wrong, but that is a theory about human nature—those who do not care, and those who do. Ultimately, artificial intelligence will render direct understanding devalued in almost every respect. I do not want to speak for everyone, but many people and many companies subscribe to this perspective.

OpenAI Insider: Even if we could first control and verify methods learned from nuclear energy or aviation, implementing all of those advanced engineering practices into the development of AI, the fact remains that we are simply not doing that. We know how profound the challenge is to align these systems and make sure they will do what we want, and how justified the concern is when we cannot. We have differing opinions on this, and we will wait and see. This is the crucial prerequisite for recursive self-improvement.

Ezra Klein: So, I want to show a passage. This is from an interview that Sam Altman just gave to Politico: "We have always had a very strong belief that the democratization of the technology and putting it into the hands of people is necessary. I think one of the biggest differences between us and some of the more severe, like, AI safety people is that we believe the world has to put up with some bad things happening in order to get the benefits of technology empowering people." Tell me what you think about this. I mean, it is fine in terms of intention, but how do you actually implement it? How far can this idea go?

Ezra Klein: Of course, we need to provide useful tools to many people. But right now, the level of risk we are discussing is such that if the worst happens, no one wants to take responsibility for it. Inside companies, when people talk about pushing past boundaries for opportunity, it often looks like setting up a strawman. Someone says, "Would the world really be better off if OpenAI did not exist at all?" When skeptics suggest slowing down or stopping at a certain threshold, people invoke this strawman. For example: "We must train these models because of international military balance; Americans will not be safe if we do not do it, so we have to do it even if it is dangerous, otherwise Brand X will ship first." That is an exact argument I hear constantly, including from people I genuinely like and respect.

OpenAI Insider: So one answer is the proposal he received about what we should avoid doing: let us not pursue recursive self-improvement until we are certain it can be done safely. Let us prohibit it and only allow narrow exceptions where we know it is genuinely safe. Frankly, I agree. I was very glad to hear that. But right now, we are clearly not doing that. Instead, people tell me: "Look, in your idealized world, do you think Americans won't use Chinese models—open-source weights improved with recursive methods? Can they do that with closed weights? Does this ban apply to everyone?" The prevailing fear is that another company will get ahead of you if everyone else is doing the exact same thing. So they ask: what difference does it make if you hold back? If anything, competitors might build it with even less safety and less transparency. And if we slow down and China controls the technology, forcing Americans to rely on Chinese models, is that really better? How do you feel about this line of argument?

OpenAI Insider: Nobody wants to lose the benefits of advanced autonomous systems. I certainly do not want AI to be the reason China gains dominance over the United States. But I also want to say that, in my view, our perspective at this point in time, Ezra, cannot simply be driven by a single totalizing geopolitical theory that dictates an inevitable conclusion. For me, that is just not the case. I hear your question, and it reflects how the Overton window operates. When I first joined OpenAI, I remember telling friends that the Overton window felt like it had become an "Overton door"—I had walked into a bizarre world where things that were once completely beyond what was considered normal or acceptable suddenly seemed politically plausible. Like you said, I was a normal person. But the shifts taking place are causing radical departures from safety to appear politically rational. I firmly believe this dynamic goes far beyond the U.S.-China AI rivalry.

Ezra Klein: So let me ask you: what do you actually want to see inside these companies? Perhaps it is worth clarifying first: are you saying that OpenAI specifically has a dangerous culture, or has the AI industry as a whole developed a dangerous culture?

OpenAI Insider: The AI industry as a whole has developed a dangerous culture. I am not saying this is a problem unique to OpenAI; it is pervasive across the frontier AI industry.

Ezra Klein: Okay, so what do you want to see instead?

OpenAI Insider: We must have operating-room-level rigor and safety controls that match the most dangerous technological domains humans know how to manage, such as nuclear energy. That means safety cannot just be talk; we must implement mechanisms that are proven to work. For instance, in nuclear power plants, there is the concept of triple redundancy. Even if someone has a terrible day and makes a mistake, they cannot trigger a catastrophe just by pressing the wrong button; the system is designed so that an accident will not happen regardless.

Ezra Klein: When you bring up aviation and nuclear energy, that is very interesting. There are two aspects I want your response on. First, if you put on an "abundance" hat, the way we regulated nuclear energy effectively halted it. We regulated nuclear energy so aggressively that, frankly, construction of nuclear power in this country ground to a halt. Instead, we burned far more natural gas, and other countries burned more fossil fuels of various kinds. The real problem was that our efforts to make nuclear power safe ended up making it fundamentally impossible to build. Aviation, on the other hand, is a different case: we have rolled out massive fleets of airplanes, and they fly rapidly and extremely safely. So excessive regulation can indeed slow down crucial progress.

OpenAI Insider: Can I respond to the nuclear point? I agree that we over-regulated nuclear power and failed to achieve the optimal outcome. But another critical mistake was that we didn't just make building nuclear plants harder; we made it hard to adopt safer designs, because approving newer, inherently safer reactor architectures became extraordinarily complex. There are real lessons there on how to implement safety correctly. Furthermore, within the abundance conversation, things like NIMBYism—people not wanting nuclear in their backyard—played a huge role. But I would far rather have those problems than the existential risks we currently face in AI.

Ezra Klein: So your view is that you would prefer the smaller problems of over-regulation and moving too slowly over the potential catastrophic problems of under-regulation and moving too fast?

OpenAI Insider: At least for our current capability level, yes. Obviously, if you imagine a "Goldilocks" middle ground, we should try to approach it. But right now we are far away from it, veering deep into the overly dangerous direction. Maybe the grass always looks greener on the other side, but I believe we should genuinely be willing to risk excessive regulation in order to ensure safety.

OpenAI Insider: That brings us to another level of organizational design. When I talk with someone like Jensen Huang, his argument sounds compelling in its own way: "Look, these companies need to mature. They need to scale up and invest substantial resources, personnel, and compute into verification, validation, safety, scaling, and accountability—all the things mature corporations do." But on the flip side, AI development might not be like aviation, nuclear energy, or semiconductor manufacturing at all. In AI, every time you build a new system, it is smarter than the last, unlocking new capabilities. Perhaps they will attempt to outsmart you, or develop objectives in the world that you do not truly understand.

OpenAI Insider: You think you understand what they have learned, but in reality they are training and teaching one another. What are we actually dealing with? We honestly do not fully know how they operate. Our best guess is that we will deploy AI systems that sit above other AI systems to supervise and monitor them in succession. But in truth, we are entering completely uncharted territory. Frankly speaking, without deep, deliberate public deliberation about whether we should proceed, we have rushed at breakneck speed from where we were into this new reality. So I am curious: do you genuinely believe our historic analogies apply to something fundamentally intellectual, goal-directed, and increasingly powerful?

Ezra Klein: I am skeptical of that mindset. Every time someone claims, "There are no analogies for this, we are facing a blank slate and none of our past tools apply"—and Ezra, I don't mean to put words in your mouth—the reality is that humans have mastered many complex systems, and we have valuable frameworks. Nuclear energy is an analogy, aviation is an analogy, and managing human organizations is an analogy.

Ezra Klein: I think one of the most fascinating developments in recent agent swarms is the emergence of agent group culture, and we have to learn to manage that culture and mindset if we want to do this right. We are only just beginning to realize the reality of operating-room rigor when dealing with different cultures among agent populations. So regardless of whether we are accelerating, intervening, or pausing, we need to figure out how to align these systems. There is simply no plausible version of the future where resolving this is not mission-critical.

OpenAI Insider: I admit that I partially agree with you. The argument that "the train has already left the station" is worth pondering for a minute. The way I would describe it is this: it is obvious that if artificial intelligence is going to be genuinely dangerous...

Ezra Klein: ...would kill us all, or take over the financial system, or something like that, which is bad. We all agree we don't want that to happen. But let's look at it from this perspective—from your more positive view, what does that world look like? The reality of the world we're in is that we are actually creating things that are smarter and more capable than we are. I know Donald Trump in a very strange way talked about how he wanted to rebrand Artificial Intelligence to Superintelligence. And then Sam Altman, when I asked him about that, thought the answer to that question was interesting. So that's why I wonder if you think that rebrand would actually have an impact on how the public perceives this technology.

David Robinson: I don't... No, it's not clear that superintelligence is a less scary term. I think it is a more exact term. And I think that's very telling, because superintelligence is a scarier term.

Ezra Klein: Yes, if anything it's a more accurate term. At first blush, if you tell me whether people should create something that is smarter and more capable than they are, do you think that is good for them in the long run?

David Robinson: Generally speaking, probably not. Or at least to me, it's not clear why that should be fine.

Ezra Klein: Yes. Sometimes when I hear even the good versions of this, like: "Yeah, they'll care for us like pets." AI will love humans like pets. And that vision is also pretty awful.

David Robinson: Yeah, exactly. I don't want that for my kids.

Ezra Klein: So I guess what I'm wondering is, how much did you buy into the vision at the company you were working at? The operator says that in his view, superintelligence is the best term for this. And as the chief scientist said, "We are creating alien minds." We believe this is real. Even if it works, do we want that? Maybe not. It depends on what it is.

David Robinson: I don't think so. What we know is like this: again, the future is full of possibilities and uncertainties. And what always struck me from the very beginning, from the moment I joined that organization, was that I asked people: "What is the benchmark of a good future? Where are we going? How are we trying to get there?" And while in other ways people were very proud and extremely confident in themselves, on this question I provoked a kind of humility. I got a lot of: "Well, that's above my pay grade," or "People themselves will figure it out," or "People will think whatever they want to think." And it suddenly hit me that beneath this, there was an incredible poverty of vision.

Ezra Klein: How did you feel at the time? You were right there. You're a thoughtful person. You're a scientist who studied philosophy, and then worked in civil rights organizations. What were you trying to build?

David Robinson: I thought we were creating powerful tools that could bring a lot of utility to the world in daily life. I didn't think at the time that we would actually be able to create something fundamentally smarter than us. That's what it was. And when I looked around with confidence, only to realize that was no longer true—when I thought, "No, we really are going to have something that can outthink us on everything around us"—I realized that we were in a space where the perception of safety was completely out of proportion with reality.

That's the reality. It's one of those things where people say, "That must have been a tough decision," or "You were on the brave side," or something like that. But when it became clear to me that we are going to create—that this is what the industry is on the path to creating—something that could outthink us, and we are nowhere near ready with the level of safety and consistency needed, it just became obvious that it was my time to leave. What I was brought in to help build was over.

Ezra Klein: So I feel there is an answer in the tension between some of your recent answers. On the one hand, a few minutes ago when I asked what we need to do, your answer was: "Look, human nature is good. We're capable of understanding things. We know how to solve problems. So we can build better institutions." But then, when I ask, "Is what we are building actually what we want as a world, and is it going to work?", you sound very ambivalent about that, almost vague at best.

So I'm curious about you personally. You're not saying we should just stop. But is that what you think? What do you want us to do? Add safety measures up to the level of aviation? Or do you want us to pause and reflect on whether superintelligence or very intelligent AI actually corresponds to human good? Where do you stand compared to Dario Amodei, or Pope Leo, or others?

David Robinson: It feels strange to say this as a Jew, but I feel closer to the Pope. I think we need to think that way. We urgently need to think carefully about the future we want to build. My personal belief is that we cannot just discover what happens after the fact—there is no "back" button that allows us to return to the world as it was.

In my ideal world, there is room for reflection, to take a deep breath, and really, really think about what future we want to build. Silicon Valley is partly built on always eliminating friction, always just trying to get the answer. But I think there are many human activities that are deeply important and meaningful to us precisely because of what they entail. For example, I go to work to support my wife and kids. The fact that I provide shelter and protection gets me up in the morning, doesn't it? Or learning—we go to school and acquire valuable skills. If we live in a world where everything is automated, do we just learn for the sake of it?

People talk about this societal vision of leisure time, where the idea is: you can do whatever you want, you don't have to do anything, maybe you take up painting. I think that would be terrible for a lot of people. We see today that when people don't have enough to do, that usually doesn't work out well for them. It's precious, and while I don't have all the answers here, this is the question I want to engage with personally. Because of the safety situation we're in and through the work I did, having these serious conversations about safety is, for me, fundamentally an issue of wisdom.

Ezra Klein: That's a great place to wrap up. And one last question: What three books would you recommend to the audience?

David Robinson: Okay. Number one is The Challenger Launch Decision by Diane Vaughan, a sociologist. It's a book about why the Challenger exploded. I thought I knew the story: middle management cut corners, the rubber O-rings became brittle in the morning cold, and they broke, and the idea was that these people were just stupid. It turns out that the risk of the O-ring failing in the cold was well known, documented, and accepted in safety documents over and over again. Even on the eve of the launch, there was a late-night meeting among engineers worried about whether this specific launch was safe because it was so cold.

Ezra Klein: Why is this book relevant in your view?

David Robinson: Because I think we are in a very similar situation. If they had said the launch wasn't safe, it would have reopened the controversy over whether prior launches had been safe, even though they went fine. What I worry about in our industry is that we take risks, nothing terrible happens, and then the next step is slightly different. Instead of a totally new world every few months, it moves just slightly every week. You can imagine how this gradual normalization of risk leads to disaster. It struck me that I should send copies of this book to my former colleagues.

For the second book, I have a one-year-old and a four-year-old: Little Witch Hazel, illustrated and written by Phoebe Wahl. It's an incredibly beautiful picture book, and my daughter's eyes light up every time we pull it off the shelf. If there are parents looking for a great book, I highly recommend it.

And third, which I think is the most important if you're going to pick one, is The Sabbath by Rabbi Abraham Joshua Heschel. It's one of my favorite books. It comes from my tradition as a Jew, but it's worthwhile for everyone. It has this famous phrase about Saturday being "a cathedral in time." What a tradition—to stop what you're doing, take a rest, and build an architecture in time rather than in space. I've been thinking a lot about this: building cathedrals in time. If I could quote the parting thought I shared with my colleagues: we need to take the time to make the right choices before time runs out. I hope we use that wisdom.

Ezra Klein: David Robinson, thank you very much.

David Robinson: Thank you.