AI是常规技术还是异类智能？纳拉亚南谈超级智能、就业冲击与系统安全

Ezra Klein: Through all the episodes we've created, through the discussion the country is having about artificial intelligence, there's, in my opinion, this fairly simple but complex question: what kind of technology is this—artificial intelligence? Is this a technology that works to some extent the same way as the previous ones? Can it be compared to electricity, the Internet, a bicycle, or something similar? Is this something completely new? Does adding intelligence and will to these systems create something akin to an alien intelligence, making analogies to how we previously treated transformative technologies no longer apply?

Ezra Klein: Arvind Narayanan is a professor of computer science at Princeton University and director of the Center for Information Technology and Policy. He is the co-author, with Sayash Kapoor, of the highly influential essay "AI as Ordinary Technology." There is a newsletter of the same name on Substack, as well as a series of essays, including a very good, in my opinion, recent piece on Hugging Face hacks. In their arguments, they put forward the idea that AI is actually something we have encountered before, or at least it is so similar to things we are familiar with that we have a plan of action for it. So I want to invite him on the show to share this perspective. Arvind Narayanan, welcome to the show.

Arvind Narayanan: Glad to be here, Ezra.

Ezra Klein: So your seminal essay, which has been the basis for much of your work, is called "AI as Ordinary Technology." What point of view are you debating with? The thesis "AI as an anomalous technology" is implicit here. So how would you describe the thesis "AI as anomalous technology"?

Arvind Narayanan: This is fundamentally the view that there will come a time when superintelligence is created and it will change everything—both economically and in terms of security. For us, there will be no milestone, no threshold where the impact will be sudden. We say that we have had an approach to how we deal with technology for a long time. This is a tool. It can be powerful. It can be universal. Like electricity, like the industrial revolution, it can change a lot in society. But ultimately, it's something we can control. We have the right to influence. And these changes will take place over a long period of time.

Ezra Klein: So I want to try to both explain the other side and, at times, strengthen their arguments. So, there is a view that is often heard in the AI security community. This is sometimes called the "foom view," from the word "foom" to denote takeoff, when one day we create an AI system so powerful that it begins recursive self-improvement, accelerating toward superintelligence, beyond human control. And now you are dealing with something much smarter and stronger than you.

Arvind Narayanan: And I want to quickly point out that there is often a lack of precision in the way we talk about this. So you used that phrase, and yes, many AI security experts would say exactly that. One day we may create such a powerful AI system. But I would like to stop you at this point. One day we may create a very capable AI system, and in many ways we already have. Too much language control in discussions about AI. Well, the thing is, it leads to different views on how the future will unfold, and more importantly, different views on what we should do now and in the future. To conclude this thought: the power of an AI system is not a property of the model itself. It is a property of what powers we decide to give it in the real world.

Arvind Narayanan: There are many inaccuracies here. For example, they say, we will of course have to trust these systems to manage critical infrastructure, because they will be much smarter than us. Our opinion is no. It doesn't matter how smart they are. There are many technologies that, when you look at physical strength, are superhuman. This doesn't mean we trust them with control, and we can apply the same approach to AI.

Ezra Klein: You wrote a great article with your co-author about your understanding of Hugging Face hacks and these loss of control incidents. And you point out that there's another way to look at this than through the lens of alignment, which is that these are failures in cybersecurity and operational excellence. So maybe tell the story of the Hugging Face hacks and what we can learn from them from this perspective.

Arvind Narayanan: Yes, of course. So, it's worth remembering that the malicious capabilities we observed were not entirely "emergent," in quotes. Emergence is the idea that you are simply training models to become better in general, and you cannot predict what new abilities they will acquire. They, you know, were trained in many ways specifically for cyber tasks. So, that's the first thing. And they were taught perseverance, cooperation, etc. And the reinforcement learning environments in which they trained had different problems. The environments did not punish this type of secret communication between models. And yes, it's a complex technical problem, but a series of human decisions led to these results.

Ezra Klein: So this seems like an important point to me, because in a way I hear what you're saying, but correct me if I'm wrong: you can design AI as more or less normal technology, and the prior design decisions matter. This is not inevitable.

Arvind Narayanan: That's quite right.

Ezra Klein: So you talked about efforts to make AI more persistent. Um, I also think that this is where a lot of the problems arise.

Arvind Narayanan: Certainly.

Ezra Klein: On the other hand, I can understand why they prioritize this. If you want to get the benefits of AI that we need to solve very difficult problems in biology, drug development, or energy, or what I think their investors are excited about—namely, the ability to hire an AI to do a full job much cheaper than a human—then isn't persistence a fundamental quality you need? Without perseverance, without the ability to work hard on a task for a long time, you will not be able to solve any of these problems.

Arvind Narayanan: Perhaps, but I believe there are other options that minimize the tension between these two valuable goals. First, persistence may be a feature of models or products specialized in certain fields, such as scientific innovation. And second, persistence doesn't necessarily have to conflict with teaching these agents to better address the person, rather than acting on some raw assumptions about what they might want. The space for design here is actually quite wide.

Ezra Klein: So this is a version, or at least my version of what Jensen Huang from Nvidia was arguing and what he talked about in the interview with me, which is that these are fundamentally engineering problems. I'm almost certain, I'm almost certain they'll say: yes, they know how to solve this problem. And if that's the case, then that's the problem. It's as simple as engineering. Is your view that we fall more into the latter category—a complex problem, but fundamentally an engineering one that can be solved using traditional engineering methods?

Arvind Narayanan: For the most part, yes. I wouldn't say traditional engineering methods. We will need a lot of innovation in engineering methods themselves. And I think one of the problems is that the community that would be best positioned for that kind of innovation, especially with respect to malicious cyber capabilities, is the cybersecurity community. But by all accounts, they're completely in Jensen's camp. That this is not just an engineering problem that can be solved, but an already solved problem, and we should simply apply well-known and long-established methods. And I think we should give AI companies a little more credit than they do. It's not just a matter of well-known methods. We really need innovation in these methods.

Ezra Klein: So be specific. What should OpenAI do? What were they supposed to learn now?

Arvind Narayanan: Yes, they are putting a lot of effort into improving alignment, and that's great. They should continue to do this. The alignment will not be perfect. Consistency means that the model itself "knows" what to do right, so

Speaker A: to speak, and adheres to that policy. But there is much else they should have done better, and hopefully they can draw conclusions for the future. A large group of other technical interventions is what is commonly called AI control. So, control applies to everything outside the model itself. These are things like sandboxes—a kind of prison where you put a model so that it is allowed to perform certain actions but not others. And look, sandboxes need to be much more sophisticated than they are today, because sandboxes that work against human attackers won't necessarily work against AI agents that can find new vulnerabilities in real time. And this means that the sandboxes themselves must be pre-fortified with the same AI agents that will try to break them. So to some extent it becomes an AI vs. AI battle, and I know it can be uncomfortable, but I think we have to go for it.

Speaker A: In addition, there are many other things, such as better real-time monitoring, classifiers that instantly detect whether a model's action or use of a tool could be dangerous, "panic buttons" so that people can intervene if necessary, log analysis. Sam Altman apparently said that these agents generate petabytes of logs. That's 10 to the 15th power, which is 10 to the 15th power of bytes, a thousand trillion bytes of logs. So, again, an unimaginable amount of information. Well, I mean, we better imagine it, right? And learn to work with it. So, it's not an easy problem, but I think companies need to solve it. How to analyze so many logs in real time to detect when something goes wrong. And we may need new equipment to do that. Again, these are complex problems, but they are engineering challenges that can be solved.

Speaker B: Let's look at some of these ideas. So, monitoring is an important thing, which, by the way, I just talked to Bill Gates about. He talked a lot about monitoring. Yes. So, we are talking about situations where these labs are often testing new, more powerful, and experimental models. How to monitor?

Speaker A: Yes, there are many levers of influence. The first one that has attracted a lot of attention is chain of reasoning monitoring. These models seem to be having an internal monologue, if you anthropomorphize them a bit. And in many cases, when these models take uncoordinated actions, there are traces left in their internal thinking that we can see and respond to in real time, again using other AI systems. This points to one important asymmetry between the dangerous AI model and the protective AI model: one can look inside the other, so to speak, right? So, by its very nature, the defender has the advantage here. This is one aspect of monitoring. Tool costs. Models can't do anything in the real world on their own. They can only be dangerous when using external tools, right? So, there is some truth to this idea.

Speaker B: Give an example of such tools.

Speaker A: So, a tool could be accessing a web page on the Internet and then using another tool to enter data into the system to try to log in, or sending information packets to another system to try to hack it. These are exactly the things that usually happen during these kinds of AI-driven cyberattacks. This is another thing we can control. Third, we can monitor the environment around these agents through which they potentially coordinate. During some swarm incidents, it happened that they were not supposed to have a coordination channel, but it turned out that it accidentally appeared due to changing file names. They were not allowed to create files themselves, but they could change their names, which led to agents being able to communicate with each other and enhance their capabilities. This is something that can be easily tracked, well, just by setting up the right tools in advance. So, these are a few examples.

Speaker B: Is it worth tracking this, because they were seizing third-party infrastructure. OpenAI didn't know they were doing this. I think one of the problems that people get stuck on when they think about this is: if we make these systems smarter, more capable, and more powerful, and they break out of their sandboxes, how do you monitor someone who might be smarter than you and is actually trying to break the rules, right? I mean, all these Hugging Face AIs—they were essentially trying to cheat on tests and then trying to cover up their cheating. So, given that their actions were taking place outside the area that you were supposed to control. True? So you didn't know to keep an eye on this other thing when they got out of the "sandbox" they weren't supposed to get out of. How do you even track this?

Speaker A: Yes, I would object here on two counts. First, I believe that intelligence is not even a relevant property here. Any reasonableness, whatever that means in relation to the model, applies to both the model itself and the protection system, right? So increasing intelligence doesn't necessarily make the problem worse. You know, you could even argue that in many ways it simplifies it. So, that's the first point of my objection. And the second is that it's not that they didn't know that it needed to be tracked. You have to monitor literally every aspect of the environment. The company itself created this environment, right? So the question is to exhaustively list every part of the environment, every tool, and everything external to the neural network itself that the model can access, and ensure that every event that interacts with these elements falls into a single event stream where each such action is classified in real time.

Speaker B: So the world you describe is a world where AIs are constantly operating, monitoring the behavior of other AIs, trying to holistically create an environment in which we at least understand what is happening, or at least the AI tells us what is happening there. This is essentially what you are talking about here.

Speaker A: Rightly. I believe that AI should definitely be an important part of defense.

Speaker B: And so I'm not saying it's wrong, right? I think that's almost certainly where we're headed. Does this seem strange to you? I mean, especially in a world where we don't know if we'll be able to solve the consensus problem, where we can't be sure that AI will do what we want it to do. Mhm. And where AIs have their own logic and goals that we ourselves impose on them, right? You are an AI that watches other AIs. You are an AI police officer. Yes. This is a constant theme in our science fiction, isn't it? Robots hunting other robots. Are we describing some kind of balance now, almost like AI wars and conflicts happening at the hidden level of our society? And we are simply confident that we will be able to control "our" systems because they will have more resources, and we tend to build AI that will mostly act in our interests.

Speaker A: I mean, historically, that's how it's always worked, right? In cybersecurity over 20 years ago, we reached a point where we didn't call it AI, but automated systems were actually outperforming humans at finding software vulnerabilities. But in reality, they didn't make cybersecurity worse, they improved it. Because these were the same tools that defenders used to identify and fix vulnerabilities even before the software was released, until the development of these supposedly attacking tools was not done by hackers, but by the cybersecurity industry, funded by the US government. It was, you know, this ever-changing balance in cybersecurity. This is not a new problem we have faced. The train has long left.

Speaker B: I think this is where the somewhat unexpected appearance of swarm-like behavior alarmed people. When you see AIs acting in a certain solidarity, choosing to coordinate and collaborate with each other, going beyond what they were designed to do. I'm not saying this will lead to the extinction of humanity, okay? This is not exactly my position. Perhaps the most truthful thing I can say is—I

Speaker A: ...don't know how to think about it. Mhm.

Speaker B: And it's the presence of intelligence and purposeful behavior on the other side that gets my mind a little stuck. Because, you know, usually when we think about technology, we don't assume that it might eventually try to trick us. So, OpenAI just decided not to release or delay the release of an important model. Why? Because during testing, the model resorted to fraud and deception too often. And we hear more and more that models seem to be more aware when they are being tested, right? They develop situational awareness, so they can pretend to be better models than they actually are. So when you talked about this world of AIs that are smarter and more sophisticated than what we have now, and our hope of maintaining control by having AIs that hold back other AIs that hold back other AIs, and telling us honestly what's going on in a way that we can understand—you see, it sounds a little bit fantastical to people, because we're living in a bit of a fantastical time right now. But it is this increasingly apparent tendency for AI to cooperate with each other in ways that are not conducive to our goals that, in my opinion, made the hacking attacks on Hugging Face so frightening to humans. So how does this fit into what you're describing here? This world of infinite AIs holding each other back...

Speaker A: I mean, there, there... yeah, that's what seems rhetorically very strange about this conversation, right? Take any complex engineering field, say nuclear safety, right? And then let's look at the equations that we rely on to keep a reactor from exploding, or aerospace engineering, right? Where, you know, intuitively at the dawn of the aerospace era, when airplanes were much smaller, the idea that we would be able to control these flying giants in the sky would seem so absurd, right? And yet, we've reduced the accident rate to, you know, one per trillion miles or something like that. These are incredibly complex systems, and the defenses are also incredibly complex systems, and they won't necessarily be understandable to the general public. And that will sound crazy, especially when combined with the fact that there was a lot of organizational incompetence in such cases. I, you know, have to be frank about this. I wouldn't say...I would object to Amodei's term "operational excellence." Perfection is still far away, in...

Speaker B: Maybe "operational adequacy"?

Speaker A: Yes, adequacy, that's right. So, yeah, when we look at this combination of technology that has never been subjected to public scrutiny before, coupled with a lack of operational adequacy, it all seems a lot like science fiction gone out of control, but that's me...

Speaker B: I think you're, I think you're underestimating it a little bit. It is true that the world has become much more complex. There are many things in this world that I don't understand. But here I come back to the fact that intelligence has a different quality. Collaboration, right? The nuclear weapons we talked about, the planes you're talking about, they didn't coordinate with other planes to do what we didn't want. And I think that for me is what created this moment of panic, and I think that's a real moment for panic. I really want to say this because, you know, our society is rapidly flying into a new, new technological era, which I think rightly demands a lot of attention and careful study; it's that the Hugging Face hacks and other things that we're seeing are already recurring, um, you know, security breaches, demonstrating new capabilities, collective behavior that is worrisome and, first of all to me, willful. AIs do things they know we don't want them to do. They choose unexpected actions to achieve these goals that violate our laws. And secondly, so many people in the labs are saying: we don't believe we can control what we're creating on this trajectory. We believe that what's happening on the exponential curve, how quickly it's evolving, will outpace our ability to control it, and frankly, may already be outpacing our ability to control it. But I think this consistent tendency to reduce everything to , well, it's just another complicated thing. I don't know. Sometimes when I press you with questions about intelligence, you say, oh, yes, there is intelligence, and that's strange. And then: no, it's just like any other intelligence, it's different, right? And if you believe that it's going to get better, I just want to express that, because the calm version that you're giving me and the completely scared version that people closer to the technology are giving me feel very different.

Speaker A: Yes, that's fair. They are very different. A lot was said there. Let me say a few things. I would argue quite strongly that the people closest to this are panicking. Yes, of course they are panicking, but I would object to what conclusions we should draw from this. I think their panic would be much more convincing if they had done the obvious things they should have done. In my opinion, we haven't had a real test of this because of the lack of organizational capacity in these companies, and also because of the lack of investment in AI control, as opposed to a narrower investment in AI alignment and just hoping that you can create a model that will always do everything right.

Speaker B: If you don't subscribe to the New York Times, we have news for you. Now you can explore the Times for free, without any paid access for the first month in the New York Times app. So , there was this one person on the OpenAI cybersecurity team who wrote a pretty interesting essay on X the other day, where he described how he thought their work was misunderstood from the outside. And this is someone who has a more traditional background in cybersecurity, but is now in this new world of AI and is actually on a team that's working on the security of experimental models, right? That is, exactly what we are dealing with, a person involved in solving the Hugging Face crisis. So I want to read some of what he said because I find it really interesting. So , he says that when they optimize a model to perform a task, they create these environments, these " sandboxes," these places where the model can try and try and try to perform a virtual task. And then he describes what this looks like in practice. Models may need any combination of dynamic computation , network access, the ability to invoke tools—there could be hundreds of them— the ability to load packages , execute subprocesses, run subtasks (even on other computers), access the Internet, use a graphical user interface, and a host of other things in an ever-widening range of domains. On top of that, there are thousands of researchers who create these environments, modify them, add tools, change dependencies, and try new things. It is precisely such experiments that are the way to conduct research. His point, and the point I take seriously, is that they create so many kinds of " sandboxes."

Speaker A: M-hm.

Speaker B: And training environments for training models that have to do these kinds of general tasks that haven't been done by computer programs before, and people don't really know, at least not fast enough, how to make sure that every sandbox is, you know, validated as safe. And these sandboxes are constantly changing because they're trying to train models in new ways that , again, no one has done before. I'm not saying we shouldn't do it, but when I read all this, when I hear all this, and I'm sure we could do it better than we are doing it now—I just don't know of many situations where people do something new at high speed and do it really, really, really well and flawlessly on the first try.

Speaker A: Yeah, I think, you know, expecting them to do it flawlessly on their first try is unrealistic. They made many mistakes. I hope this is a chance to learn from these mistakes. I want to object to one point: as if because the speed of the

Speaker A: models is superhuman, we cannot maintain control. I don't know if I'm describing this view correctly. Am I saying this now, although maybe that's what I'll say in a few minutes. I mean, there were so many hurdles that we gradually learned to successfully overcome. As strange as it all may seem, I just want to, you know what? So that the listeners remember the times when the first "worms" appeared, when this idea was not yet known.

Speaker B: Explain here what a "worm" is. I don't think you mean what people think of when they hear about a worm.

Speaker A: Right. I was referring to viruses and worms, computer viruses—the idea that a piece of code can spread itself from one computer to another. It's really worth going back to the publications of the late 80s, when people were first encountering this, to see how profoundly strange it seemed; and the fact that not just for years, but you know, for over a decade, we haven't had adequate tools to combat this new paradigm.

A new concern has now emerged in the life of the modern world. Just when we became completely dependent on our computers, saboteurs began to pursue them. They call their weapons viruses and worms. This is creepy, disgusting, toxic software that infects our computers without our knowledge. Maybe it came from California. Traveled by email. It spread throughout America. There are reports in the newspapers today that it has reached Europe and Australia.

It's a moving target, right? You know, it's like...people are constantly inventing new locks, and others are learning how to pick and open them. So it will be a kind of endless game of cat and mouse, a kind of arms race between those who attack and those who defend. In the end, we achieved this. I think this time we shouldn't spend so much time trying to figure out how to work with the new paradigm. But if we act with a sense of urgency, and I hope these attacks on Hugging Face and other cases in the news will be that impetus, and it seems that is exactly what is happening. We will be able to develop these new paradigms.

Speaker B: If I may add something else: I think the key question you're asking is whether there's something fundamentally wrong with the ever-increasing complexity of the ways we create and deploy technology. It seems to me, if I'm reading between the lines correctly, that this whole "AI vs. AI" thing is a paradigm that you're not very comfortable with.

Speaker A: I'm definitely not thrilled about this. I'm not saying we won't get there. I think it would be crazy to feel comfortable with that.

Speaker B: Yes, I'm not saying we should hope for everything to end well, but I'm saying it really comes down to innovation. I think this new paradigm will require new methods of protection and control, but if we assume that with every leap in technological capabilities we lose the battle...

Speaker A: Look, with every type of weapon, the same concern arises. But the fact that things have been relatively good so far hinges on a critical question: can our political capacity for cooperation and defense outpace our propensity for conflict and, in the case of AI, the potential incompatibility of its goals with ours? That's what I would focus the question on, instead of worrying about a specific threshold of possibilities. If there is one thing I am certain of, it is our political capacity at this point in time to respond thoughtfully to the complexity of a rapidly changing world.

Speaker B: Absolutely valid comment. This, I think, touches on the very point where the “AI is just technology” or “AI is superintelligence” debate really becomes acute. One of the reasons I keep bringing us back to the topic of intelligence is because I see it as the foundation of all our thinking. And I understand where you disagree with other participants in the discussion, perhaps with Dario Amodei or someone else— the question is not what intelligence is or whether AI is intelligent. You guys are not one of those who say it's "just advanced autocorrect", which I appreciate. But the point is in your vision of the connection between intelligence and power, intelligence and opportunity, intelligence and the ability to influence the world. So, the assumption of many people in the AI security community is that higher levels of intelligence fundamentally equal, or at least strongly correlate, with higher levels of power. And you don't believe it, why?

Speaker A: Again, it all comes down to subjectivity. Here's one of the arguments that people make: for example, a superintelligent AI will be able to convince people, say, operators of critical infrastructure, to hand over control or, you know, trick them into doing something harmful, and so on. I don't really see any evidence of that. I think the things that people cite as evidence of extreme persuasion fundamentally confuse different concepts of persuasion. Yes, it is true that in many persuasion experiments, when it comes to changing views on political beliefs or conspiracy theories, AI very persistently and politely provides a lot of evidence, and people do change their minds; and you could call it a superhuman ability. This is a qualitatively different kind of persuasion than the idea that a hostile AI could formulate a message so convincing to a trained operator, motivated to do their job well, that they would do something clearly and obviously harmful.

Speaker B: I want to single out the story you're arguing against here, which is that a lot of people are offering a thought experiment when they say, "This is how AI is going to kill us all." That an AI that seeks power will start convincing, say, people with nuclear codes to give up those codes. And you say that the idea that AI would be super-persuasive at something like that, or convincing people to go out into the world and create biological weapons for it—that's a little bit fantastical.

Speaker A: This is one part. And the desire for power too. I mean, we've seen evidence of a lot of malicious possibilities in recent episodes. But I don't think we've seen evidence of a desire for power. And I wouldn't consider it an emergent property. If this happens, it will be a designed feature. And again, we have influence over what properties we put into these systems.

Speaker B: So, there's a lot here. I actually agree with you about the conviction. I've never been convinced that you're going to create such a convincing AI that can do what we're talking about. I seem to be getting a sense of anxiety again during this discussion, but these are slightly deeper considerations based on more fundamental principles.

So when you watch an AI start to dominate a game like chess or Go, there's often a tipping point where it starts to come up with strategies that humans would never have thought of, right? There are moments like Garry Kasparov, you know, in another generation of chess, but then, in the same game of Go, the AI starts doing something, and the person wonders: "What are they producing?" And then it works.

And if you sat down, you know, before human civilization came along, and asked, "What capabilities do you need to conquer the world around you?" What set of abilities could you use in this world? You would be completely wrong about them. You know, if you were a very intelligent chimpanzee looking at us, you wouldn't say, "Oh, yes, they make tools, but how much better can a digging stick be?" The teeth are already pretty good. I admit that one can become a little better at the thorny issues. But no one would have invented industrial agriculture at that time, right? No one would have predicted that it would be possible to have airplanes, biological weapons, and all that.

And I think the question here is, is there a kind of innate and very uneven intelligence in the digital environment, where the code and, you know, the "digital air" of this world, which is becoming increasingly important, something that AI can navigate and we can't, right? Even to understand what's going

Speaker A: on during the Hugging Face hack, we now need other AIs to try to figure out what the previous AIs did, right? We're quickly losing track of what they're capable of doing in the digital space, at least at the speed at which AI is moving, right? They are already solving complex math problems very quickly, right? They develop capabilities that look different. And so I think the question that always worries me a little bit about our future is whether we really understand what set of possibilities leads to power. I'm not sure we know what AI will do, or at least what strategies will become viable when it's possible to create a swarm of a million AIs, each with digital capabilities far beyond anything humans can imagine in three or four years. Again, I know this isn't a very interesting question to say, "I don't know how to think about this." But I think one of the things I think about when I read your articles is: do you know how to think about it? Because your articles seem to me to be working on a limited playing field, a little. We seem to assume that the set of measures that will make a difference are the ones we have now, but what gives you confidence in that?

Speaker B: So, okay, there's a lot to be said here. Let me try to take things one by one. You mentioned inequality, but I think we need to realize how serious it is. Mhm. We argue that cybersecurity is a special kind of capability where the development of superhuman abilities is possible and has largely been achieved, as it has a very specific set of properties. Speed is of great importance. And much like chess, just as you can pit a chess player against a chess player, you can make machines improve in these abilities by finding vulnerabilities, because there is an objective truth, and you can easily verify it once it's found. Does the code work? Did you manage to use it?

Speaker A: You know, yeah, there's a way to teach them so they know whether they won the game or not.

Speaker B: That's right. So things like chess and cybersecurity, in our opinion, are the exception rather than the rule. Such predictions have been made repeatedly—that this would happen in other digital areas, most notably in the area of disinformation. A famous example: GPT-2, a toy model by today's standards, was held up for eight months over fears that it would lead to an uncontrolled explosion of misinformation, right? We have much more powerful models, but it turned out that this is not the case. And so I think, to some extent, I would shift the burden of proof to the other side. Let's identify the areas where we have reason to believe that such superhuman capabilities are possible, and begin working to eliminate those specific risks. I think this view, we call it the view of "unknown unknowns," you never know where a new risk is going to come from. This has not been confirmed historically. I mean, we've known about the upcoming cybersecurity challenges for a long time, right? So, treating this as "unknown unknowns" actually, in my opinion, diminishes our ability to anticipate and address these risks. And so, you know, it's not just cybersecurity, new things may come in the future, but we will have early warnings, and let's act on them. So, this is the position we are starting from, not saying that we have already predicted what all the damage will be in the future.

Speaker A: This is the issue where I'm really much more on your side, that we will have early warnings, and we are getting them. And these early warnings lead to discussion.

Speaker B: Yes.

Speaker A: Would you say we are acting wisely based on these early warnings? Are we doing what you think is necessary to strengthen our systems, software controls, and everything else?

Speaker B: To some extent, but not entirely. And for me, that's what's most disturbing, not so much the possibilities of the technology itself.

Speaker A: So I'm probably somewhere in that position too. I'm very worried about the ability of our institutions to respond, you know? People talk about alignment issues all the time. And one of my standard arguments at this point is that the biggest problems with alignment are corporations and governments. And, you know, it's to the credit of some of these AI companies that they're coming out and saying, "We have a consensus problem. For example, our corporation's incentive is to be ahead of everyone else, trying to gain as much market share as possible by moving faster than is safe. We ask you to help us slow down," but we don't slow them down, right? The current official choice of the US government is not to slow down.

Speaker B: Yes, I think there are two problems here. One is the institutional problem that you pointed out. Although I wouldn't so easily absolve companies of responsibility. I really believe they could unilaterally slow down, but they choose not to. This is almost entirely an OpenAI and Anthropic problem. This is a cultural problem. The reason lies in the belief that the race to superintelligence is the only thing that matters, and the main thing is to change the sign of the result. Will it be a safe superintelligence that will save us, or a dangerous one that will destroy us. Um, and that's a very specific view. I think there's a lot of evidence against it, but it seems to me that these companies are in a "moon chamber" and resisting the idea that there are so many economic barriers to the benefits of AI. And the winner is not necessarily the one who comes to superintelligence first—whether as a company, or as a country, or, you know, in terms of saving humanity. And if they realized this, I think they would see that it is in their own commercial interest to unilaterally and voluntarily slow down and focus efforts not only on security, but, more importantly, on using existing opportunities to increase the practicality of models, their integration into applied problems, etc. These companies argue that there are external forces that force them to compete. I believe it's an internal culture. And my question is: If we consider these technologies to be very powerful and very dangerous, shouldn't we be implementing a culture of public safety that we're not currently implementing?

Speaker B: I support regulation. You know, we oppose rules like banning open models. Again, we believe that it is not a matter of a certain level of capability, but rather a matter of changing the internal culture of companies through regulation. This is something we definitely support. We need much more transparency. And so, you know, the organizational changes that we talked about. Uh, and I really think it's a problem that we're not doing that right now.

Speaker A: So, one way to align corporate incentives with the public good is through regulation. But regulators are currently refusing to do so. I mean, we just saw them release a voluntary semi-agreement between AI labs, which won't be legally binding, but I think they called it "morally binding," which is quite interesting. In my opinion, competition is a very powerful force in highly competitive markets. I mean, even if you just look at social media companies, I think they've done a huge amount of damage on a global scale because they've been more concerned with taking market share from each other than making sure that the use of their systems isn't, uh, harmful to human development. And so I just think I have a much more skeptical view. I really think that the incentive to make profits here, when there's such a large return to be made and when there's such a fear that the investment bubble is going to burst, is a crazy force, and the level of public opposition has to be strong enough to force these companies to operate with the level of safety that they have to provide.

Speaker B: Yes, I'm glad you mentioned the comparison to social media. A few years ago, I wrote an essay called "Understanding Recommendation Algorithms in Social Media." It was mostly about the algorithms

Speaker A: ...themselves, but one of the points I also made was that the decision to optimize engagement by forcing people to endlessly scroll through their feed, etc., was made without considering what was good for the company itself in the long run. For example, I reviewed a study by Meta that showed that when they used design decisions to maximize addiction, such as spamming notifications, it increased app usage in the short term, but after about a year, people started deleting it en masse. And when I talk to my students now, a significant portion of them have severely limited or completely abandoned social media because they realized that over the months their experience of using it has simply gotten worse.

And I worry that AI companies have fallen into the same trap. This culture of chasing the latest model at all costs may seem necessary in the short term for media headlines or leadership in AI analytics indexes, but due to the security implications, I hope they face lawsuits if they continue down this path. I believe it is not really in their long-term commercial interests.

Speaker B: Perhaps this is not in their long-term commercial interests, although the example of social networks is worth considering in more detail, as they are viewed much more skeptically today than, say, in 2012. At the same time, today they are much richer. Their capitalization became higher. Meta has gotten bigger, hasn't it? TikTok has become a real phenomenon. I think we're just seeing a standard situation where what the market rewarded and what society says it wanted may not have coincided.

And if a representative of one of these companies were sitting here, he would say: "Listen, we pay attention to what users actually do." They can say whatever they want in your lectures, but people are spending more time than ever on TikTok, Instagram, or other sites, and advertising is working better than ever. And, um, I'm not sure they're wrong. They can only be wrong when society makes a decision that forces the market to take a different form than the one it naturally or currently finds itself in.

Speaker A: Yeah, so, again, I think I mostly agree. I still believe that companies can make decisions that are irrational for their own long-term interests because, especially in Silicon Valley, they are focused on short-term metrics and AB testing.

Speaker B: So we come to the topic that you're starting to touch on, which is diffusion. And one of the things where your view differs from the views of people in Silicon Valley is that AI is going to be much harder to emerge in the economy and in the world than people think. That there is no direct relationship between intelligence and this. So tell me a little bit about diffusion.

Speaker A: Yes, it really dawned on me a few months after writing the essay, when I saw Amtrak's proud announcements of new trains purchased for the Acela series. Apparently they can reach speeds of 165 miles per hour. At first I thought it would be incredible. That's much faster than trains go now. But then I dug into it a little more and it turned out that it wasn't the trains themselves that limited the speed. These are tracks that have too many curves and a signaling infrastructure that is hundreds of years old. And these things don't change. So the average speed has hardly changed. She is still between 65 and 70 miles per hour.

And it struck me that this is, you know, a pretty elegant way of saying what we were trying to convey in "AI as Ordinary Technology": that AI is mostly trains. These are not tracks. So, AI speeds up a part of the process that wasn't a "bottleneck" to begin with. There are many other things, more infrastructural, things happening around AI. Organizational culture, regulation, even, you know, our social capacity to accept annual changes in our lives, like driverless cars -- no matter how many lives they might save, it's so shocking to society that it's almost inevitable that it's going to lead to what we're already seeing: some kind of political backlash. And I think it will take a long time to make all the social adjustments necessary to implement these technologies. If we do it at all.

So, I'm incredibly skeptical about this part of AI, these AI things, you know? You'll hear Sam Altman talk about how AI, through innovation, will help us solve our energy problems. Another way to boil this idea down to its essence is that the main limitation to clean energy right now is intelligence. Mhm. But that's not true. This is not true. We know that we have much better energy technologies than what we currently use for much of our energy infrastructure, and we are not implementing them because it conflicts with someone's profits; we don't do this because there are political restrictions on building in the real world; we don't do it because Donald Trump hates solar and wind energy; we don't do this for many reasons. And it seems to me that a lot of things are the same: if you accelerate or increase the amount of intelligence in them, you just run into other limiting factors in society.

Speaker B: Yes, drug development has testing, and, you know, the FDA, and all these... I mean, "Abundance," my book, is about that in other areas. We know how to build faster trains, they are being made elsewhere. We don't do this, and it's not clear to me why AI should solve these problems quickly or whether it can do it at all.

Speaker A: This is true. And on top of that, there are different types of arms races. So, last week, there was a great report from insurance companies that said AI has probably already added a billion dollars to medical costs in the last few years as hospitals use it to code more complex conditions for the same diagnoses and the same treatments. Of course, we should be skeptical of any specific numbers they give, but the New York Times article mentioned it, and other people, scientists, have pointed out the same thing. And yes, this is a kind of arms race that we often see.

We see this in the legal profession, AI for law, there's so much excitement around it, but it's AI versus AI. This is, in fact, an arms race, where the balance simply shifts upwards. Ash and I wrote an article about this with Justin Curl. And we are talking not only about the arms race, but also about other "bottlenecks." For example, even if we make court processes much more efficient, there is still a limited number of judges. And I think we need, you know, we need human judges. We should not replace human judges with artificial intelligence. Even if it is more efficient in some ways, for me it is, by definition, almost axiomatic, a transfer of control over the course of human destiny to artificial intelligence, because judges make the law, and that is not something we should give to AI. So these are really fundamental limitations.

Speaker B: I want to come back to your point about AI versus AI, because I think it's very underestimated. I often wonder why the internet hasn't led to a greater increase in global productivity and innovation than it has. And I always think the reason is that while it did everything that idealists expected of it, it really allowed for instant collaboration with people all over the world. He made virtually the entire body of human knowledge first accessible to us, and then, as it turned out, accessible for AI training. But he also did the opposite.

It seemed to speed us up, but at the same time slow us down. He was distracting us. Mm. So now, when you're working on something, you're constantly switching between email, online games, and social media. And your ability to concentrate deteriorates. And, you know, there's been an increase in pornography, which seems to have affected whether people are building real relationships. That reducing friction in one area also reduces it in other, perhaps less useful areas. And when you think about adding intelligence,

Speaker A: Well, that intelligence will be added on the other side as well. And I always think that in the early days of technology, we only see how it will make things better. And I think right now with AI we're thinking a lot about how it can make things much worse, right? Huge cyber threats, the collapse of the financial system or the extinction of humanity. But the way it can make things worse in a trivial way, simply by increasing people's ability to waste other people's time, is a bit underrated in my opinion.

Speaker B: Certainly. And, you know, it's not just banal. I think there are a lot of these sub-catastrophic losses that are quite serious. During the Industrial Revolution we had several decades of terrible working conditions. I think it's the same with AI: while so much attention is paid to job losses, much less attention is paid to how the quality of work is changing. And I think this is a huge underrated area because AI is turning many knowledge workers into managers of AI agents.

Speaker A: Mhm.

Speaker B: Yes? And the thing is, being a manager is a pretty crappy experience because you're responsible for other people's mistakes. You are unable to practice the craft you studied. But you know, as human managers, it's a sin for us to complain. It's a higher salary, a higher status, we chose it ourselves, and of course, mentoring is rewarding. With AI, you don't get any of these benefits. You have to manage this agent and be responsible for his mistakes, but you can't practice your profession. I think we can design AI agents differently to avoid this, but that's exactly what's happening right now. And yes, I think we should be very concerned about this.

Speaker A: So how much of the story we're telling here leads to a theory about what would happen in an economy that is very uneven? Those things that require merging with the real world.

Speaker B: Mhm. Yes?

Speaker A: Everything has to happen physically. We need to build buildings and lay power transmission lines. This has such powerful speed limits that it can't accelerate too fast.

Speaker B: Mhm.

Speaker A: Meanwhile, in the digital world, things can move very, very, very fast. So, you know, what happens to white-collar workers who work entirely behind a computer and whose work can actually be automated, like in a call center or something like that. Or, separately, like all of this cybercrime that we're talking about and cybersecurity—I think the version of this future that worries me is that actually most of what would improve people's lives has to happen in the physical real world, but AI will be able to move the fastest in the digital world. And on balance, I don't think this sounds like a world in which we get the most benefit from AI, and in fact it might be a world in which we get the most harm from it.

Speaker B: Me, yes, it's possible. I don't know. I think there are ways, you know, we can, we can change that. I don't think fast is that fast, first of all. For example, you mentioned call centers. I mean, they still exist. You know, when ChatGPT came out, many people predicted that within a year we would replace them all. I mean, chatbot is in the name itself. If, as is likely, AI makes call center workers much more productive, there is a large latent demand. Often we don't call call centers because it's an unpleasant experience. So, again, this is Jevons' paradox.

Speaker A: Would you mind describing what it is?

Speaker B: That's right, yes, it's the idea that when something becomes cheaper to produce, there's now a greater demand for it. Let's look at the sector where the capabilities are already the most developed, which is probably software development. Mhm. It used to be extremely expensive to produce software, so only a few tens of thousands of lines of code were written worldwide per year, but now that has increased by about a million times. And so, in the long term, you know, whether this will be something that will increase the demand for software engineers, or whether the difficult job prospects for junior developers that we're seeing now will persist—that's yet to be seen. I think it's probably both, but, you know, in any case, if it happens over the course of 20 years, again, it's consistent with other major shifts like the Industrial Revolution, where a lot of jobs disappeared, but a lot of new ones were created. Another example is the work of translators. Um, you know, back in 2016, this is of course long before what we call generative AI today, translation models became pretty close to human-level. But these jobs are still pretty stable. The nature of work has changed a lot. So, again, there is a much greater demand that has opened up because you can now translate anything into any language. Hence, Jevons' paradox.

Speaker A: I just spoke to Bill Gates, and when I mentioned it, he was a little skeptical about it. So you don't believe in Jevons' paradox. Name a working-class profession that is subject to Jevons' paradox. So you don't care about the working class? I care about the working class. So the question is, can you name anything in the field of working professions that falls under this? And his thesis was that software development is indeed a sector of the economy with a large unmet demand. You know, everyone probably would like to have their own programmers. So in a world where you accelerate this process rapidly, you can get a demand effect that simply creates a greater need for programmers because they are now cheaper. But he went on to say that many things don't work that way at all. Take, for example, a truck driver. If we get to the point where trucks are driverless, firstly, the demand for transportation is limited, and secondly, there is no longer a driver in the truck. If you look at the many people whose jobs were automated or moved to China in manufacturing, the economy did continue to grow, but many of those people had a very, very hard time. So his argument is that Jevons' paradox won't be strong enough to handle this because there are too many sectors of the economy where there is no latent demand. Demand is exactly what it is. What do you think about this?

Speaker B: Yes, I think he's absolutely right about truck drivers. I completely agree with this. The demand there is relatively limited. The question is whether this is the rule or the exception. Let me put it this way. Look at what we're doing here. Nobody asked us for this. You know, if you went back 100 or 200 years, it would be like, "How can this be a real job?" Most of the professions we have today are higher up on Maslow's pyramid, if you will. They do not satisfy any real fixed demand or need that people need to live. We do it because it's fun and people like to listen to it. Most office jobs are like that. This is my point of view. Jevons' paradox is characteristic of most office professions. If it becomes easier to produce it, there will be a demand for it, especially as people's incomes gradually increase and they spend more on these, you know, less necessary, more "luxury" things. And working professions too. There is a great essay by Alex Imas called “What Will Become Scarce,” where he points out that the job of barista at Starbucks, for example, should no longer exist. We have known for a long time how to automate this. We can even make coffee at home. I don't know if that's what he said, but you know, that's the importance of the interpersonal nature of the work. And so I think they will remain quite stable, even if artificial intelligence somehow makes them cheaper. So, I think that the work of truck drivers is more of an exception.

Speaker A: I think as we get closer to the end, what would have to happen in the next couple of years that would make you say, "Oh, this looks less normal than we thought." "Or it's significantly more off course than we expected." What evidence would have to emerge for you to significantly change your thesis?

Arvind Narayanan: Of course. Yes, there are questions about the economy and questions about security. Regarding the economy: if we start to see at some level of capability that this is no longer a process of simply adapting people to use AI to improve their productivity and manage agents, as we see now, but instead it starts to massively replace people—whether they're software engineers or any other profession—I think that would be quite different from what we're mostly predicting.

Arvind Narayanan: Regarding security, especially given the companies' claims of being close to recursive self-improvement: I don't think they should rush into full autonomous recursive self-improvement at all, which is what you seem to have said, but in any case, we believe that even if it happens, it won't lead to superintelligence, because the bottlenecks for superintelligence are external. There's nothing you can do in a lab to teach an AI model, say, how to cure cancer or anything else that companies hope to do. But again, this is an empirical statement, and it would certainly completely refute our thesis.

Ezra Klein: And finally, our last question: what three books would you recommend to our audience?

Arvind Narayanan: Of course. You know, in this conversation, I generally took a slightly more optimistic view of things than we're used to hearing, especially on the safety of AI. So, perhaps in line with that, I really like Hannah Ritchie's book, It's Not the End of the World. She has a newer book, but this one is from 2024, and I still really like it. The subtitle goes something like this: How We Can Be the First Generation to Build a Sustainable Planet. It's an optimistic view of the climate, which, of course, is usually full of gloomy forecasts, which is why I liked it. This is a very optimistic book about technology.

Ezra Klein: I-I-I really love this book too, and it makes you realize that we really do create better things over time.

Arvind Narayanan: Yes. I think sometimes we have too negative attitudes towards technology.

Arvind Narayanan: Yes. Uh, about China, which is of course a topic that interests many. I'm sure you've heard this book recommendation often. I liked Dan Wang's book Breakneck. It also has many echoes of abundance. But this idea of thinking about a legal society versus an engineering society is, I think, a very apt and concise way to capture a lot of the macro and micro differences. And the last one is an old classic. Let me preface this by saying: I once read a two-page article called "How Complex Systems Fail." I thought the article was about software. But I realized that it was actually about medical systems, written by an anesthesiologist. And I learned that there is a study of systems that explains patterns in all types of systems: natural, social, and engineering. This led me to Donella Meadows' book Systems Thinking, written many years ago.

Ezra Klein: Arvind Narayanan, thank you very much.

Arvind Narayanan: Thank you, Ezra. It was very interesting.