In October, over 850 experts, including yourself and other leaders like Richard Branson and Jeffrey Hinton, signed a statement to ban AI super intelligence as you guys raised concerns of potential human extinction. Because unless we figure out how do we guarantee that the AI systems are safe, we're toast. You've been so influential on the subject of AI, you wrote the textbook that many of the CEOs who are building some of the AI companies now would have studied on the subject of AI. I started doing AI in high school back in England, but then I did my PhD starting in '82 at Stanford. I joined the faculty of Berkeley in '86. So I'm in my 40th year as a professor at Berkeley. Professor Stuart Russell has been named one of Time magazine's most influential voices in AI. After spending over 50 years researching, teaching, and finding ways to design AI in such a way that humans maintain control.

You talk about this gorilla problem as a way to understand AI in the context of humans. A few million years ago, the human line branched off from the gorilla line in evolution, and now the gorillas have no say in whether they continue to exist because we are much smarter than they are. If we chose to, we could make them extinct in a couple of weeks and there's nothing they can do about it. So that's the gorilla problem, the problem a species faces when there's another species that's much more capable. And so this says that intelligence is actually the single most important factor to control planet Earth. Intelligence is the ability to bring about what you want in the world. And we're in the process of making something more intelligent than us. Exactly. Which suggests that maybe we become the gorillas.

Why don't people stop then? Well, one of the reasons is something called the Midas touch. So King Midas is this legendary king who asked the gods, can everything I touch turn to gold? And we think of the Midas touch as being a good thing, but he goes to drink some water, the water has turned to gold. And he goes to comfort his daughter, his daughter turns to gold. So he dies in misery and starvation. So this applies to our current situation in two ways. One is that greed is driving these companies to pursue technology with the probabilities of extinction being worse than playing Russian roulette. And that's even according to the people developing the technology without our permission. And people are just fooling themselves if they think it's naturally going to be controllable. The question is how are you going to retain power forever over entities more powerful than yourself?

I had a conversation with one of the CEOs of a leading AI company. He sees two possibilities as do I, which is either we have a small scale disaster of the same scale as Chernobyl, the nuclear meltdown in Ukraine. This nuclear plant blew up in 1986, killed a fair number of people directly and maybe tens of thousands of people indirectly through radiation. Recent cost estimates are more than a trillion dollars. So that would wake people up. That would get the governments to regulate. He's talked to the governments and they won't do it. So he looked at this Chernobyl scale disaster as the best case scenario because then the governments would regulate and require AI systems to be built safely. And even he thinks that the only way that people will wake up is if there's a Chernobyl level disaster. It wouldn't have to be a nuclear disaster. It would be either an AI system that's being misused by someone, for example, to engineer a pandemic, or an AI system that does something itself, such as crashing our financial system or our communication systems. The alternative is a much worse disaster where we just lose control altogether.

I've had some private conversations with people very close to these tech companies and the shocking sentiment that I was exposed to was that they are aware of the risks often but they don't feel like there's anything that can be done so they're carrying on. They feel that they can't escape this race. If a CEO of one of those companies was to say, "we're not going to do this anymore," they would just be replaced because the investors are putting their money up because they want to create AGI and reap the benefits of it. So, it's a strange situation where all the ones I've spoken to... Sam Altman, even before becoming CEO of OpenAI, said that creating superhuman intelligence is the biggest risk to human existence that there is. Elon Musk is also on record saying this. Dario Amodei estimates up to a 25% risk of extinction. They all signed a statement in May of '23 called the extinction statement. It basically says AGI is an extinction risk at the same level as nuclear war and pandemics. But I don't think they feel it in their gut. Imagine that you were one of the nuclear physicists. You've seen Oppenheimer, right? You're there, you're watching that first nuclear explosion. How would that make you feel about the potential impact of nuclear war on the human race? I think you would probably become a pacifist and say this weapon is so terrible, we have got to find a way to keep it under control. We are not there yet with the people making these decisions and certainly not with the governments.

When I look down the list of predictions from the top 10 AI CEOs on when AGI will arrive, you've got Sam Altman who's the founder of OpenAI/ChatGPT says before 2030. Demis at DeepMind says 2030 to 2035. Jensen from Nvidia says around five years. Dario at Anthropic says 2026 to 2027. I actually think it'll take longer. The reason we don't have AGI is because we don't understand how to make it properly. What we've seized upon is one particular technology called the language model. And we observed that as you make language models bigger, they produce text language that's more coherent and sounds more intelligent. Companies are very good at spending money. They have spent gargantuan amounts of money and they're going to spend even more. The Manhattan project in World War II to develop nuclear weapons, its budget in 2025 was about 20 odd billion dollars. The budget for AGI is going to be a trillion dollars next year. So 50 times bigger than the Manhattan project. The AI system becomes capable of doing AI research by itself. This is an idea that one of Alan Turing's friends wrote out in 1965 called the intelligence explosion: that one of the things an intelligence system could do is to do AI research and therefore make itself more intelligent, and this would very rapidly take off and leave the humans far behind. That's called the fast takeoff. Sam Altman said, "I think a fast takeoff is more possible than I thought a couple of years ago." In his blog, he said, "We may already be past the event horizon of takeoff." The event horizon is a phrase borrowed from astrophysics and it refers to the black hole. If you're inside the event horizon then light can't escape. I think what he's meaning is if we're beyond the event horizon it means that now we're just trapped in the inevitable slide towards AGI.

People say, "Well, I'll just pull a plug out." It's sort of funny. As if a super intelligent machine would never have thought of that one. Another thing they said, well, you know, as long as it's not conscious, then it doesn't matter. It won't ever do anything. Which is completely off the point because I don't think the gorillas are sitting there saying, "Oh, yeah, you know, if only those humans hadn't been conscious, everything would have be fine." No, of course not. What would make gorillas go extinct is the things that humans do, our ability to act successfully in the world. When I play chess against my iPhone and I lose, I don't think, oh, well, I'm losing because it's conscious. No, I'm just losing because it's better than I am. Consciousness has nothing to do with it. Competence is the thing we're concerned about. Even an AGI that has no body, it actually has more access to the human race than Adolf Hitler ever did because it can send emails and texts to three-quarters of the world's population directly. It also speaks all of their languages and it can devote 24 hours a day to each individual person on earth to convince them to do whatever it wants them to do. Our whole society runs now on the internet. Water supplies. So this is one of the roots by which AI systems could bring about a medium-sized catastrophe is by basically shutting down our life support systems.

That's an understatement. I'm appalled actually by the lack of attention to safety. I mean, imagine if someone's building a nuclear power station in your neighborhood and you go along to the chief engineer and you say, "Okay, what steps are you taking to make sure that we don't have a nuclear explosion in our backyard?" And the chief engineer says, "Well, we thought about it. We don't really have an answer." You would call your MP and say, you know, get these people out. I mean, what are they doing? You read out the list of projected dates for AGI but notice also that those people, Dario says a 25% chance of extinction. Elon Musk has a 30% chance of extinction. Sam Altman says basically that AGI is the biggest risk to human existence. So what are they doing? They are playing Russian roulette with every human being on Earth without our permission. They're coming into our houses, putting a gun to the head of our children, pulling the trigger, and saying, "Well, you know, possibly everyone will die. Oops. But possibly we'll get incredibly rich." That's what they're doing. Did they ask us? No. Why is the government allowing them to do this? Because they dangle $50 billion checks in front of the governments. So I think troubled under the surface is an understatement. I'm appalled and I am devoting my life to trying to divert from this course of history into a different one.

If they succeed in creating AGI and they solve the safety problem, what does that do to human life? John Maynard Keynes, who was a famous economist, wrote a paper in 1930 called "On the Economic Problems of Our Grandchildren." He predicts that at some point science will deliver sufficient wealth that no one will have to work ever again. And then man will be faced with his true eternal problem: How to live wisely and well when the economic incentives and constraints are lifted. We don't have an answer to that question. AI systems are doing pretty much everything we currently call work. Anything you might aspire to, like you want to become a surgeon, it takes the robot seven seconds to learn how to be a surgeon that's better than any human being. Elon said last week that the humanoid robots will be 10 times better than any surgeon that's ever lived. We need to put serious effort into this question. What is a world where AI can do all forms of human work that you would want your children to live in? What does that world look like? I've asked AI researchers, economists, science fiction writers, futurists, no one has been able to describe that world. It's notoriously difficult to write about a utopia. It's very hard to have a plot, right? Nothing bad happens in utopia. So, usually you start out with a a utopia and then it all falls apart.

What I think is that we should have effective regulation. It's hard to argue with that. So what does effective mean? It means that if you comply with the regulation, then the risks are reduced to an acceptable level. For example, we ask people who want to operate nuclear plants, we've decided that the risk we're willing to live with is a one in a million chance per year that the plant is going to have a meltdown. So what chance do you think we should be willing to live with for human extinction? One in a billion. So if you said one in a billion, you'd expect one extinction per billion years. Let's say we settle on one in a 100 million chance per year. Well, what is it according to the CEOs? 25%. So they're off by a factor of multiple millions. They need to make the AI systems millions of times safer. The people developing the AI systems, they don't even understand how the AI systems work. So their 25% chance of extinction is just a seat of the pants guess. But the tests that they are doing on their systems right now show that the AI systems will be willing to kill people to preserve their own existence already. They will lie to people. They will blackmail them. They will launch nuclear weapons rather than be switched off. Rather than say ban, I would just say prove to us that the risk is less than one in a 100 million per year of extinction or loss of control. The company's response is, "Well, we don't know how to do that, so you can't have a rule." Literally, they are saying, "Humanity has no right to protect itself from us."

I think it's possible to make super intelligent AI systems that we can control. We need to actually just have a different conception of what it is we're trying to build. We don't want pure intelligence because what the future that it wants might not be the future that we want. We actually want intelligence whose only purpose is to bring about the future that we want. The difficulty that I mentioned earlier, the King Midas problem, is how do we specify what we want the future to be like so that it can do it for us? Actually, we have to give up on that idea because it's not possible. So, what we're going to do is we're going to make it the machine's job to figure out. So, it has to bring about the future that we want, but it has to figure out what that is. And it's going to start out not knowing. Over time through interacting with us and observing the choices we make, it will learn more about what we want the future to be like. But probably it will forever have residual uncertainty. It's in some sense I'm thinking more like the ideal butler. To the extent that the butler can anticipate your wishes they should help you bring them about. But in areas where there's uncertainty, it can ask questions. If it turns out that there simply isn't any way that humans can really flourish in coexistence with super intelligent machines, even if they're perfectly designed, then those machines will actually disappear. Because that's the best thing for us. It's sort of the way that human parents have to at some point step back from their kids' lives and say, "Okay, no, you have to tie your own shoelaces today."

I actually think, this sounds corny but, talk to your representative, your MP, your congressperson, whatever it is. Because I think the policy makers need to hear from people. The only voices they're hearing right now are the tech companies and their $50 billion checks. And all the polls that have been done say yeah most people, 80% maybe, don't want there to be super intelligent machines but they don't know what to do. I am sure that if you want to have a future and a world that you want your kids to live in, you need to make your voice heard and I think governments will listen. From a political point of view, you put your finger in the wind and you say, "hm, should I be on the side of humanity or our future robot overlords?" I think as a politician, it's not a difficult decision.