[Host]: So much of what's happening today in the AI industry is extremely inhumane.

[Karen]: But this is me playing devil's advocate. And logically, it could be the case that the civilization that accelerate their research with AI is going to be the superior civilization.

[Host]: No, it's not. This is a prediction that you're making, right?

[Karen]: Making Zuckerberg's making.

[Host]: And do you know what the common feature of all of them is? They profit enormously off of this myth. You know, I have all these internal documents showing that they're purposely trying to create that feeling within the public so that they can extract and exploit and extract and exploit. So, what do we do about it?

[Karen]: We need to break up the empires of AI.

[Host]: You know, I've been covering the tech industry for over 8 years, interviewed over 250 people, including former or current OpenAI employees and executives. And I can tell you that there are many parallels between the empires of AI and the empires of old, right? like Lelay claimed the intellectual property of artists, writers, and creators in the pursuit of training these models. Second, they exploit an extraordinary amount of labor, which breaks the career ladder because someone gets laid off and then they work to train the models on the very job that they were just laid off in, which will then perpetuate more layoffs if that model then develops that skill. And when they talk about that there's going to be some new jobs created that we can't even imagine, a lot of the jobs that are created are way worse than the jobs that were there. And then there's the environmental and public health crisis that these companies have created and how they're able to also spend hundreds of millions to try and kill every possible piece of legislation that gets in their way and will censor researchers that are inconvenient to the empire's agenda. But what I'm saying is not that these technologies don't have utility. It's that the production of these technologies right now is exacting a lot of harm on people. But we have research that shows that the very same capabilities could be developed in a different way that doesn't have all of these unintended consequences. So let's talk about all of that.

[Host]: Karen, how you've written this book in front of me here called Empire of AI: Dreams and Nightmares in Sam Altman's Open AI. I guess my first question is what is the research and the journey you went on in order to write this book we're going to talk about and the subjects within it today

[Karen]: I took a strange route into journalism I studied mechanical engineering at MIT and so when I graduated I moved to San Francisco I joined a tech startup I became part of Silicon Valley and I basically received an education in what Silicon Valley is about because a few months into joining a very missiondriven startup that was focused on building technologies that would help facilitate the fight against climate change. The board fired the CEO because the company was not profitable. And this was in hindsight a very pivotal moment for me because I thought if this hub is ultimately geared towards building profitable technologies and many of the problems in the world that I think need solved are not profitable problems like climate change. Then what are we actually doing here? like what how did we get to a point where innovation is not actually necessarily working in the public benefit and sometimes even undermining the public benefit in pursuit of profit. In that moment, I had a bit of a crisis where I thought, well, I just spent 4 years trying to set myself up for this career that I now don't think I am cut out for. And I thought, well, I might as well just try something totally different. I've always liked writing and that's how after 2 years I landed at a role at MIT technology review covering AI full-time and that gave me a space to then explore all of these questions of who gets to decide what technologies we build how does money and ideology also drive the production of those technologies and how do we ultimately make sure that we actually reimagine the innovation ecosystem to work for a broad base of people all around the world. And so that is kind of how I then set off on this journey of ultimately writing a book. I didn't realize that I was working towards writing a book, but starting in 2018 when I took that job was essentially the moment in which I began researching the story that I I document in it.

[Host]: A very timely time to start working in artificial intelligence. For anyone that doesn't know, this is pre OpenAI chat GPT launch moment that shook the world. But in writing this book, you interviewed a lot of people and went to a lot of places. Can you give me a flavor of how many people you've interviewed, where it's taken you around the world, etc.

[Karen]: I interviewed over 250 people. So over 300 interviews, over 90 of those people were former or current OpenAI employees and executives. So the book covers the inside story of opening eyes's first decade and how it ultimately got to where it is today. But I didn't want to write a corporate book. I felt very strongly that in order to help people understand the impact of the AI industry, we would also have to travel well beyond Silicon Valley. These companies tell us that AI is going to benefit everyone and that's their mission. But you really start to see that rhetoric break down when you go to the places that look nothing like Silicon Valley, that speak nothing like Silicon Valley, and that have a history and culture that are fundamentally different as well. And that's where you start to really understand the true reality of how this industry is unfolding around us.

[Karen]: I think we should start with when AI started as a field. So this was back in 1956 and there were a group of scientists that gathered at Dartmouth University to start a new discipline, a scientific discipline to try and chase an ambition. And specifically an assistant professor at Dartmouth University, John McCarthy decided to name this discipline artificial intelligence. This was not the first name that he tried. The previous year he tried to name it Automata Studies. And the reason why some of his colleagues were concerned about this name was because it pegged the idea of this discipline to recreating human intelligence. And back then, as is true today, we have no scientific consensus around what human intelligence is. There's no definition from psychology, biology, neurology. And in fact, every attempt in history to quantify and rank human intelligence has been driven by nefarious motives. It's been driven by a desire to prove scientifically that certain groups of people are inferior to other groups of people. There are no goalposts for this field and there are no goalposts for the industry when they say that they are ultimately trying to recreate AI systems that would be as smart as humans. How do we even define what that means? And when are we going to get there if we don't know how to define the destination? And what that effectively means is that these companies can just use the term artificial general intelligence which is now the term to refer to this ambitious um goal to recreate human intelligence. They can use it however they want to and they can define and redefine it based on what is convenient for them. So in OpenAI's history, it has defined and redefined it many times. When Sam Alman is talking with Congress, AGI is a system that's going to cure cancer, solve climate change, cure poverty. When he's talking with consumers that he's trying to sell his products to, it's the most amazing digital assistant that you're ever going to have. When he was talking with Microsoft, you know, in the deal that OpenAI and Microsoft struck where Microsoft invested in the company, it was defined as a system that will generate hundred billion of revenue. And on OpenAI's own website, they define it as highly autonomous systems that outperform humans in most economically valuable work. This is like not a coherent vision of one technology. These are very different definitions that are spoken out loud to the audience that needs to be mobilized to ward off regulation or get more consumer buy in into the the industry's quest or to get more capital more resources for continuing on this journey with ambiguous definitions. I mean, speaking about different definitions through time, in 2015, in a blog post that Sam Waltman wrote before open air was officially announced, he explicitly outlined the existence risk by saying, "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen, for example, an engineered virus, but AI is probably the most likely way to destroy everything in general." When Alman is writing for the public or speaking for the public, he does not just have the public as the audience in mind, there are other people that he is trying to motivate or mobilize when he says these things. And in that particular moment, Alman was trying to convince Elon Musk to join him on co-founding OpenAI. And Musk in particular was spending all of his time sounding the alarm on what he saw as a huge existential threat that AI could pose. And so in that blog post, if you look at the the language that Alman uses side by side with the language that Musk was using at the time, it mirrors all the things that Musk was saying

[Host]: identical. I mean, 10 years ago, Musk was going on podcast saying, tweeting, whatever, that the greatest existential risk to humanity was AI.

[Karen]: Yeah. And so you know like his parenthetical there are other things that we that might actually be more likely to happen like engineered viruses. It's because up until then Alman had been talking just about engineered viruses. And so now that he needs to pivot to speak to an audience of one to Musk. He needs to kind of resolve the contradiction between what he's now elevating as his new central fear to be the same as Musk's new central fear with what he had previously been saying. So that's why he's like I think this is now even though before I said this

[Host]: and are you saying that Sam Alman manipulated Musk because Elon did end up donating a huge amount of money to um open AAI and co-founding it I believe with Sam Alman.

[Karen]: Elon Musk did end up co-ounding it with Altman. And certainly from Musk's perspective, he does feel manipulated because he feels like Alman was engineering his language in a way that would make Musk trust him as a a partner in this endeavor. And of course then Musk is leaves. Um and through some of the documents that came out during the the lawsuit that Musk and Altman are engaged in now, it has become clear that there was a degree to which Musk was actually muscled out a little bit. And so that's why he's left with this very intense personal vendetta against Altman, saying that somehow Alman tricked him into being part of this.

[Host]: And what you're saying here is you're saying that Samman was just mirroring the language that Elon was using to get Elon involved in open open AAI. And later it appears and again there's a legal case taking place now that Sam might have muscled Elon out in some capacity.

[Karen]: Yeah. So we know from the lawsuit and the documents that have come out in the lawsuit that Ilia Sgver who is the chief scientist of OpenAI at the time and Greg Brockman chief technology officer at the time when they were deciding whether or not to maintain OpenAI as a nonprofit because it was originally founded as a nonprofit. They decided okay we need to create a for-profit entity but the question was who should be the CEO of this for-profit entity. Should it be Musk or should it be Alman? because it's they were the two co-chairmen of the nonprofit. And in the emails, it became clear that Ilia and Greg first chose Musk to be the CEO. But through my reporting, I discovered that Altman then appealed personally to Greg Brockman, who was a friend of his that they had known, they had known each other for many years through the Silicon Valley scene, and said, "Don't you think that it would be a little bit dangerous to have Musk be the CEO of this company, this new for-profit entity, because, you know, he's a famous guy. He has a lot of pressures in the world. He could be threatened. He could act erratically. He could be unpredictable. And do we really want a technology that could be super powerful in the future to end up in the hands of this man? And that convinced Greg and Greg then convinced Ilia, you know, I think there's a point here. Do we really want to give this much power to Musk? And that is why Musk then leaves because then they the two switch their allegiances. They say, "Actually, we want Altman to be the CEO." And then Musk is like, "If I'm not CEO, I'm out."

[Host]: I guess this begs the question, what do you think of Sam Orman?

[Karen]: I think he's a very controversial figure. You did an interesting pause. It's a pause where someone tries to select their words. Well, this is this is this is what's so interesting about those interviews is people are extremely polarized on Alman there. No one has in between feelings about him. Either they think he's the greatest tech leader of this generation akin to the Steve Jobs of the modern era or they think that he's really manipulative and an abuser and a liar. And what I realized because I interviewed so many people is it really comes down to what that person's vision of the future is and what their goals are. So if you align with Altman's vision of the future, you're going to think he's the greatest asset ever to have on your side because this man is really persuasive. He's incredible at telling stories. He's incredible at mobilizing capital, at recruiting talent, at getting all the inputs that you need to then make that future happen. But if you don't agree with his vision of the future, then you begin to feel like you're being manipulated by him to support his vision even if you fundamentally don't agree with it. And this is the story especially of Daria Amade, CEO of Enthropic, who was originally an executive at OpenAI. So for people that don't know, Dario now runs anthropic which is the maker of Claude. A lot of people probably are more familiar with Claude. Yeah. And it's one of the biggest competitors to OpenAI. And Amade at the time when he was an ex executive at OpenAI, he thought that Alman was on the same page with him and then over time began to feel that Altman was actually on exactly the opposite page of him and felt that Altman had used Amade's intelligence, capabilities, skills to build things and bring about a vision of the future that he actually fundamentally didn't agree with. And so that's why people end up with this bad taste in their mouths. And so, you know, I've been covering the tech industry for over eight years and covered many companies. I've covered Meta, Google, Microsoft in addition to Open AI. and OpenAI and Altman is it's the only figure that I've seen this degree of polarization with where people cannot decide whether he's the greatest or the worst.

[Host]: So Sam Alman gets kicked off the OpenAI executive team. Did you find out why that happened?

[Karen]: Yeah, there's a scene by scene recounting. I can't remember the exact number of sources, so I don't want to misquote myself, but it was around six or seven people that were directly involved or had spoken to people directly involved in the decision-making process. So, Ilia Satskever is seeing these serious concerns about the way that Altman's behavior is leading to bad research outcomes and poor decision-m at the company. He then approaches a board member, Helen Toner. Ilia, for anyone that doesn't know, is the the co-founder we mentioned earlier. The co-founder of OpenAI we mentioned earlier. Yes. And he kind of does a bit of a sounding board thing to Helen just because Ilia is freaking out. He's like he's been like sitting on this these these concerns for a while and he's like if I tell this to someone, this could also be really bad for me if Alman finds out. And so he asks for a meeting with Toner and in that first meeting he's like re like he barely says a thing. He's just like dancing around trying to figure out hey is this someone that I can maybe trust to divulge more information. And Toner's role and responsibilities at OpenAI were she was a board member. Just a board member. Yeah. And and specifically an independent board member. So opening eye when it was a nonprofit the board was split between people who had a stake financial stake in the company and then people who were fully independent and this was meant to be a structure that would balance the decision-m to be in the benefit of the public interest rather than to be in the benefit of the for-profit entity that opening I then created and Ilia as a non-independent board member was approaching toner as an independent board member her to try and see whether or not she was potentially seeing or hearing the same things that he was about the effect that Alman was having on the company. This then sets off a series of conversations first between Ilia and Helen and then between Amir Moratti and some of the board members. Samir Moratti was at that point the chief technology officer of OpenAI where these two senior leaders essentially through these conversations and through documentation that they're pulling together like email, Slack messages and so forth, they convey to the independent board members, three independent board members, we are very concerned about Altman's leadership like he is creating too much instability at the company and it is like he is the root of the problem. It's not they they they were trying to say to these independent board members like the problem will not be fixed unless Alman is removed because of the way that he's pitting teams against each other and creating this environment where people are unable to trust each other anymore and they're competing rather than collaborating on what's supposed to be this really really important technology.

[Host]: When you say instability, that's a that's quite a vague term. That could mean lots of things. Like instability could mean pushing people hard to work harder, right? What do you mean by instability in spec as specific terms as you can possibly say them?

[Karen]: When chat GBT came out in the world, OpenAI was wholly unprepared. They didn't think that they were launching a gangbusters product. Yeah. They thought they were releasing a research preview that would help them get the data flywheel going, collect a bunch of data from users that would then inform what they thought would be the gang busters product, which was a chatbot using GPT4 and chat GBT was using GPT 3.5. And because of that, there were servers crashing all the time because they they weren't they had to scale their their infrastructure, you know, faster than any company in history. And there were um there were all of these outages. They were trying to also hire faster than any company in history to try and have more personnel there. And they were then sometimes hiring people that they were like, "Actually, we made a mistake. We shouldn't have hired you." So they were firing people left and right. and people were just disappearing off of Slack and that's how their colleagues would learn that they were no longer at the company. And so it was yes like many fast growing companies a very chaotic environment and a particularly chaotic environment because it was extra fast like they had to accelerate more than any other startup. And on top of that mirror Morati and Ilasgiver felt that Alman was making it worse like he was not actually effectively ameliorating the circumstances of the chaos. He was actually sewing more chaos, getting these teams to be more divided. And this is where it's important to understand that the executives and the independent board members, they're all operating under this idea that they're building AGI and that AGI could either be devastating or utopic to humanity. And so it's not yes it's like any other company and no it's not like any other company. You cannot have like in their view you cannot have this degree of chaos as the pressure cooker for creating a technology that they in their conception could make or break the world. And so that is basically what the independent board members also begin to reflect on. They have these conversations amongst themselves where they're like, "Well, based on what we're hearing about Altman's behavior, like if this was an Instacart, would that warrant firing him?" And they concluded, "Maybe not, but this is not Instacart." And that's why they were like, "Well, crap. Maybe this is actually this does rise to the to the bar where we should consider replacing him because we are ultimately building a technology that we think could have transformative impacts either in the positive or negative direction. And so that is what happens. It's like these two executives and then the independent board members also they were hearing other feedback as well from their connections within the company with other people in the industry. At one point, Adam D'Angelo, who is one of the independent board members and the CEO of Kora, uh, which is, you know, start a tech startup in the valley, he is at a party in San Francisco, and he starts to hear some of these rumors that there's something weird about the way that OpenAI has structured its OpenAI startup fund, which was this fund that they the company had created to start investing in other startups. And he realizes they'd never really seen documentation about how the startup fund had been set up from Alman. And finally they get the documents and it turns out that OpenAI startup fund is not OpenAI's startup fund. It's Altman's startup fund. And this was something like one of several experiences that the independent board members were also having where they're like there's something not right about the fact that there continuously are inconsistencies inconsistencies between the way that Altman is portraying what is being done versus what is actually being done. And so when these two executives approach the board or the independent board members, then they're like, "Okay, this lines up with also the experiences that we've been having." And at that point, they then have this series of very intense discussions where they're meeting almost every day talking about should we actually really consider removing Altman? And in the end they conclude, yes, we should. And if we're going to do it, we need to do it quickly. Because they were very concerned that the moment that Alman found out, his persuasive abilities would make it impossible to do. And so they end up firing Altman without telling anyone. You know, they don't talk to any stakeholders to get them on the same page. Microsoft gets a call right before they execute the action saying, "We're going to fire Altman."

[Host]: Why do you call these companies empires of AI? What do you mean by an imperial agenda? What does that term mean?

[Karen]: Empire is the only metaphor that I've ever found to fully encapsulate all of the dimensions of what these companies do and the scale that they operate and what motivates them to do what they do. And there are many parallels that you see between what I call the empires of AI and the empires of old. They lay claim to resources that are not their own in the pursuit of training these models. That's the data of individuals, the intellectual property of artists, writers, and creators. Their land grabbing in order to build these supercomputer facilities for training the next generation models. Second, they exploit an extraordinary amount of labor. They contract hundreds of thousands of workers all around the world including in the US to ultimately make these technologies. We can talk about that more. And they also design their tools to be labor automating so that when the technologies are deployed, it also affects labor rights because it erodess away labor rights. And this is a political choice that they have. Third, they monopolize knowledge production. And so they project this idea that they're the only ones that really understand how the technology works. And so if the public doesn't like it, it's because they don't actually know enough about this technology. They do this to the public. They do this to policy makers. And they've also captured the majority of the scientists that are working on understanding the limitations and capabilities of AI. They are. Yeah. So if most of the climate scientists in the world were bankrolled by fossil fuel companies, do you think we would get an accurate picture of the climate crisis? No. And in the same way they employ and bankroll the AI industry employs and bankrolls most of the AI researchers in the world. So they set the agenda on AI research in soft ways simply by funneling money to their priorities so that only certain types of AI research are produced. But they also will censor researchers when they do not like what the researcher has found.

[Host]: What do we do about it?

[Karen]: Okay. So, one of the analogies that I always use is AI is like the word transportation. Transportation can literally refer to everything from a bicycle to a rocket. And we have nuanced conversations about transportation where we always say we need to transition our transportation towards more uh sustainable options. We need a transition towards you know public transport, electric vehicles. And we don't we don't ever say everyone should get a rocket to do every to serve all of their transportation needs, right? Like we're in Austin. If you use a rocket to fly from Dallas to Austin, like that would just make not no sense. It's just a disproportionate use of resources to get the benefit of getting from point A to point B. This how we should think about AI. So all of the models that we've been talking about, I like to think of them as the rockets of AI. They use an extraordinary amount of resources and they provide benefit some dramatic benefit to some people but they're also exacting an extraordinary cost on a large swath of people because of the like the costs of developing this technology. Why don't we build more bicycles of AI? This is things like deep minds alpha fold which is a system that predicts how proteins will fold based on amino acid sequences. It's really important for accelerating drug discovery for understanding human disease and it won the Nobel Prize in chemistry in 2024. And the reason why it's a bicycle of AI is because you're using small curated data sets. you're just you just have data that has amino acid sequences and protein folding. So that means you need significantly less computational resources to develop the system, which means significantly less energy, which means less emissions, so on and so forth. And you're providing enormous benefit to people.

[Host]: Karen, thank you. Empire of AI: Dreams and Nightmares in Sam Alman's Open AI by Karen How. I'll link it below for anyone that wants to read this book. I highly recommend you do. It's a New York Times bestseller for good reason. Karen, thank you.

[Karen]: Thank you so much, Stephen.