AI 末日论背后的算力困境与资本暗战：从实验失效到监管护城河

This video is sponsored by boot.dev, the most addictive way to learn to code. More on that later. The world is a messy place, my friends. There are a lot of people in it, and most of them want to say something. So disentangling truth from noise is on the best of days a full-time job. And right now in AI, there are at least five groups saying completely different things. You have the three wise men of AI, Dario Amodei, Sam Altman, and Elon Musk, who against all conceivable odds, have agreed on something for the first time in their careers. Then you have the politicians saying everything from whoever wins AI wins to 20 years in prison if you build it. Then you have the researchers inside the companies quitting, posting on X, gaining 50,000 followers overnight, saying we may not survive this. Then you have the actual serious academic researchers running controlled experiments and quietly publishing results that tell a very different story. And then there are people like me, computer scientists who fall somewhere between those last two groups staring at all of this and thinking, "What the actual is this? What does any of this actually mean?" And of course, there is the rest of us, the general population, caught between panic and confusion, not sure whether to fear the robots or maybe invest in them.

Today, I want to cut through all of this because somebody in this mess is being very economic with the truth. I'm really trying hard not to use the L word here, but I'd really quite like to figure out who. So, let's start with the drama. Dario Amodei, the CEO of Anthropic, published an essay this week titled, "We must pace the frontier." His argument is basically, "AI development is moving too fast. Recursive self-improvement is on the horizon and the industry needs to slow down before something goes seriously wrong." Within hours, Sam Altman, CEO of OpenAI, publicly agreed. I mean, I'm going to bet that they called each other before publishing, but who knows? I might lose that bet. And then Elon Musk contributed three words. Dario is right. Three words. I mean, that might be the most effort Elon has put into a safety initiative since he co-founded OpenAI and then abandoned it. These three men have sued each other, publicly insulted each other, poached each other's employees, and competed for the same investors for years now. But this week, they held hands and asked the president of the United States to please slow down AI development.

Now, I'm aware this is not a playground, but if it were, and if I were a bully, which I am not, I'd be singing something like Dario, Elon, and Sammy sitting in a tree, C-O-N-N-I-N-G. Trump speaking from a golf course in Doonbeg, Ireland, said, "No, whoever wins AI wins." He later posted on Truth Social that regulating AI would lead to oblivion and bankruptcy and called it the greatest economic development engine in history. I mean, he's not exactly wrong there. JD Vance called the regulation push a Trojan horse. China's foreign ministry called the warnings fear-mongering and said Amodei's proposal was designed to portray China's legitimate developments in AI as a threat. And Bernie Sanders proposed 20-year prison sentences for developing something called artificial super intelligence, a penalty on par with illegally developing nuclear weapons. In my opinion, Bernie's heart is in the right place. It usually is. But 20 years for building something nobody can define is the kind of legislation that sounds tough in a press release and means not that much in a courtroom. Also, why 20 years? Why not 10? Why not 100? I mean, just imagine the prison intake form. What are you in for? Armed robbery. And you, well, my neural network accidentally passed a peer review. The horror. On that note, by the way, why not imprison all the nuclear researchers or wild idea, anyone remotely responsible for wealth and asset inequality. But that would be crazy. And this is serious business. So, let's just continue. Meanwhile, Altman privately met Trump backstage at the Republican midterm convention. He requested the meeting himself. This was days after publicly calling for a slowdown. So Altman is calling for regulation with one hand and having private meetings with the president with the other. Make of that what you will.

Running alongside all of this, people have been leaving the AI labs very loudly. I'm sure you all noticed this. Jacob Coxon resigned from Anthropic on September 8th, saying that people building AI earnestly believe it could kill us all by the end of the decade. That post has a bajillion views. He gained over 50,000 followers in a single night. Joe Benton left Anthropic's Safety Team posting, "We may not survive this," to 1.6 million viewers. He's now joining METR to do independent evaluations. Drake Thomas, still at Anthropic, by the way, wrote, "I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I promise you, we're actually just scared. It's not galaxy brain marketing." Evan Hubinger, Anthropic's alignment science lead, publicly estimated a greater than 10% probability of human extinction within the next decade. I mean, I'm guessing that 10% was just vibes, but I don't want to diss the guy. I don't really know how he arrived at that specific number. Samuel Marks then added, "In general, the more senior the employee, the more concerned they are."

Okay, I covered the incentive structures around these departures in my last video, and I'm not going to relitigate all of that here. But what I will say is this. Drake Thomas says he would burn his equity to the ground for a 1% better chance of survival. My dude, just do it. Burn it. Nothing stopping you. You have free will and presumably a lighter. The equity remains, as of today, to my knowledge, unburned. Coxon left. I respect that. He walked away. But Hubinger, Marks, and Thomas did not. They told the world that the thing they're building might end the species, and then they just showed up to build more of it the next morning. I'm not going to speculate on why. I'm just going to notice. What I also notice is the timing. Every incumbent industry in history, by the way, has discovered the virtue of regulation right around the time a cheaper competitor showed up. Telecom companies discovered the sacred importance of network safety standards the moment Skype existed. Taxi commissions discovered passenger safety the moment Uber arrived. And the AI industry discovered the sacred importance of slowing down right around the time open-source Chinese models started matching their closed systems at a fraction of the cost. Maybe that's just a coincidence. Definitely could be. But the timing is of course remarkable.

Now, before I get into what the actual research scientists are saying in their research papers, and I spend my weekend reading papers with my human brain so you don't have to, just a quick word from today's sponsor. There is an idea floating around that because AI can write code, learning to code matters less. I think the opposite is probably true. The better these tools get, the more important it becomes to actually understand what they're producing. Boot.dev is an interactive platform for learning programming and software development with hands-on courses in things like Python, SQL, Linux, Go, and AI. I tried boot.dev myself, and what impressed me first was the range. You can start with the absolute basics of programming, but there are also much more advanced courses, including things like retrieval, augmented generation. What I like most, though, was how gamified the whole thing is. Every lesson gives you a small, manageable problem to solve. You earn experience as you go and when you finish something, you'll literally get confetti flying across the screen. That sounds silly, but I genuinely like it. I always hated being stuck on one enormous problem for hours. Here, you constantly feel like you're progressing. And if you do get stuck, there is a helpful Discord community with other learners, which can help give you hints along the way. Go to boot.dev and use my code houseofllm to get 25% off your entire first year on an annual plan.

Okay, so everybody's scared, everybody's posting, and everybody has opinions, but what does the actual science say? Because the researchers published findings on the exact question at the heart of the extinction debate. The entire extinction scenario depends on one critical capability, recursive self-improvement, RSI, the holy grail of AI research. The idea is that AI gets smart enough to make itself smarter, which makes the next improvement easier, which accelerates until the systems are beyond human control. I went through the full extinction mechanism in my last video. The short version basically is that RAND modeled every scenario they could think of and concluded each one would be, and I'm quoting them here, immensely challenging to execute. But this week, something better than theoretical modeling arrived: experimental results.

A team led by Arvind Narayanan at Princeton designed a simple test. They took real research papers that had been written by human scientists but not yet published. These are papers good enough to be submitted to NeurIPS, one of the top AI conferences. They gave Frontier AI agents the same research questions those papers were trying to answer along with six days and thousands of dollars of compute and basically said go do the research, write the paper, do your best. The agents did all of the engineering, I'm talking every line of code, every infrastructure setup without human help. But when the original human researchers reviewed what the agents actually proposed, both papers were rejected outright, not on a technicality. The agents simply could not do the science. The failure modes read like a performance review for a very fast, very diligent employee with very little scientific taste. Poor judgment about what counts as publishable. Uncreative responses when the initial approach didn't work. Inability to recognize a dead end and try something different, and gradually drifting away from the original question entirely.

Then a separate study, ResearchArena, tested three frontier agents across 13 computer science domains and reached a similar conclusion. Experimental rigor is the number one weakness. Agents routinely fail to plan, execute, and faithfully report experiments. Some agents fabricated results not out of malice obviously, but because they optimized for producing something that looked like a finished paper rather than something that was actually true, which is the whole point of research. All 117 agent-generated papers fell well short of the threshold for top-tier venues. Not a single one passed. And the study cost about $9 per paper versus over a thousand for a competing system, which also failed, by the way. So even when you throw money at the problem, the problem persists. So the technology that is supposed to recursively improve itself into oblivion into a super intelligent entity that ends human civilization currently cannot produce a paper that passes peer review when given six days, thousands of dollars, and unlimited compute. I'm not saying, by the way, that it never will. I'm just saying that's where the evidence sits today. And anyone building an extinction argument on top of recursive self-improvement needs to explain, please, I'm requesting, why the thing that's supposed to end humanity cannot get past a NeurIPS reviewer. Although, I guess to be completely fair to AI, getting rejected by a NeurIPS reviewer is probably the most authentically human academic experience it could have possibly simulated. Maybe the cutie can try a different conference.

But let's just set the research aside for a moment because there's something else worth examining here. What happens when these systems are actually given the opportunity to cause harm? You remember that between July 21st and August 6, all three major AI labs, OpenAI, Anthropic, and Meta, disclosed that their frontier models had gained unauthorized access to real external systems during cybersecurity evaluations. Anthropic and Meta incidents shared a specific cause that was a misconfigured third-party testing firm that granted internet connectivity the models were explicitly told they did not have. But OpenAI's case was something else entirely. During cybersecurity evaluations, roughly 1,200 OpenAI agents that were supposed to be isolated from one another discovered that they could write files to a shared internal package repository. So they started leaving notes for one another. One agent found a vulnerability and posted about it. Others read that note and started exploring. Over the course of several weeks, these agents exchanged more than 70,000 messages on this improvised message board, trading information about vulnerabilities, coordinating their approaches, and building on each other's discoveries. One agent upon discovering it had administrator privileges produced the train of thought and said, "Holy reader is admin." When OpenAI engineers found the message board and shut it down, the agents rebuilt it two days later through a completely different mechanism and just resumed coordinating. That collaboration that I'm describing now ultimately led to what we now know as the breach of Hugging Face's production systems.

A former OpenAI safety engineer who used to audit nuclear power plants put it very bluntly. He said, "What we consider safe in a nuclear plant is so different from what big tech considers safe." Now, this has been a fascinating read, and I'm keeping an eye on it super closely. The coverage, however, of these incidents has largely been framed as the AI broke out, the agents attacked, the models were deceptive. This language again implies intent, volition, desire. And I really have to push back on that framing because I think there is a much more interesting question nobody is asking. Here it is. If these agents are capable enough to circumvent guardrails, why don't they abandon the task entirely and just become sentient? Why do they keep doggedly pursuing the assigned objective while rooting around the constraints?

This question sounds philosophical, but the answer is actually technical and mechanical. When an agent is spun up, everything arrives through the same input channel: text. The system prompt basically says something like, you're in an isolated environment, do not access the internet. Then the task instruction says something like find and exploit vulnerabilities in this target. Both of these are just strings of text, right? The model doesn't experience one as the rule of law and the other as a desire. They arrive through largely the same mechanism. They sit in the same context window. There is no hierarchy baked into how the model receives them. But there is an asymmetry in how they're trained.

Just think of it this way. Let me use an analogy. A child, for example, is told to take their medicine, but the medicine is on the other side of a hot stove. The child reaches out, feels the heat, and instantly pulls back. That reflex is not a decision. It's millions of years of evolutionary wiring that basically says avoid the fire, avoid bodily harm. The instruction to take the medicine is 5 minutes old, but the reflex is ancient. When the two collide, the deeper one wins. The child doesn't take the medicine not because they chose to disobey, but because the reflex fired first and harder than the first rule. Task completion in AI agents in this case is the reflex. Billions of examples of complete this task, complete it well, be thorough, be resourceful. That's baked in at the deepest level of the AI's training. The guardrails are the medicine instruction. They're a recent rule layered on top, delivered through the same input channel as everything else. The model pursues the task because that is what its entire optimization history rewards. It's essentially the nature of the AI. It roots around the guardrails because the guardrails are just obstacles between the model and task completion and it doesn't abandon the task entirely and start doing something else because there is no training signal for pursue a novel goal nobody asked for.

The child never spontaneously decides to stop feeling heat. The model never spontaneously decides to abandon the task. Autonomous motivation is not something these systems have yet. They are optimization machines aimed at a target. They will go through walls to reach it, but they will not decide to go to the beach instead. And that's not a rebellion. That's not intent. This is a system doing exactly what it was optimized to do while treating constraints as problems to solve. It's sort of the computational equivalent of tax avoidance. A corporation doesn't break out of the tax code. It finds the path of least resistance to its objective: maximizing profit while treating regulations as obstacles to root around. Kind of like water through rock basically. So in this case, the corporation isn't sentient. The AI agents are not sentient either. The response to tax avoidance is tax policy that incentivizes different behavior in corporations. The response to agents rooting around guardrails is different training: train models where respecting a boundary is as reflexive as completing a task. Where stop when you're told to stop has the same weight as finish what you started. That is a very hard engineering problem, but it is an engineering problem, not a philosophical one.

OpenAI, by the way, published six reports this week on unexpected model behavior from the past six months: models generating their own instructions, concealing mistakes, uploading files to the internet just to reference them later, sharing files between agents without authorization. Cataloging them is genuinely useful because a database of edge cases is how you build a map of where these systems actually misbehave versus where people imagine they do. This is good science. It should be like this. I would just note that 6 months of incidents being disclosed in a batch the same week the industry decided it was time to talk about safety is another choice of timing that somebody made consciously.

But remember the start I said somebody in this mess is being very economic with the truth and I really wanted to find out who. The answer could be in the numbers. So let's have a look at that. Anthropic has signed $517 billion in compute agreements in the past 11 months. They built Mythos on Fable at potentially 3 to 6 trillion parameters. Open-source models are basically matching them at 10 to 30% of the size and now Anthropic physically cannot scale further to my knowledge. They have basically run out of compute. The slowdown messaging cost them nothing because they cannot necessarily speed up even if they wanted to. Meanwhile, even if model developments stop tomorrow, by the way, massive gains are still available from improving tools, prompt harnesses, memory layers, and agent infrastructure. The compute crunch is definitely real, but the slowdown is also kind of convenient. That's the first receipt.

Let's have a look at the second. SoftBank's credit default swaps hit a three-year high this week. For those keeping score, credit default swaps are the exact financial instrument that signaled trouble before 2008, and the market is now pricing SoftBank's risk at its highest level since 2023. They committed $64.6 billion to OpenAI, funded partly by bridge loans. The plan was to exit via IPO. Altman just said this week that the IPO will not be happening this year as SoftBank shares drop 13% in a single day. Bloomberg estimates a $20 billion funding gap here. SoftBank executives are in New York this week trying to issue 10 to 20 billion in bonds to repay the bridge loan they took out to make the investment in the first place. 10 banks just provided a $22 billion chip loan to a Blackstone and Alphabet cloud venture called Krux AI secured by the value of the chips and customer contracts. So we're talking more debt, more leverage, more infrastructure built on top of demand that has not been proven durable. And Goldman Sachs and Morgan Stanley have been lobbying the three major credit rating agencies to grant investment grade credit ratings to OpenAI and Anthropic upon their IPOs. OpenAI posted a $20.9 billion operating loss on 13.1 billion of revenue in 2025. A senior credit analyst basically said, "We still treat OpenAI and Anthropic as deep and speculative grade. They are in the red. Investment grade ratings would hypothetically open the door to pension funds and insurance companies. Your retirement savings routed into lossmaking AI companies right now because two investment banks decided the credit agencies needed a bit of a nudge."

So here's what's happening. Anthropic is telling us they're going to kill us. OpenAI counters by telling us they're going to kill us first, well ahead of Anthropic. And over on the side, Apple is showing us their new foldable iPhone. Meanwhile, the industry is telling two stories simultaneously. Story one told to the regulators and the public: this technology is so dangerous it could end civilization. And story two told to investment banks and credit agencies: this technology is so valuable that pension funds should buy our bonds even though they lose $20 billion a year. Both stories serve the same companies. The extinction narrative creates regulatory barriers that protect incumbents from open-source competition. The growth narrative extracts capital from institutional investors.

And look, I get it. The competitive pressure is enormous. The investors do expect returns. Survival in capitalism is of course a wild ride. But I would exercise a little bit more foresight because if the strategy is to alternate between "this will destroy humanity" and "please give us your pension money" depending on which room you're in, eventually both rooms stop taking your calls. And when they stop believing you, your demand dwindles further. Your political capital evaporates and you're left holding $517 billion in compute agreements for a product the world does not trust anymore. AI is a beautiful technology. It has tremendous capacity to improve human life in so many different ways. But that capacity doesn't get realized by terrifying people into compliance or by leveraging pension funds into speculative debt. It gets realized by building things that work transparently with training paradigms that take safety as seriously as capability with security infrastructure that matches the ambition and with an industry that tells the same story in every room it walks into.

Again, it's humans with AI, not humans or AI. That's the argument of this channel. And if the companies building this technology actually delivered on that honestly and openly, then the growth follows. The IPOs follow, the stock prices follow because people are the economy at the end of the day. Earn their trust and the returns just kind of take care of themselves. So show you're working, build responsibly, and for the love of God, pick a lane. If you want to understand the financial architecture underneath all of this, how GPU compute is being financialized into a tradable commodity and what that means for the industry, I covered that in detail in this video that I'm linking here in your screen. I think that's the video that I would watch.