Welcome to the Matt podcast. Today, I'm excited to welcome back Nathan Benaich, founder of Air Street Capital, to discuss the 2025 edition of his state of AI report, a must-read on where the field really is. We cover a lot including why power is a new bottleneck, reasoning and chain of action robotics and the business reality revenue margins and what it means for builders and investors. Please enjoy this great conversation with Nathan.

Nathan, great to have you back.

Thanks for having me.

The state of AI 2025 is out and as always, it's essential reading for anyone who's serious about understanding AI. This year it's 312 slice of goodness. A bit of a big year in AI.

Every year I try to cut it down a little bit. But this year I just felt like we were sharing it with various sub communities of the AI community. And each time we did that the robotics folks would be like hey it's a little bit light on robotics. Can you add some more? And then we sent it to the bio folks and like why don't you site this paper or that paper? And hence the inflation.

Amazing. All right. So we're certainly not going to cover everything in this conversation. Obviously, as always, the report is available in its entirety for free at stateof.ai. So, we're going to riff on some of the most important topics and ideas in the report, but obviously people can go and check out the report directly for more.

All right. So, starting from the top in the world of research, you mentioned that 2025 was a year reasoning got real. So how far have we come in the last 12 months?

I'd say pretty far. About 12 months ago or so we had I think the very early inklings of it with 01 preview potentially around like this time last year. And that was the first time you had a system that could kind of show its reasoning, show its step-wise process to get to a more complicated answer. And this has generally been the dream in AI for a long time. And and since then to now I would say like the progress is pretty astounding.

One of the areas that that progress has kind of unveiled itself is in mathematics and other verifiable domains where you can like explicitly say yes the system works or doesn't work and you know we saw gold medals on the international math olympiad by a couple labs including openai and deep mind that area probably with uh if you'd asked experts again how long it would have taken would probably been a decade.

Then areas a bit closer to my heart in biology and science we've seen reasoning models uh kind of be used as a as an AI co-scientist so just as a human would reading lots of papers, planning experiments, running the experiments and then doing data analysis and then reformulating their hypothesis as a result. There's examples of uh models doing that in lie of human which is exciting because there's way way too many papers uh to read you know AI people kind of complain that it's like 50,000 papers a year and say in biology and chemistry and physics is probably an order of magnitude more than that.

Um and so uh deep mind has shown that you can integrate this kind of reasoning model to sort of decipher new targets for disease new mechanisms that were actually also proven in a wet lab scenario um postfacto. We've gone from systems that were kind of dumb stoastic paris to now they can solve pretty meaningful challenges that I'd say like even a smart human couldn't.

And uh still in research you talk a little bit in the report or a lot in the report about robotics and this evolution towards uh a system of action or chain of chain of action going from chain of thought to ch of action. What's happening there?

Yeah, I mean the gist is probably two years ago robotics was kind of a dead end. Um openi had disbandled its uh its robot team that was famous for solving the Rubik's cube using uh locomotion with the hand. And so now robotics is probably you know going through a cambrian explosion. There's so much excitement uh and just as how language models informed biology, now language models are also informing robotics.

So what you're referring to here is a sort of reasoning uh process for robots where a system is no longer just perceiving the environment and deciding what to act and sort of acting, but it's uh we've separated those steps. So now you have a reasoning model that looks at a task and tries to plan steps that a robot would need to do to execute that task and then passes that plan over to uh an actuator which goes and actually implements the plan and that's what's called chain of action.

Uh and here the Allen Institute was one of the first to really push this and very swiftly thereafter uh Gemini also followed and and uh we have some companies including Serak that are implying this uh into the real world. So it does genuinely work. It's not just like a research thing. So we think the big moment for robotics is uh upon us because we all collectively have been uh talking about this for a very long time.

Yeah. Yeah.

Well, I'd say it really is upon us in the industrial sector um in logistics and warehousing uh kind of more constrained environments or very repetitive tasks. There is the sort of more holy grail of this kind of embodied um uh humanlike form factor and putting a model on that might even be the same model that's been used in uh in warehousing. Uh, a lot of money is going into that. But my personal bet is I is I think it's going to be the humanoid space is going to look much more like self-driving where we have some very good isolated demos, but the longtail will kill you.

Uh, hopefully not uh literally.

Yeah. Um, and uh and so we we're going to go through many false starts. I think this is just the start.

Okay, great. So a big year in uh robotics and reasoning. Let's move on to the business uh of AI. Uh you mentioned in the report that the business of AI finally caught up with the hype. Uh what caught your attention in terms of fact stats uh in the last 12 months?

Yeah, a couple of them. Uh again like where we came from one or two years ago was just tons of money going into this segment building models lot of usage but not clear where the revenue would come from. I think it was maybe open was making $50 million or something two years ago. it was very unclear how they would ever hit like billions of revenue.

Um, and nowadays I think if you sum sort of the top 20 or so uh major AI companies from the labs to the most popular kind of vertical applications, you know, across them they're making tens of billions of dollars of revenue. Um, you can look at the smaller scale companies which you know are growing from zero to 20 million or 20 million plus. uh as a group they generally grow about 60% faster on a quarterly basis than nonAI companies.

Um we've all seen like the famous charts about ARR or non ARR it's unclear. Uh but uh you know very steep curves for various coding companies. Um and perhaps most interestingly across a segment of 43,000 or so uh US customers we work with ramp to show that retention of uh subscriptions on AI products across this customer set has really improved marketkedly since 2022. Around 2022 was around the 50% after 12 months. uh and now in 25 it's hitting around 80%.

Um and the second stat in that analysis that was interesting was the total spend on AI products uh per customer kind of went up from $35,000 or so uh maybe two years ago. Now it's around half a million and it's predicted to hit a million dollars next year.

And you mentioned in your ramp stats 44% of US businesses now pay for AI tools. So they pay more but there's there's there's a ton of businesses using it.

Yeah. Exactly. And and there might be some sampling bias slightly to you know what kind of companies uh use ramp in the first place. Yeah. So slightly more modern you know tech forward companies but a leading indicator I think of where things could go.

And then you had your own survey right of 1200 AI practitioners. Yeah. Yeah. And what did that say?

Yeah that was I I was surprised. Uh obviously bias is more towards uh you know pretty well educated US European professionals. Um, a lot of people in there have at least undergrad, master's degrees, maybe even more. Um, but it's like 95% of people use AI in their personal life and in their professional life. U, about 76% of people pay out of their own pocket for it. It's like 10% of people pay more than 200 bucks a month for it.

Uh, and then looking at the organizations that they work at, it's like 70% of those organizations are spending a ton more or more than they did in the past on AI. the reasons that they gave for what why they might not be spending more or what problems they have. It's like all the classic like new technology stuff like it's a bit hard to configure. It's uh you know I haven't really figured out the ROI yet because I need to do more customization. There's like some data privacy issues that I have and I think all these things are kind of solvable like it's not rocket science how to solve these things.

Yeah, it feels like we very much live like this. You're in the world of shadow AI uh in companies where I mean to reconcile it's imperfect but reconcile your two stats. 44% of businesses uh use AI yet 95% of of people individually use AI. So there's a bunch of people as as you as you're alluding to that use AI at work without being officially authorized.

Yeah. And I think there's still a big like education gap. I mean there was a a study banded around a couple of weeks ago where you know it said 95% of businesses like get no value from AI.

Very very controversial survey.

Yeah. Yeah. But I think that there it turned out it was like not the models that were bad. It's like the implementations of them were not great. Uh so I think there's just a big education gap for how how you should like you know update your view of your own uh day-to-day tasks and and apply what capabilities you know models have and then uh and think about like hey should I be doing this task myself or can I farm it out to a model and and there's definitely a delta of companies that really get this done well and others that are like basically clueless.

What do you make of the margin debate uh as an investor and intory analyst? Maybe maybe uh recap uh what what that debate is and then what do you think about it at a high level?

Basically the the margin problem is um for many many customers of uh large model companies their margins are basically dictated by how much the model vendor charges them for. Uh now here there's some issues because right now model vendors are charging the same amount per token. So if you're a hedge fund analyst and I'm a student you know your use case is clearly more financially valuable than mine but we pay the same amount for the token assuming we use the same model.

There are some use cases that are more reasoning heavy towards what we discussed before and they consume a ton of tokens and the pricing that a customer uh pays for that product might not be fit for the amount of work the AI system is doing. Um and so there are cases where um these kind of vertical products are making margin gross margins of like 30%. And sometimes they get worse with scale uh because you do have some edge users that like really pump the system and you can't like price discriminate or they haven't managed to.

And then you have uh some segment of of um of model users that don't that have a both a paid plan and a free plan. And it's not clear whether they include uh the costs of running the free plan in their gross margin. So they sort of just look at their paid customers. Uh there's you know some creative accounting standards going on there.

Uh and then you have the model vendors themselves and what is their margin. Uh and I think what's interesting in the last year is you've seen um CEOs of these model companies say hey if we if we basically look at um sort of in in financial analysis terms like a layer cake of like what revenue is generated by each vintage of model over time um it looks like prior models are uh profitable. So the amount of money we've spent to build them is less than the amount of money that we've generated with them over time assuming a certain margin of inference cost.

And so so really these these labs are like not not profitable because vastly more resources going into developing next generation systems than than the prior ones. Um but as you and I both know like there are companies here that are making very very good margins on uh serving uh their AI systems like 70 80 sometimes 90% depending on the modality. Um and so like with everything the average number sucks. Um, but like when you look at the best companies, it's really good.

And just to drive it home, the the the the companies using those models, we're talking about the, you know, in part the the all the what used to be known as thin wrapper. So the vendors that happened to be powered by those models. So uh the the cursors, the wind surfs yeah Reddit like all and and all the whatever legal financial AI um startups as a as as as examples.

Um the other big debate uh in the business of AI of course uh is the bubble uh question. What's your what's your take? Are we in an AI bubble? Are we not in an AI bubble?

Yeah. I think like with most things in in markets, there are probably localized bubbles all over the place. And I think at a at a high level, what's interesting in terms of vibes and who's calling bubbles and who's not like the finance crowd in New York is definitely talking about bubbles a lot more than what we're talking about in San Francisco where their their view is like this is the golden era of AI and a lot of things are working. We have so much more to to do.

Uh you know, compute buildouts are enabling us to experiment a lot faster. uh you know this huge flood of like talent that's built the consumer internet and cloud computing is moving into AI and with that is bringing a lot of optimization techniques and knowledge that AI researchers didn't have when they built the first generations of Chad GBPT etc.

But I think you you can't ignore the fact that the the sums of money going into this industry are truly gargantuan. Um you know like 500 billion to build uh Stargate and then uh you know couple hundred billion here, a couple hundred billion there. Like pretty soon it's real money.

And then the um and then like the circularity of these deals is like is interesting. Uh of course Nvidia is at the center of this and it has incentives to use its its balance sheet to sort of spin the wheel faster. Uh and then perhaps more concerningly, you have this sort of offloading of of debt um from big companies. For example, Meta that raises tens of billions of dollars to fuel it data center ambitions, but that doesn't sit on Meta's balance sheet.

Some of this is like catnip to financial engineers. Um but uh but yeah, it rests on certain assumptions that everything is going to keep going up and to the right and that rates don't materially change. Um, and just given how like uh I suppose precarious various aspects of the economy are and how like sensitive geopolitics are, things can flip like quite quickly. Um, but I think that's like the major risk. The the risk I'm less worried about is the stuff doesn't work because I think it does work.

So it's a more question of uh timing to play it back that the supply uh phase of the market is met by an equally strong or hopefully stronger demand side.

Yeah, there's that and then just the just the the nuances of the like the terms on the debt and what trigger events are whether rates get repriced and then you know like investors behave very differently once rates change and and um and flows of money can be quite like violent.

It's interesting what you're saying about the, you know, the dichotomy between um Wall Street and the the West Coast. Um also because uh when you think about it, there's actually not that many fuel play AI companies in public markets, right? A lot of the action is happening in private markets. So effectively, if you're a Wall Street/hedge fund investor, you you you invest in Nvidia, you invest in the Mag 7. That's pretty much it, right? Palanteer C3 AI

maybe you buy soft bank for its position open AI yeah pretty much like a lot of it is indirect or you invest in power and energy or like related players coreweave I guess but it's it's very it's very small so it feels like there that tension as well

All right, let's uh switch to the physical reality that this whole uh stack sits on. So infrastructure, data centers, uh energy. You you you mentioned in the deck that uh power uh has become the new bottleneck. What is your sense of the state of play in the energy procurement game?

The the biggest stat for me is one gigawatt of a data center for AI basically costs $50 billion in capex. Um and uh on an annual running basis it costs between like another 8 to nine 8 to9 to maybe even 11 billion to run. And so uh when you have just you know casually a 10 gawatt data center uh that's like a lot of money.

Um and um and so one of the problems is like where does this energy come from? Uh you know traditionally it would be from I don't know coal uh or natural gas um potentially solar or ideally at some point in the future nuclear and what we're seeing is it right now companies are trying to do deals with anybody who has any capacity. So we chronicle some deals with uh future uh nuclear uh you know reactor companies then that would take maybe a decade or two decades to deliver.

Um you know famously yeah that's Google inking a PPA deal. Uh with CFS to buy 200 megawatt of electricity from a planned fusion plants. So the the the plan does not exist.

It does not exist. Yeah. And then last year we documented the sort of uh restarting of uh 3M island. Um the nuclear facility which was controversial in the past. Um and um and then uh in the short term what many uh GPU uh data centers are getting powered on is is just uh gas turbines and because these can get set up a lot faster but that has other issues like they're super loud um and there's demand outside of the US for these things and so now basically US tech companies are paying more to repatriate like uh some of the supply that should have been shipped abroad.

the the other issue is the the grid and like to what degree the the grid can even tolerate data centers getting plugged into it. Now obviously like these turbines are off-grid. Uh so it has some advantages but uh in China for example we do some analysis between like eur between uh the US and China with regards to energy and China has a lot more like slack in its system to plug in um for any unpredicted uh demands in energy. Uh the UK famously cannot really tolerate more data centers on its grid.

所有这些因素共同推动了数据中心向能源丰富的国家“离岸”迁移，无论是阿联酋，甚至是挪威。随之而来的是大量的地缘政治问题：这些国家是你的朋友还是潜在的敌人？你如何确保无论政府更迭，都能持续获得能源？我们发展到今天这个地步真是太疯狂了——我们只是想要一个能在电脑上运行的 AI，但为了实现这一点，你需要如此多更强大的系统协同工作。

Matt Turck: 是的，你说话的时候我正在找那张幻灯片，特别是关于美国和中国的对比，我们谈论的是巨大的差异。如果我没读错的话，2024年美国新增的发电容量是48.6吉瓦，而中国是429吉瓦。

Nathan Benaich: 另一个有趣的事情是，在美国，或者实际上在国际上，那些适合托管数据中心的州，因为能源充足，通常都极其干燥。我们在报告中也记录了冷却这些数据中心所需的水量。所以，如果你的州非常干燥，水从哪里来？这是否会影响到需要用水的人口？然后你还有整个水循环利用的问题，这可能会导致质量差的水被循环到供水系统中。

Matt Turck: 所以这一切的可持续性方面似乎极其重要，但讨论得却不够。至少这是我的看法。这个看法对吗？人们真的关心并为可持续性方面做些什么吗？

Nathan Benaich: 嗯，一两年前，大公司确实承诺到2030年实现绿色环保。但一旦他们开始与核能公司和各种能源供应商签订数据中心协议，所有这些承诺基本上都被抛诸脑后了。所以，看起来他们可能在乎，但实现 AI 的企业优先事项远远超过了环境限制。这就是已经发生的事情。

但我认为，回到政治层面，并非所有人都对此感到满意。特别是“邻避主义”（NIMBYism, Not In My Backyard）的增长，即“不要建在我家后院”。我确实认为人们普遍不希望自家后院有个数据中心。我认为这将推动未来的一些政治议程，无论是在美国还是其他国家。所以，是的，人们确实关心环保。公司们某种程度上忽略了这一点，但我认为这个问题会回来的。

芯片霸主 Nvidia 与多极化未来

Matt Turck: 如果我们谈论基础设施，显然我们必须谈论 Nvidia。感觉过去12个月对 Nvidia 来说又是一个非凡的时期。你认为 Nvidia 会继续作为市场上无可争议的头号玩家遥遥领先，还是我们迟早会进入一个多芯片并存的世界？

Nathan Benaich: 我想会是95%对5%的局面。是的，完全正确。 contextualize 一下，去年我们做执行摘要时，我们写 Nvidia 首次突破1万亿美元市值，现在我们不得不把它改成4万亿美元。

我们每年都会分析所有开源 AI 研究论文，大约有49000篇左右。然后，我们通过程序化方式确定这些论文中使用了哪些芯片组。这样我们就知道，一个 AI 研究员在研究某个新模型时，在他们的实验设置中会说“我们在某某芯片上训练了模型 X 个 GPU 小时”。如果你做这个分析，你基本上会发现90%的论文都使用了 Nvidia 的芯片。

在同样的分析中，我们确实发现 AMD 的身影开始出现，虽然还很少。Apple Silicon 也是如此，我想这只是因为 MacBook 电脑变得越来越好，人们开始在本地电脑上进行训练和实验。

Matt Turck: Broadcom 也在某种程度上复活了，对吧？

Nathan Benaich: 是的，完全正确。大约十年前，他们收购了一家公司，现在这个内部团队正在为谷歌的 TPU（Tensor Processing Unit: 张量处理单元）制造定制的 ASIC（Application-Specific Integrated Circuits: 专用集成电路）。最近，他们也宣布与 OpenAI 达成协议，为其制造定制芯片。

从宏观上看，Broadcom 的崛起很有趣，它表明，当你在硬件上运行的神经网络或其他 AI 系统的性质仍在快速变化时，GPU 长期以来一直是主导芯片组。但一旦你达到一个临界点，即某个架构趋于稳定、能够产生收入、并且开发者开始围绕它工作并确认它的地位时，你就可以转向制造定制芯片，以从该架构中榨取最大价值。所以，Broadcom 的崛起基本上告诉你，有强大的力量在说：Transformer 就是未来。

但归根结底，我们也在看，如果你想投资芯片公司，你的钱怎么花才最划算。在报告的图表中，我们比较了 Nvidia 的六个主要竞争者，基本上是说，如果你在这些公司宣布私募融资的那天购买 Nvidia 的股票，你的 Nvidia 股票价值会是多少，而投资这些公司的价值又是多少？如果我没记错的话，结果基本上是 Nvidia 带来了12倍的回报，而这些竞争对手是2倍。这个趋势和去年大致相同。所以我认为，与 Nvidia 对赌是一件有点困难的事情。

主权 AI 与开源的博弈

Matt Turck: 让我们深入探讨一下你几分钟前提到的一个话题——主权 AI。你看到了哪些动向？这似乎是今年的一个大主题。你提到了 OpenAI 在挪威、印度和阿联酋的动作。那个世界正在发生什么？

Nathan Benaich: 是的。主权 AI 的理念是，民族国家希望能够掌控自己在 AI 方面的命运。这包括运行模型、训练模型、拥有芯片。这基本上是因为民族国家希望控制自己的能源、货币、基础设施，而 AI 被认为与这些类别同等重要。

自从白宫在一月份宣布了5000亿美元的计划以来，各个国家纷纷效仿，宣布了自己数十亿美元的倡议。Nvidia 甚至开始将此作为其业务的一个新产品线进行营销，目前该业务已产生约200亿美元的价值。所以这是真金白银。

他们正在与各个国家建立合作伙伴关系，在当地提供数据中心。理论上，这应该能让各国放心，他们对 AI 的访问不会被切断。这就是其理念。我个人认为，这更多的是一种政治议程的契合。特别是在美国，这实际上关乎再工业化，即关键产业的回流和制造业的建设，我认为这也是为什么这些 AI 数据中心被重新命名为“AI 工厂”的原因之一。