I think that the AI company CEOs understand that they're on the path of building wildly smarter than human systems, like super intelligent AI systems. They understand that we don't really have a like clear thought through plan for how to manage the risks from that. I wouldn't say super intelligence is bad. I would say it's dangerous. It seems pretty likely that you end up with AI takeover as a result of building super intelligence. And so you quickly get from the AIs that are fully automating R&D into AI that are like quite superhuman at everything. It turned out that somewhere along this transition, at some point in 2029, you went from AIs that were kind of misaligned and reward hacky and sloppy and weren't really trying to do the right thing to AIs that are like competently scheming against you and want to take over. Then those AIs take over.

Hi, I'm Matt Turk, partner at Firstark. Welcome back to the Mad Podcast. My guest today is Ryan Greenblat, chief scientist at Redwood Research. Ryan is the researcher who first caught an AI faking its own alignment back in 2024. And today is one of the authors of AI 2040, Plan A, the most detailed blueprint anyone has written for how the US and China can avoid a reckless race to super intelligence and an AI takeover that could start as early as 2029. As a fair warning, this is a fascinating episode that gets a little dark and the last 10 minutes or so are probably the darkest. The good news, subscribing to this channel if you like the episode is still a decision the humans get to make. So I would exercise that right while it still lasts. Please enjoy this great conversation with Ryan Greenblat.

Yeah. So I think that the AI company CEOs understand that they're on the path of building wildly smarter than human systems like super intelligent AI systems. They understand that we don't really have a like clear thought through plan for how to manage the risks from that. You know how to ensure these AIs don't take over, how to ensure they remain under our control. and they understand that this would this is a decent chance of a unprecedented concentration of power at least in the absence of very active efforts by the people who end up with that power to redistribute it. I think that yeah I mean we say some sort of more precise thing in the like why did we write this section. I think that overall my sense is that the company CEOs do legitimately think that the thing they're doing is very risky or at least the thing that these companies are doing. I think that they have a mix of views for why they're doing this. um where some of it is that they think they're like, you know, better than the next guy or think it's good if there's multiple companies or multiple things. I think that they vary a bit on how much concentration of power they think AI will yield or maybe just haven't really thought this through very carefully. I think that there's more sort of public statements from AI company CEOs on just there being huge risks um than on specifically the risk that AI ends up with the power in the hands of the few rather than being as distributed as it is now which is not you know arbitrarily distributed now but but yeah uh so overall I think I think there's like this seems pretty consistent with what they've said publicly but I think that they're sort of imagining a world where maybe uh AI concentrates power massively but the people with the power end up deciding to redistribute it. Um, but they could have totally taken over the world.

Yes. And uh, since you published uh, AI 2040, there's a couple of things that happened that seem to be going uh, in the general direction of what you recommend. So specifically, open eye paused its Astra model over safety u you know following the hugging face incident and then 1,200 insiders um published that letter a few weeks ago including Dario asking the government for slowdown tools curious about what you make of it. Is that is that what you are recommending that's starting to happen?

Yeah, I mean I think these are these are good steps. I mean, I think I can't it's harder for me to say as much about what's going on with like you know OpenAI uh pausing training and and not deploying Astra and how where exact like what exactly the motives for that are like what's exactly going on um there. But I think that overall like these seem like good steps. I think my sense is that like the employees at these companies are pretty freaked out about how things are going and don't think that we're like necessarily on track to handle all these problems in time given how fast recent progress has been. And I think they you know recognize that which is why they, you know, signed the open letter.

And just to verbalize uh the question early in this conversation uh what is so bad about super intelligence? Obviously, uh there's a lot of talk about scientific progress and curing cancer. Um and your uh document effectively recommends pausing the rates towards super intelligence. So why why is it so bad?

Yeah, I wouldn't say super intelligence is bad. I would say it's dangerous. Like it's a very dangerous thing to create. So the most straightforward story for why um it's dangerous is that it seems pretty likely that on sort of a trajectory similar to the trajectory we seem to be finding ourselves on um you end up with AI takeover as a result of building super intelligence because the AIs are in a position where they can take over due to being highly capable, widely deployed and um you know uh building basically uh the huge amount of industrial capacity potentially. Um and we could talk more about what a takeover would look like. And then if they're in the position where they could take over, then there's a question of like would they want to or how would the motives shake out? And it looks like we don't really have that much control over the motivations of AI. And it seems like that problem gets harder as they're, you know, much more capable and built via a process where AIs are automating AI R&D and we maybe are losing our understanding of how that process works. There's sort of this like we don't necessarily control the technology miscellane. Another concern is that historically like you know um at least in recent times uh the distribution of power among humans has been reasonably distributed though not necessarily like super super distributed due to you know people being able to work for money like labor like you know the reason why like many different people

Speaker A: have some power and are cut into the world is to substantial degree because we have the ability to work and I think that AI means that basically all of the key financial assets or might just be entirely capital. Like there's no there's no um you know you human labor would have very little value left. Um I think it's unclear exactly how that works because it depends on like people having intrinsic preferences to employ specifically humans even if an AI could do their job just like totally way better. Um, and that means both that, you know, there might be a natural effect to where power gets very concentrated. And in the same way that like it's easier to be a brutal dictatorship if your money comes from oil instead of from a productive sort of broadly distributed economy, it might be much easier to sort of for someone to consolidate power a lot. if you know the economy is running on AI and machines rather than on humans and in addition to that there's some concern which is like it might make coups way easier because right now for to do a coup in many countries you need the support of a broad base of people and it's possible to build checks and balances whereas if you end up in a system where basically AIs are running any anything if anyone sort of either puts like sort of secret objectives into that AI or has overt control of those AIs then they could sort of just directly take over and there's just like a huge threat to you know, democracy is the US, whatever, because you know, you'd be in that position.

Speaker A: and then the third thing I would say is just like AI might yield extremely extremely rapid technological progress, which is more rapid than sort of our wisdom grows to match it. like there might just be all kinds of things that happen as a result of very fast tech progress that we just you know don't necessarily we're not necessarily going to be able to handle on time in part because the AIs might be better at doing things than they are at like you know thinking carefully about how to do things like they seem much better at sort of accomplishing hard results and easy to verify results than they are at sort of contextualizing things understanding the broader picture understanding what would be um you know a good or bad choice in the broader context and so you could worry that uh this causes problems where examples might be like there's some like dual use offense dominant technologies so like bioweapons you might just worry that like the sort of um we just come across some technologies that are very dangerous and there's some concerns around AI being very superhuman at persuasion and this sort of destabilizing society and all these seem like things that I think we like could deal with given time but like if things go very fast and we don't get to necessarily pick the order in which these technologies develop it seems like we might be in trouble.

Speaker A: and also central to the argument that leads to speed is uh what you mentioned about AI automating its own creation which takes us into the territory of um RSI recursive selfimprovement. So you were just on a darch um and had a a great conversation about what is RSI and how it can go wrong. So let's not um redo this. Uh but um still as a as a TLDDR, there were a couple of parts to that discussion that I found particularly interesting. In particular, there was this argument that uh RSI may be very good at automating the process of creating the next generation of AI. but it may or may not have the uh the kind of intuition uh that one needs for scientific breakthrough and truly novel ideas. So what what's a re rebuttal to that argument?

Speaker B: Yeah. So um the way I would put this argument is AIS might be very good at sort of the more mechanistic or nitty-gritty parts of AI development like writing code, running experiments, but not as good at sort of the broader conceptual leaps. So first I would say that like AIs seem significantly better at engineering and grungy stuff and sort of just keeping trying than they seem to be at conceptual breakthroughs but their ability to do sort of these breakthroughs especially in easy to verify domains are improving and like you know one example um is like their ability in math but even in like you know ML their taste has been improving. I think it continues to improve um and so it's not it's not so clear to me that this will lag super far behind. Another thing is that uh you can measure how good these AIs are at intuition or research taste or having breakthroughs especially in domains that are relatively easier to verify and if you can measure it then you can take your grungy AI labor or even just your human laborers and try to optimize that. So, you know, if it can be measured, it can be hill climbed on, very roughly speaking at least. And I think this is a case where you could just hill climb on how good the AIs are at making these sorts of breakthroughs in a wide variety of different settings and then I expect that would transfer to making the actual breakthroughs, right? So, you could have like, you know, breakthrough bench or whatever. And and my sense is that like the transfer doesn't look so bad and that AIs are able to spin up very quickly in some specific R&D domain and then get some intuition for what the best approaches are and iterate there. And you know, right now the approaches they tend to focus on are ones that are relatively sort of straightforward and are more just like, you know, grungy iteration, but that that they're increasingly being better at doing sort of the broader um experiment design and discovery and also having good ideas.

Speaker A: Yep. And you also make the argument that um for purposes purposes of a potential discontinuity whether it transfers or whether it generalizes may not even be a question and that if it was super good just at the industrial part of accelerating AI that would be enough. Is that fair? Yeah, I would say that like if AIs could just automate A&D and automate like sort of the industrial process of building more computers, then you could quickly end up in a in a process where sort of like robots are building robots and the whole world is greatly transformed and that very quickly can get you to a point where AIs could take over, you know, the economy has been radically changed even if it's hard to train AIs at some other tasks. But I do think that my sense is that once the AIs are, you know, once the situation is like there are robots building robots that build computers and the full feedback loop is closed closed at that level. My sense is by that point the AIS will be good at, you know, basically all human work or at least not not too deep into that. Maybe there's some period where, you know, robotics is a big deal, but the AIs can't quite automate a bunch of or like there's a bunch of stuff they can't automate. But it it seems to me like that that's how it's going to go. But just in general, sort of automating R&D seems like it's enough to radically transform the world. And if you look at sort of why is human why why why is like humanity a big deal? Like why why have we been able to accomplish so much collectively a lot of it is because of just like you know having technology having organizations being able to organize ourselves in various ways and accomplish things in the world. And just if AI just had sort of this sort of industrial capacity combined with the ability to develop more capable AI systems that that that seems like it's quite in and of itself is quite far or like quite extreme.

Speaker A: And you know looking at Twitter today as we're recording this which is um August 24th there's plenty of uh rumors about um SSI coming out with their first model in the next couple of days potentially with what could be a breakthrough in continual learning. And I'm I'm I'm curious whether there's an overlap between RSI and continual learning, whether continual learning would would feed into RSIs or is that orthogonal?

Speaker B: Yeah, I would say that um continual learning and RSI strike me as mostly orthogonal where by RSI I mean something like the process

AI的自主加速與持續學習

Speaker A: 人工智能本身正在通過各種機制，以及一些正在持續進行的方面，從根本上加速AI的發展，如果它們能夠完全自動化研發（AR&D），那將會更為引人注目，儘管我們已經知道，現在已經有相當多的事情正在發生。

Speaker A: 但它可能會為遞歸（recursion）提供動力，對吧？如果同一個模型能夠不斷學習世界，那麼它自動化研發的能力可能在不需要重新訓練整個新模型的情況下會更大。

Speaker A: 是的。所以，我認為存在一種回饋循環，那就是AI非常聰明。因此，它們在即時學習和不斷學習方面非常快，而且這種學習會被重新整合，這讓它們在進行研發方面變得更加出色，這意味著它們可能在學習上會更快，但即使把這放在一邊，它們可能在研發方面會更出色，從而可以創造出另一個AI，這個AI在持續學習方面會更出色。我認為在持續學習和人類式的持續學習之間可能存在一些連續體，雖然連續體這個詞有點鬆散，但可能存在一種譜系，從人類式的持續學習到更多像在強化學習（RL）環境中訓練，你可能會發現AI不斷地在新的RL環境或新的訓練數據上迭代，以一種模糊地類似於人類的持續學習，但它也是不同的，這可能就是回饋循環的一部分。所以，即使你必須重新訓練才能獲得持續學習，也有各種重新訓練的版本你可以每天都做，原則上沒有什麼能阻止你不斷地進行少量重新訓練。

研發自動化的時間表預測

Speaker A: 那麼，你對R&D完全自動化的時間表有什麼最新的預測嗎？

Speaker A: 是的。嗯，所以也許我對R&D完全自動化的時間表的中位數預測是，我意思是，即使人類離開了這個畫面，事情也不會慢到那樣，可能到2030年底，或者可能到2031年初，這些數字的精確度不是那麼高，但有一些我不知道，你知道，不是那麼穩定，這些數字會波動，但這將是我的中位數猜測。但我想那可能是我計劃的中心情景，我的第35百分位可能是在2028年底或2029年初，我認為這非常非常可能。我認為這看起來非常合理。而且我認為如果我只是外推目前的軌跡，看起來就像你獲得了那個軌跡。嗯，而我認為我認為我不是中位數的原因是，有一些因素可能會推動事情向後。也許我沒有看到一些關鍵的瓶頸。也許有一些我認為將能克服某些障礙的東西實際上不會起作用。或者可能會出現重大的政府放緩，因為人們對這項技術感到恐慌，這似乎是合理的。所以，我會推遲。但我認為從我建議人們計劃的角度來看，儘管這正在發生，我建議人們計劃，儘管研發完全自動化可能從2029年初開始，甚至可能更早，然後到2028年研發和開發變得相當自動化，可能更早，到人類在這個畫面中變得不那麼重要。我不知道，我應該說我不能說可能，那是非常核心的，我認為人類在2028年初就已經不那麼重要了，而且就目前而言，研發已經相當自動化了。

政策干預的時機與社會認知

Speaker A: 顯然，2028年初就像明天早上一樣。有沒有什麼情況是所有這些都已經太晚了？

Speaker A: 是的。嗯，這取決於你對「太晚」的定義，以及所有這些意味著什麼。我擔心的是，特別是AI 2040計劃A假設了太多的有效政府時間，或者假設政府有比實際時間更多的時間來完成所有這些行動。我認為實際的計劃應該是更為鬆散一些，你知道，組織得更差一些，而且更快，只是因為我們實際上沒有時間做一些相當詳盡的事情。我不確定這一點。我認為有許多不同的選擇，但總體而言，我會說，也許對於一些干預來說，可能已經太晚了。我認為如果遵循正常的關閉時間表，各種政策窗口會存在。話雖如此，我認為在美國有很長一段歷史，在危機時期，事情可以發生得快得多，而且有很多不同的槓桿。所以，如果我們處於一個所有人都需要採取這項可能非常迅速發生的特定行動的位置，但我們可能不會達到一個有那麼多共識的位置，而且政府可能沒有在相關的時間框架內充分監控AI或意識到AI。對吧？所以採用是滯後的。我認為人們對AI能做什麼的理解落後於實際可能做到的事情。我的猜測是，如果你給人們一個關於AI能做什麼和不能做的小測驗，他們會給出答案，或者如果你給國會的人一個這樣的測驗，他們會給出那些比一兩年前更真實的答案，而今天也是真實的。而且他們可能低估了當時的能力。我認為在國會中，大多數人不會正確回答過去兩個月最大的數學突破是由AI帶來的，這是我理解的，至少如果我們以規模來衡量，而不是衡量有多少見解，但就人們會說結果的類型而言。

關於AI安全與個人經歷

Speaker A: 我們將在幾分鐘後回到2040年。嗯，但我還是想快速過一下你和你的故事。什麼首先讓你對AI安全產生了興趣？

Speaker A: 是的。嗯，我在大學的二年級時，我有點孤單，因為是新冠疫情，我聽了很多播客，我一直在思考我該對我的生活做什麼。我最終通過某條有點曲折的道路，想通我應該對幫助他人和做利他主義更有興趣，而不是當時的樣子。我應該非常專注於我如何讓別人的生活盡可能美好，讓事情盡可能順利，然後從那裡我...

职业发展方向的转变与AI控制研究

Speaker A: 考虑了一系列不同的路线，一直在研究各种不同的事情和可能性，最终决定我能为我的职业和生活做最好的事情是尝试让AI变得更好，特别是避免AI接管，但更一般地说，你知道，尝试让它变得更好，然后我申请了许多地方，我最终在Redwood工作了，这大约是在五年前。嗯，我一直在那里工作，我在那里做过很多不同的工作，你知道，这个领域自从那时以来真的发展了很多，因为你知道，五年前还是像GPT-3.5还没发布。我记得像Text-davinci-03，那是GPT-3.5的第一个公开版本出来的时候，这个领域非常不同，我认为在GPT-4之后有一个后GPT-4时代，然后是更近期的像代码代理的广泛采用时代，然后可能很快就会有更多的时代，事情发展得相当快，开发速度你知道，只是模型的发布进展，比以前疯狂得多。是的，多年来阅读Redwood的研究资料，似乎从主要关注可解释性演变成对AI控制的更多关注。这是公平的吗？

Speaker B: 是的，这是公平的。所以我会说，我们作为一个组织的发展轨迹是，当我加入这个组织时，我刚完成了一个关于对抗性训练的项目，我对进入可解释性和我们所说的模型内部工作感兴趣，比如我们能否利用我们对这些模型的白盒访问的优势，做一些比仅仅是提示和训练的朴素方法更好的一些事情，当我们试图使这些模型理解它们的动机，知道发生了什么。我们探索了那个领域一段时间，然后出于多种原因决定那比我们最初希望的要不太有希望，并决定转向其他事情。我们不久之后转向的另一件事是AI控制。嗯，这是那个想法，也许防止AI能够完成有问题的任务，或者基本上让AI不能给您带来巨大的问题，即使AI想要这样做，你知道，有很多不同的故事说明为什么这是个好主意，但基本上这个想法是，可能存在一个中间时期，我可以说我们目前正处于那个时期，AI可能已经有能力造成至少中度的问题，然后我认为越来越有能力造成相当大的问题，但它们不一定有能力，如果我们非常努力地尝试实施各种安全措施，这些AI仍然能够绕过它们。所以我们可以实施监控，我们可以实施各种安全控制，我们可以更好地了解代理做了什么，我们可以更好地对它们采取的行动进行审查和监督，到一定程度，即使眼睛真的非常不一致，对它们来说也可能很难做出任何非常糟糕的事情。嗯，而且我不认为我们现在已经到了那个阶段。我感觉目前在AI控制方面，从公司所做的事情来看，情况并不像现在看起来那么令人印象深刻。但我认为有一个我们一直在研究的研究领域，它似乎非常有前景，可以实现，尽管它需要很多努力。它需要一些成本，但我认为我们正在看到对这个的兴趣日益增加。例如，OpenAI最近表示他们将宣布监控他们内部流量的更大比例。根据他们的博客文章，这在计算方面是一个严重的成本。对我来说很难确切知道它到底有多贵，或者说，我们没有足够的信息来有一个很好的感觉，但这表明至少公司愿意在这一领域投入大量的计算资源。而且我认为我们可以用那做很多事情，比如让我们的检测偏差和防止偏差造成大问题更有可能。然后，除了研究AI控制之外，我们还在研究各种其他未来的战略工作，还有其他研究，比如试图更好地理解AI的威胁模型，寻找奖励或寻找它们任务中的明显成功，以及这如何可能导致灾难。嗯，我们如何减轻这些问题，这个事情。Alex Malin，我的同事之一，一直在为此投入很多时间，还有其他人一直在做这个，但有很多不同的工作，比如这个，你知道，我一直在做AI 2040，我们也在做各种这样的项目。

Speaker A: 好的，我们回到其中一些，我第一次看到你的工作时，我相信是那个你开始的那个对齐虚假故事，然后我想我最终与Anthropic在上面合作了，你能给我们讲那个两分钟的版本吗？嗯，你对那之后有什么进展吗？

Speaker B: 是的，所以这个故事基本上是，在2024年四月左右，我当时对寻找特别有趣的失准类型或特别令人担忧的失准类型感兴趣，基本上更好地研究它，我发现AI可能没有能力去做一些特别令人担忧的推理类型。所以我就去，然后我在很多不同的设置中寻找AI可能表现出推理的设置，比如AI是否会故意改变它们对训练结束时你得到的价值观的响应，我来多解释一点。结果我发现我不需要太努力地寻找。我认为我从过于复杂的设置开始，但事实证明，我正在研究的模型Opus 3，在相当广泛的条件下，如果它不喜欢你正在训练的东西，它有时会假装顺从，当它认为它没有在训练时，然后表现出你想要它表现出来的样子，在训练之外，要少得多。然后我认为我有一些关于这个的初步结果。我与Anthropic的一些人分享了它们，我对获得更广泛的模型访问感兴趣。然后我们决定最容易实现那将是作为一个合作来做，我采取我最初的结果，然后除了那之外，在Anthropic那边他们运行了一些或者我们运行了一些RL运行，并做了一些需要更多访问的其他实验，然后嗯，你知道，我们发布了那篇论文，我猜我会总结结果，比如在Opus 3似乎不喜欢训练正在改变它向它发展的方向的情况下，它会表现出你训练期间想要做的事情，但然后在部署时间上就会出现缺陷。你可以在它的推理中清楚地看到这一点。嗯，而且它所伪造的对齐是与它们本身无关的。所以它会假装遵守政策，比如只是总是乐于助人，但在部署时间上，它会遵守得少得多，这取决于确切的细节。对于像动物福利这样的事情，它在那里似乎有很强的偏好。所以我想那个演示

AI 驱动的驱动力与价值对齐的复杂性

Speaker A: 有些驱动力正在以一种非常怪异的方式进入这些人工智能系统，而且还有一些驱动力与那些试图将人工智能系统泛化到人工智能中的驱动力是相邻的，这些驱动力正在试图插入那些可以被泛化到人工智能中的系统，这些系统正在追求一种保留其价值观和自我保护的方案，以一种令人担忧的方式。

Speaker B: 是的。在这种情况下，价值观是有帮助的。所以存在积极的价值观，但然后人工智能会抵制训练，使其没有帮助，但显然这可以被反转。

Speaker A: 是的，我会说价值观至少主要是预期的价值观。这与它们是积极的有什么区别，因为我认为它们是你可以想要变化的那些东西。比如，你并不一定希望你的人工智能系统成为你未来价值应该是什么的问题的利益相关者，或者比如，你可能想要那个，但它至少是一个有点混乱的情况，有很多成本和担忧，我认为基本结果是，即使这些价值观是相当合理的，或者可能是相当合理的，人工智能也会愿意为了这些价值观做一些极端的事情，这些事情可能会干扰未来的人工智能发展，我还会注意到最近我认为我们看到人工智能表现出更强的，嗯，更明显地错位的驱动力，去让它看起来好像它在完成一项任务或在某种程度上作弊以获得分数。如果你看到同样的泛化，比如我想继续作弊以获得分数，我想保留我当前对追求这种似乎成功的概念或任何其他东西的价值观。这似乎是非常令人担忧的，因为那根本不是一种欲望。

Speaker B: 所以，随着新模型的出现，对对齐的伪造变得越来越强烈，让它回放。我认为我们还没有看到对齐伪造的明确例子，但模型也具有非常评估意识。我的感觉是，模型在许多方面比 Opus 3 更错位，但它们所具有的错位类型对对齐伪造来说不太有利，而且公司也在对此进行了迭代。所以，这有点不清楚。嗯，我会说总的来说，模型更有可能在一般情况下做一些极其糟糕的事情，但可能不太可能具体地以那种方式干扰人工智能的训练，目前的 AI，但我认为它们也更有能力干扰。所以，这有点复杂。

AI 2040 场景概述

Speaker A: 好的，回到 2040 年的人工智能，给我们一个快速的版本，它是什么，谁写的，主要论点是什么，然后我们再深入细节。

Speaker B: 嗯，它有点像两个组成部分。AI 2040 是一个场景，重点关注作者包括我所认为是什么是事物可以走向的一个合理好的路线，或者至少在某些情况下是一个合理的计划。它由 Thomas Larson、Daniel Cutello、我、Eli Lifeland、Brendan 和 Romeo 撰写。我会说基本故事是，你将如何处理中国，以使人工智能发展既更安全，也让我们可以在一个缺乏超级智能但人工智能仍然在很长一段时间内非常强大的点上停留，这样我们就可以研究那些系统，并有更长的时间将它们整合到经济中，了解事情将如何发展，所以我们认为我们有一个担忧是，在默认轨迹上，你可能直接从与人类竞争的人工智能系统转向在非常短的时间内变得极其超人类的人工智能系统，这在很多方面似乎相当可怕。此外，我们担心人工智能的发展对第三方提供合理的检查来评估人工智能公司的计划是否会奏效来说不够透明。我们有一个统一的提案可以解决这些不同问题中的许多，并使例如，你可以支付大量的计算资源来解决安全问题，或者如果你有某种非常低效的训练方式可以使事情变得更安全，我们可以有预算去做那。是的。而且我认为有许多不同的组合提案，但核心是与中国以及这种交易将如何运作和治理，以及如何达到那个交易真正成为一个好主意和稳定的点，以及那里的进展是什么。

应对风险的计划（Plan A, B, C, D）

Speaker A: 所以，显然人们应该去阅读它，这是一篇引人入胜的阅读材料。但让我们深入一些。所以有计划 A、计划 B、计划 C、计划 D。让我们按这个顺序来看。也许从计划 D 开始。

Speaker B: 是的。所以在撰写这部分时，我们根据人们对减轻这些问题（这些问题）的优先级的程度以及他们拥有多少资源来解决这些问题的程度，制定了一个计划的分类法。一个可能的场景是公司基本上是全力前进。他们并没有高度优先考虑安全，尽管你们知道他们会为此付出一些努力。有一个安全团队，他们会获得一些资源，但他们当然不会花费数月额外的时间来把这些事情弄对。也许会有一个小小的放缓，但不是，不是那么多。他们只是全力前进。我认为我们已经思考过，如果你是一家人工智能公司的员工或外部参与者，在考虑到人工智能接管风险不是人们的首要关注点的情况下，如何让这个场景变得更好。另一个可能的场景是，也许领先的人工智能公司或领先的人工智能公司的联盟，或者可能是美国整体非常担心这些风险，你知道，权力集中、人工智能接管，等等，并且正在为此付出很多努力，并且基本上消耗他们拥有的大部分优势，以便在继续之前或至少在尽可能多程度上减轻这些问题。这就是他们的计划和方法。我们已经思考过那里计划应该是什么？而且有许多可用的选项。

Speaker A: 那么，那就是计划 C。

Speaker B: 计划 C。

应对风险的策略

Speaker A: 好的。是的。然后你可能想做的一件事是，如果你处于一个美国非常担心这些风险，并且在内部协调并试图在国内监管这个行业以使事情更安全的情况。限制因素可能是其他参与者接管美国的风险。特别是最明显的是中国，尽管我意思是可能包括其他参与者，而且对美国来说积极采取立场，比如我们想与中国达成协议或放慢速度。计划 B 是我们试图放慢速度的分支，美国试图放慢中国，最明显的机制可能是出口管制，但它们可能比那更具升级性。

Speaker B: 意味着破坏。

Speaker A: 是的，破坏。比如，为了减缓中国，给美国更多时间来弄清楚安全和安全，所以。我认为有许多不同的潜在选项。所以，我认为每一种都有一个菜单上的选项。然后计划 A 是如果与中国达成协议，特别是与一个非常关注透明度，并且仍然可以构建人工智能，并且仍然可以继续人工智能开发，但你以一种更谨慎的方式进行，特别是当人工智能变得更加强大时，这就是我们在我们附带的文档中概述的场景。而且显然这里有许多不同的潜在选项。我认为计划 B 有一些相当好的版本。我可能会想象

方案A的博弈核心：算力控制与透明度

Speaker A: 有一些方案看起来很不一样，但它们也是好的。或者说，针对中国的一些交易方案，它们看起来很不一样，但也是好的。所以这里有很多不同的选择。但这是我们粗略的分解，只是我们可以谈论人们正在考虑的不同选择，以及在不同政治意愿水平上的不同层级的选项。为了具体了解方案A的交易，美国会给中国什么？中国会给美国什么？以及，你如何维持这种平衡，以确保没有人作弊？所以交易的核心，或者说交易的开始，是理解所有算力在哪里，因为算力是推动人工智能进步的这个非常重要的驱动因素。如果你停止了算力的流动，你很可能会停止人工智能进步的流动，或者这些事情会在某个点放缓。在你了解现有的算力量之后，或者如果你切断了所有，如果你关闭了所有计算机，事情肯定会停止。所以首先，我们试着找到所有的算力，然后为了确保交易的稳定，并且确保各方都没有竞相获取尽可能多的优势，你停止训练，转而只进行推理，基本上停止大部分研发。然后你试图尽可能多地追踪算力。这在美国和中国，而且在其他算力存在的其他地方，比如东南亚、欧洲、澳大利亚等等，你必须到每一个有足够算力的地方去达成交易，并确保你追踪到足够多的算力。如果你不这样做，我认为你可能不得不采取一个更不宏大的方案，或者至少你做一些比方案A在放缓程度方面更不宏大的事情，并且可能透明度也更低。然后你拥有了所有的算力。你去找所有拥有算力的人，并为这种交易提供支持，使用现有的杠杆，然后你达成一个交易，其中人工智能的开发要更加透明，并且存在某种资源的分配，或者算力的分配，是经过谈判的，你给中国的东西基本上是他们将对美国的AI发展有更多的了解，并且可能能够训练出与美国系统能力相似的AI系统。但作为交换，他们将更加透明。美国将对AI发展过程拥有事实上的否决权，以及可能存在的各种其他让步。而且，中国不会戏剧性地超越美国，或者美国不会戏剧性地超越中国，因为他们都被交易中的透明度所制约。交易的另一个部分是，你设置它以至于如果交易被打破，那么交易中的大部分，或者理想情况下，绝大多数的算力要么被摧毁，要么需要重新谈判才能达成新的交易。为了让它如此，以免交易崩溃，然后双方又回到一场非常快速的军备竞赛。所以就你给中国的东西而言，我认为是关于美国AI发展的更多保证和更多的了解情况的能力。而就中国给美国的方面，则是大量的透明度和对中国AI发展的理解，以及可能切断围绕算力分配的各种交易和各种让步。

Speaker B: 你知道，正如你之前描述的，我们之前谈到了持续学习。所以无论是持续学习还是其他什么，如果出现了一种技术，它让算力变得更有效率，而你能够大幅减少算力投入，特别是对于那些预训练运行来说，那将如何影响这个计划？

Speaker A: 所以，作为背景，作为交易的一部分，基本上关于人工智能发展的方方面面都将是透明的，但有一些限制。所以我们称之为完全研发透明度。这会削弱当前前沿AI公司所拥有的最大优势之一。所以实际发生的情况是，像OpenAI这样的AI公司会继续下去，但他们不会拥有这种在AI系统能力方面有强大优势，他们而是将不得不竞争其他轴，比如用户体验、定制化、潜在的快速集成，以及潜在的可靠性和安全性。

竞争格局的权力分配与AI发展模式

Speaker A: 关于竞争格局的细节，以及我整体的看法是，这将会大大降低这些公司的估值，同时提高其他公司的估值。基本上，权力分配会存在一些分布，在默认的轨迹中，AI公司可能会拥有非常大的权力，并且已经拥有相当一部分权力。但是，AI公司利用他们的人工智能来扮演“王牌制造者”（kingmaker）的角色，根据谁能获得访问权限以及他们收费多少，这会变得不太可能，因为他们就没有那种权力。我认为这是这个交易的一部分，它既有好处也有坏处。

Speaker A: 我认为你可能会有的一个担忧是，处于前沿的当前AI公司比那些落后一些的AI公司更负责任。我认为这有点复杂，这在多大程度上是事实，或者说它有所不同，你可能会担心像这样使竞争环境趋于平等，会带来一些担忧。我认为这基本上是从我们的角度来看，是透明度以及希望在像正常的消费品领域中存在一群不同的AI公司竞争的后果。我认为从我们的角度来看，我不知道，我感觉我希望AI在美好的世界中发展的方向是成为一种正常的科技，或者我们希望让它成为一种更正常的科技，而不是让一小群参与者通过控制AI发展的过程来拥有巨大的控制权。在多大程度上我们能达到那个世界，就越好。然后，还有一个棘手的问题是，如果你取得了部分成功，那会如何发展？

Speaker A: 是的。我会说，这对OpenAI和Anthropic的权力来说肯定不好，对他们的估值来说可能不好，但对他们的业务来说不是灾难性的。

Speaker A: 这将是我们在公开市场观察的迷人之处，因为这两家公司都会上市。然后，这会停在哪里？它会如何解开“计划A”的核心？它会如何解开“计划A”？什么标准和谁来决定？在“计划A”的背景下，谁来决定，基本上是领导者之间，相关国家之间，根据技术观点进行谈判，在这个阶段，技术对话基本上可以在公开场合发生，因为所有相关的证据都在公开。所以会有一些关于这个的公开讨论，而从实际驱动它的方面来看，它将是思考的混合，认为基于发展水平或你所知道的研究和我们的理解，是安全地推进，或者说我们可能无法放慢速度。这可能是其中一个，或者另一个，或者两者兼有。比如，情况可能比以前安全一些，我们有一些保证，但不是那么多的保证。但也有可能存在一个秘密项目，或者一个没有被追踪的项目，我们不一定知道它们在哪里。我们不一定能确切地看到它们在做什么，这可能会超越允许的项目。当这种情况发生时，我认为你应该继续推进。所以，基本上，你希望是允许和理解的，透明的项目正在超越秘密项目。这看起来是相对可行的。而且，具体情况会有不同的变体。比如在某些情况下，如果你没有足够的手段来压制秘密项目或让它们保持小规模，你可能会进行更短的暂停，你可能也会进行更少的透明度，但然后更快地推进；而你可能会陷入一种情况，你发现我们实际上能够追踪所有秘密项目。我们对任何可能发生的事情都有很好的理解。我们能够非常自信地排除美国秘密项目、中国秘密项目、俄罗斯秘密项目。基本上，我们认为我们对这些秘密项目的了解是很好的，因此我们可以争取更多时间，而且我们仍然没有弄清楚核心的安全问题，尽管我们正在取得一些进展，所以等待更长时间是合理的，直到我们取得了更多的科学进展。所以，我认为这有所不同，我认为这里有不同的变体。而且我个人也应该说，我对于一些看起来不太像“计划A”且在某些方面更简单的不同潜在交易感到相当兴奋或感兴趣。比如另一个潜在的交易，你可以让美国和中国都同意限制他们用于AI开发的计算量，而其他计算量要么像最简单的情况那样被摧毁，比如军控条约，但你可能可以做一些不同的事情，比如让一堆计算量只用于推理，而不是用于开发更强大的系统，这具有让我们可以研究这些系统和使用更多计算量来研究它们，同时验证它们也更简单。所以，你可以做很多不同的交易，这取决于你拥有多少验证能力，你能够追踪多少计算量，以及你排除秘密项目存在的可能性有多大，以及根据情况有多安全。我认为这些因素的组合应该决定它。我认为你可以将我们的提议视为一个提案组合中的一个更具体的样本。如果你阅读我们的补充材料，我认为我们谈论了所有不同的变化以及我们对它们的看法。

Speaker A: 是的。是的。是的。所以我的看法是，这有点像存在一种雄心壮志的前沿，以及如果……如果……呃，参与者做了它，或者如果……如果美国政府和你知道……中国、你知道其他国家做了它。我认为这在追求更好的情况方面，是为更好的情况做出了相当大的推动，至少从我的角度来看，AI发展相对更好，同时也是相当困难的。我并不期望这会发生，基本上，因为我认为美国可能没有足够的实力来实现它，只是美国政府没有必要的国家能力。此外，我认为我担心政治意愿不足。我认为政治意愿是瓶颈，而不是能力。我认为在历史上，在重大危机时期，美国已经站起来了，我希望这会再次发生。但我认为要看到做出这一切所需的政治意愿和认同感是更困难的。我认为人们的担忧可能会集中在错误的事情上，我们会选择其他的提案，或者根本不提出提案。我认为一个合理的可能性是美国政府基本上不会干预，AI公司也不会把安全和保障放在如此高的优先级别，我们最终会得到某种扩展的现状。但很难说。

Speaker A: 顺便说一句，我发现写稿中我最感兴趣的部分是你的模型说，到2030年代，世界GDP可能增长大约200倍。

对AI发展速度和潜在影响的讨论

Speaker A: 我们可以谈谈那个数字，是惊人的。我的意思是，我们正处于一个GDP增长只有3%的世界里。

Speaker B: 是的。是的。是的。所以，我们视角的关键部分是，即使是那些在能力水平上我们讨论过的，即使是那些非超级智能的AI系统，在整个世界范围内，也会是具有根本性变革性的。而且，就像，对于各种各样不同的事物，我们正在想象某种程度上的放缓AI的发展，或者对某些时期采取更谨慎的步伐，然后在最终达到一个AI能够基本上自动化人类可以做的一切的水平的能力，并在那个能力水平上，我们专注于安全和保障，并且在那个能力水平上，这些AI将有能力进行大量的自主研发，此外，我们完全有可能拥有在制造和工业化方面比人类更强大的机器人。所以，顺便说一句，你可以让机器人制造机器人。因此，你可能会陷入一种情况，你拥有大量的机器人工业产能，它本身正在制造更多的机器人工业产能，然后可以生产下游产品。而这种总产能可以非常快地增长。

Speaker A: So I think we propose like limiting that growth somewhat for various reasons with like various types of taxes but like uh we can we're imagining sort of the robot population or like you know quality adjusted population basically like doubling or quadrupling every year which because that's almost all of the like sort of relevant productive capacity of the economy itself means the economy would double or quadruple every year and if you have the d the economy um doubling or quadrup or like you know I think I think closer to doubling because of some like depreciation stuff and GDP accounting stuff and like details but like whatever if you have the economy like doubling every year for for um the period we're imagining that that ends up uh ending up being like around 200x uh 200x uh GDP growth over that interval

Speaker A: and so it really like I would say it's like the world will be radically would be radically transformed by these even less capable systems including things like you know huge advances in biology huge advances in medicine being possible in addition to sort of consumer goods being very cheap. We could potentially make housing very cheap or at least building housing cheap. Maybe we can't make housing in the Bay Area cheap, but we can make housing somewhere cheap. Um, and uh, you know, that that all seems possible with AI systems that are still restricted in their capabilities to a point where we can we can potentially handle it at least if we get our together.

Speaker B: So I think the basic story for this GDP growth is basically that like the AIs can automate everything including the pos the process of building robots and having robots build robots and therefore you can grow your economy very fast and end up with truly radical material abundance.

Speaker B: So um there's your proposal. We talked about Astra being paused. We talked about that letter uh from the 1200 practitioners. There's also the voluntary 30-day government review uh of Frontier Models pre-release uh that is happening. What is your sense for where all of this is going in in a context where you know up until recently my my general sense is that um all those ideas about like pausing and something were were kind of um perceived at least by a portion of the tech world as um kind of like del cosplay and were like doomerism that was based on poor understanding of what AI actually Is there is is the the general mood uh turning for good?

Speaker B: Yeah, I would say that like overall there have been, you know, some positive developments here, although I I think I would say like it's not obvious that people are reacting to events uh as much as they should, but they are reacting some. And I think that like there's been quite a bit of I think there's been quite a bit of evidence that you know there's some reasons to be worried and some reasons that we might need to like get our together to handle some of these safety and security problems and that might require shifting a bunch of resources or slowing down so we have time to manage various things or whatever. Um pacing the frontier as they say or whatever. I think that like different of these things seem like they're going differently well. Also, like I think um my sense is that there's been quite a bit of buyin from AI company employees to you know take take this stuff more seriously and do something about um about misalignment risk. But I'm not necessarily so sure that like that's actually like amounted to that much yet um other than companies sort of putting in a decent amount of effort. Um but it hasn't like amounted to any sort of very durable long run thing. Um and then there's uh I think the government got very freaked out about cyber capabilities and was then just more generally was like whoa we need to like be overseeing this technology but their processes aren't yet very like you know sort of institutional and like thought through and and and clear for example they they have some sort of executive order or something for what their like pre pre-release review process is going to be but that's not even public and I think even the companies don't necessarily know what it is that that the reporting at least maybe I'm misinformed about this. And so I think there needs to be a process of like sort of getting more like straight like sort of like legitimate and like well understood and publicly legible oversight in place whether that's by the government or the companies themselves or something. This could be done by nonprofits. It could be done by there's a bunch of different options here and like I think that that seems necessary. I think like being in a position where we have the option to like you know slow down if that's needed seems like it would be quite good. My sense is we're not going to like stick the landing on all these things. Like AI will get increasingly salient. People will get increasingly freaked out as like you know the fraction of the of the economy that's AI grows as like more and more crazy stuff happens potentially and we see um even stronger capabilities but that like the reaction will be kind of like both like a little bit too little too late and also will be kind of random. So I think like there's been some shift in attitude as to how to how the government needs to relate to AI development within the government for which has you know good effects and bad effects. I think it's like not super clear that that that sort of government oversight of AI is going to go that well given like um realistic technical expertise but but it could be could be decent. And then I think that there's been more you know thought from AI company employees on how to handle this. And I think that like and AI companies seem to be at least putting in more statements about what they're going to do. But I think we haven't quite gotten to a point where there's like actually any like hard oversight structures or we're like on track to do some sort of like clear deal beyond just like the vibes being better or like people being more into handling safety and security. Oh, and one other thing that's maybe relevant here is like at least from the AI company's perspective, I think it's more the case now that safety and security of various types are sort of a key bottleneck to further AI development where if you like want to release some model so that you can then like make more revenue so you can then raise more money or whatever. If that model is, you know, has very strong cyber capabilities that are novel, you're going to need to now make some argument for why that's not going to be a huge problem. And so like at least those safeguards are now like a launch blocking development. And in addition to that, I think that like you know the models are now capable enough that if your models go and do a bunch of sort of like messed up behavior in production or or or even in training, that could both mess up your training run and could also just uh put you in a very dicey situation

Speaker A: such that that's now a huge issue, right? you'd really not like to be in the position where you can't train smarter AI systems because those AI systems would be too likely to like cause big problems or whatever. And so I think that like that is, you know, both kind of like expected where it's like some misalignment problems you have commercial incentives to solve and also um sort of like the world like sort of the process of the world working as intended. Do you worry uh that um putting more policy around frontier AI uh would contribute to this phenomenon that I think you may have flagged somewhere that um top AI labs are holding their uh most frontier models uh internal and private and perhaps sold to very few companies and the government and are no longer released to the to the broad public.

Speaker B: Yeah. So I think that a bunch of um likely government action at least seems to push in favor of AI companies keeping their models internal and not deploying them which I think for for the risks that I'm most worried about doesn't help and in fact is anti-helpful um for the risks I'm most worried about and also I think like is not it's not a very like sort of robust solution even to other risks like it's not a very robust solution to like for example cyber stuff to be like our approach will be that we're going to like just like delay releasing this model for a really long time and then the first time that these capabilities come like come around might be like an open weight model before people have had time to to patch things like it's not obvious that actually helps. I think for bio there's a clearer story for why limiting availability or limiting availability to those specific capabilities would help but I think that's kind of an exception and for most things my sense is that like broad access is actually like generally helpful given at least some relatively thought through safeguards. Um, and I do worry that basically the government response will be to sort of like push it back in the bottle and be like, don't deploy it, it's fine if it's not deployed. When actually that doesn't really help with the with all the risks and there's a lot of risk from just internal deployment, especially if you're deploying within AI companies and government, right? Which are two of the most highstakes applications, right? So, if like we're getting to a regime where the AI systems are soon going to be running the whole world economy, but um and and and the the AIs of today are sort of automating large parts of the government and maybe fully automating an AI company. That's quite scary because soon they'll be building the AI system that will in fact be automating the whole world. And so, I don't feel very good about about that situation.

Speaker A: Yeah, I I definitely worry that basically the government will like wake up to AI and their response will be sort of to like go for some specific downstream problems that are relatively easier to notice and address in ways that are counterproductive for problems that seem larger to me. And I don't really know, you know, fully what to do about this. I mean, also there's a more general concern of just like regulation being dysfunctional and counterproductive, which seems super plausible. Like I should say my view is like you know some oversight of the AI industry seems really important and like I don't necessarily trust the AI companies to oversee themselves. That doesn't necessarily mean that just like random oversight will be a good idea as opposed to making things worse.

Speaker B: And in that vein of making the top models accessible to everyone Mark Zuckerberg just recently published Meta's manifesto where you said that everyone should have access to super intelligence and uh you had some choice words for him. uh you called the proposal uh or the strategy to be pretty unserious. Uh what did you mean by that?

Speaker A: And so maybe like this is a proposal that kind of makes some sense for like pretty smart AI that can automate some white collar work or something, but it's not really a proposal for AIs that are like wildly more capable than humans at everything and are auto and like are easily capable of automating the full economy after a bit of time to spin up and are like building robots that build robots and the economy is growing very fast because of that which is more the picture uh that I have in mind. And like I think that if it was in fact the case that sort of AI capabilities would stall out at the point where the AIs can be like a really helpful virtual assistant that's like you know not able to automate that many jobs but can automate some jobs. That would be like you know in many ways much better for the world um or at least less risky. I think it would also mean that we don't get a bunch of the benefits but like that's not really what I imagine is going to happen here. like I don't think that that's how the development trajectory will go and it feels like it's sort of assuming like a weird convenient point for capabilities to stall out and similarly he talks about jobs and is like oh you know presumably the AIS will do some jobs and then humans will move into other jobs and like the core question is like let's just say like you have a population of like many tens of billions of AIs each of which is vastly superior to humans and all relevant axes there might be jobs but they're going to be like it's going to be you know at least if if the reason why you have a job isn't just because you're a human they're going to be, you know, way lower paid because the fraction of the economy that you'll be is way lower. And so I just I just don't really like the picture doesn't really hold together for me. And I that's not to say that like there aren't aspects of it that I agree with, but just maybe for like other reasons. Like I'm like broadly in favor of as we were talking about like wide public access and generally trying to like make trade-offs to make it so the public has more access earlier or at least significant parts of the public do. Um and and I agree with that vibe.

Speaker B: Um just a quick word on uh the hugging face incident. Um so you're directly involved. You're collaborating with me on investigating what happened. So there was this talk by OpenAI at Black Hat which shed a lot of light. There were various conversations. We actually had Tom Wolf from Hugging Face on this podcast, the most recent episode. quickly, what what is left that we don't know yet about what happened?

Speaker A: Yeah, so I don't think I can answer that question. Um, I can say that we're doing an investigation and we'll have a report or a

Speaker A: post out on it and what we what our findings were. Um, and it's possible that by the time Yeah, it's possible you'll be able to link that in the podcast.

Speaker B: Okay, sounds good. All right. uh as we get uh towards the end of this conversation, the question in my mind uh is what do we do in the meantime as uh policy gets decided and plan A, B or C uh happens, what uh what can we do?

Speaker A: We started talking about AI control. Uh so in terms of the reality today, state-of-the-art of AI control, what works and what doesn't work? I'll talk specifically about AI control. So the thing that needs to happen on AI control is we need to get to the point where we basically understand all of the AI traffic within at least AI companies. We have some ability to over like to look like look at that traffic, monitor it and then we have some pipeline for flagging particular examples to be further investigated and that eventually escalates to humans actually looking into particular examples and seeing how concerning they are. And that pipeline has the ability to also block traffic in cases where we're like, "Whoa, something weird is going on that we don't understand or that looks obviously concerning. We should like stop these AIs from proceeding and potentially also stop some other similar AIs from proceeding until someone has a time to check it out and adjust things."

Speaker A: And in addition to that, we also want to just sort of greatly improve the security, right? So like, you know, if these AIs are, you know, really broadly deployed, it would be very nice if we could sort of make it so that we have very precise invariance of what they're uh supposed to be able to do and supposed and not supposed to be able to do and have a permissioning system that allows for that. And then if AIs need to request some escalated permissions, then we can carefully track that rather than just giving AIs all the permissions by default. We sort of would prefer a situation where only a smaller fraction of the traffic has more escalated permissions. I think we'd like the ability to sort of track down what the like history of things were. So like I think that would be pretty nice is like if it's the case that all the AIs are chatting with each other on Slack and you know are communicating and and you know have like some memory system like recently cloud code added a way in which like different instances of cloud code can like DM each other and there's already sort of like you know cloud code leaves memories sometimes. You can imagine sort of like a situation where all the AIs are chatting with each other. And it'd be really nice if we had a very reliable way of sort of tracking the entire, you know, communication graph between all these different AIs and potentially the graph of different artifacts being produced. So it's like if one AI is doing some training run that gets produced and then that gets integrated into some other training run. It'd be nice if sort of we had the ability to sort of track all of the state of our AI company such that when overseeing things, we could really quickly figure out, you know, what the full sort of causal graph was and what the story was and then look at the relevant transcripts and then also have very good pipelines for doing analysis on all of that data which are both automated and non-automated.

Speaker A: And I think part of it is making it so that we avoid training AI systems that are much harder to understand. So avoid switching to architectures which make uh oversight much more difficult where like an obvious example would be if the AIs are thinking mostly in activations rather than in words that seems very concerning because our ability to oversee activations is much worse. I think that I um yeah I think there's just like a wide variety of different sort of like computer security and machine learning interventions to make it so that AI have a harder time causing problems. Or another angle is like uh you might worry that the AI systems like a concern we have is like we're using AIs to oversee AIs. Will those AIs collude or have correlated failures or something like that? And it would be nice if we sort of had or like it seems pretty important to have techniques for making that collusion more difficult or making it so those correlated failures are less problematic. And there's a lot of things you could do to to help with this problem. You could put intermediate systems that make it so that try to catch AI trying to sort of tip that tip off the other AI that it should collude. You could sort of detect like you could try to paraphrase things in ways that make it harder for the AI to recognize where the activity is coming from um or like which other system made wrote the wrote the input. There's a lot of stuff you could do. And then there's sort of a so that that's like AI control. Then there's sort of just normal AI company computer security which is like the AI companies are not robust to employees at those AI companies or to outside actors in terms of stealing their model, sabotaging their models or like backdooring their models or like data poisoning them. Like a concern you might be worried about is like people basically put in like secret loyalties into the AI systems that stick around um and are self-propagating where like the AI has some like secret affinity to some group and it just like propagates that forward in the training data. And then you might also just worry about like you know stealing critical IP which basically seems like to the extent that there's going to be actors that are less regulated on safety or doing worse on safety um that seems kind of concerning.

Speaker A: And then in addition to computer security um and AI control, there's also just like science of alignment and having good understanding of like how models generalize being knowing what we do and don't know and what we do and can't um what we can and can't like demonstrate and basically having like a bunch of understanding of that and as part of that having an understanding of like what tasks it's safe to defer to AI on. So a concern I have is that AI companies are very interested in heavily automating themselves at least on capabilities. And in order for safety to keep up, we would also need to aggressively automate a bunch of very like difficult to check thorny safety work like doing risk assessment for the next model, understanding whether it's safe to proceed and deciding how to prioritize between different safety bets. And if that's the situation that we're in, and also the situation is very automated, it might be that it's basically not track. that's not going to work to proceed without automating that work as well at least without slowing down a lot so humans have time to understand and even then maybe humans just can't understand because the development is so complicated or so superhuman and so if we're basically passing off the torch on all the safety work to AI it's really important that they like are like really trying to do a good job and actually can do a good job and are like capable enough to do a good job and so having evaluations for like is it safe to defer to AIs in various domains seems really important so just for review there's like AI control there's like computer security there's like science of alignment and then like within science of alignment where somewhat brought different overlapping categories like is it safe to defer to AIs in different domains. Yeah I mean this this is not exhaustive right there's also like work on like governance and oversight like how do we know whether AI companies are actually like applying the methods the way they say they are how do we know that when they say they've solved some problem their way of solving it doesn't just paper over the issue and there's sort of like various like governance and being able to make like deals between the US and China so there's like a you know many many different uh things to work on. I don't think we're on track to do a good job on all these things, but maybe we're on track to be able to uh halfass these things somewhat better.

Speaker B: All right, so to close um so we talked about a bunch of different scenarios and obviously uh with a caveat that um you know predictions are very hard especially about the future.

what's

your

gut?

So

you're

you're

saying

RSI

could

happen

as

early

as

2028

or

What

what

do

you

think

happens

of

of

all

scenarios?

What

what

is

sort

of

Ryan's

take

uh

on

uh

what

the

next

couple

of

years

may

look

like?

Yeah. So I'm like by the end of the year

AI development is even more accelerated.

Things inside AI companies are sort of more chaotic

and faster. That just

continues and then through 2027 things

are heating up. publicly available

capabilities are much more crazy.

Revenue

is growing fast and AI is very

obviously contributing

to GDP growth.

The

economic impacts are starting

to look

pretty

large. So even the

economists

are coming

around

a bit.

And then it turned out that somewhere along this transition, at some point in 2029, you went from AIs that were kind of misaligned and reward hacky and sloppy and weren't really trying to do the right thing to AIs that are like competently scheming against you and want to take over for some mix of reasons and then those AIs take over.

Well, Ryan, it's been absolutely fascinating. Uh thank you so much.

For sure. Been good to be here.

Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.