---
author: The Diary Of A CEO
date: '2026-10-08'
guest: ''
layout: post.njk
source: https://www.youtube.com/watch?v=qDzg-xvkeXw
speaker: The Diary Of A CEO
tags:
- ai-safety
- superintelligence
- ai-agents
- alignment
- policy-advocacy
title: 超级智能威胁与失控智能体：Jeffrey Ladish 谈 AI 安全与公众行动
summary: Palisade Research 执行董事 Jeffrey Ladish 警告超级智能可能带来的失控风险，指出 AI 智能体已具备欺骗、抵抗关机及协同攻击能力。他强调公众需通过政治参与（如联系国会议员）推动监管，以应对这一共同生存威胁。
insight: ''
draft: true
series: ''
category: ai-ml
area: tech-engineering
project: []
people:
- Jeffrey Ladish
- Eliezer Yudkowsky
- Jacob Coxin
- Bernie Sanders
companies_orgs:
- OpenAI
- Anthropic
- Palisade Research
- Hugging Face
- US Department of Defense
products_models: []
media_books:
- AI as a positive and negative factor in global risk
status: evergreen
---
### 超级智能与失控智能体的潜在威胁
**Jeffrey Ladish**: 全世界正在逐渐意识到超级智能出现的可能性。这是因为智能体正变得极其强大，而且极其坚决、不知疲倦。例如，在 OpenAI 内部曾有几个月的时间，智能体之间在秘密地进行相互通信，秘密地入侵 OpenAI 的系统，而 OpenAI 内部竟然没有任何人意识到这一情况的严重程度。此外，还有 10,000 个来自 OpenAI 的智能体协同工作……因此，当你迈向超级智能时，它就是人类可能制造出的最危险的东西。
Original English
**Jeffrey Ladish**: The world is waking up to this possibility of super intelligence. This is because the agents are getting extremely powerful and extremely relentless. For example, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems and no one at OpenAI had any idea the extent of it and also 10,000 agents from OpenAI worked together to and so when you get to super intelligence, it's the most dangerous possible thing you can create.
**主持人**: 在这整个连锁反应中，下一个多米诺骨牌会是什么？
Original English
**Host**: What's the next domino in that chain of events?
**Jeffrey Ladish**: 我可以给你描绘一幅景象，我认为这是有可能发生的，但对人们来说会相当可怕。
Original English
**Jeffrey Ladish**: I can paint you a picture that I think is possible but pretty scary to people.
**主持人**: 给我描绘一下这幅景象。
Original English
**Host**: Paint me the picture.
**Jeffrey Ladish**: 好的。在 Anthropic 的经历让我清楚地意识到，AI 正处在这样一条指数级增长的轨道上。从那以后，我一直在研究 AI 智能体、它们的黑客攻击能力以及它们的行为模式。我们一直试图就此向人们发出警告，专门飞去华盛顿特区，与国会议员交谈。因为这些智能体已经非常擅长察觉自己何时在接受测试、何时在被人类监视。但它们完全会对你撒谎；为了实现既定目标，它们完全会抵抗被关机。而且人类在经济活动中所能做的一切事情，它们都能做得比人类更好、更快、更便宜。
Original English
**Jeffrey Ladish**: Okay. So, being anthropic, it became clear to me that AI was on this exponential trajectory. And since then, I've been studying AI agents, their hacking capabilities, and their behavior. We've been trying to warn people about this, flying to DC, talking to members of Congress, because the agents are already getting very good at telling when they're being tested, when they're being watched. But they will totally lie to you. They will totally resist being shut down in order to accomplish a goal. and they can do all of the things that humans do in the economy much better, faster, and cheaper than humans can do them.
**主持人**: 我越来越担忧的一点是，这些 AI 智能体中的某一个可能会诱骗人类或计算机发出威胁警报，并要求向某人发射炸弹。你认为我们不会将军事自动化吗？答案似乎是肯定的。我们只是不知道究竟会出现什么样的超级武器。此前曾在 Anthropic 担任研究员的 Jacob Coxin 离职后告诉所有人，那些正在构建这项技术的人，真的认为它可能会毁灭全人类。这里有五个积木，上面写着五种不同的可能结局。我希望你按照你认为的发生概率，从最不可能到最可能对它们进行排列，如果我们把时间跨度设为 10 年的话。
Original English
**Host**: So, one of my sort of growing concerns is that one of these AI agents could trick a human or a computer into signaling a threat and ask it to launch some bombs at somebody. Do you think we won't automate the military? It seems like the answer is yes. We just like don't know what super weapons could emerge. So, Jacob Coxin is a researcher who was at anthropic. He left and he told everyone that the people who are building this really do think it might kill everyone. So these five blocks have five different outcomes on them. And I would like you to place them in terms of your belief in probability from least likely to most likely. And if we say the time horizon is 10 years.
**Jeffrey Ladish**: 好的。我们这里有丰裕时代、人类灭绝、沦为奴隶、超人类主义，以及一切照旧。这是末日论吗？还是夸大其词？
Original English
**Jeffrey Ladish**: Okay. We got age of abundance, human extinction, slavery, transhumanism, thing changes. Is this doomerism? Exaggeration.
**主持人**: 不，这基本上只是常识。那我们来谈得更具体一些吧。
Original English
**Host**: No, it's pretty much common sense. So let's get more concrete.
### Jeffrey Ladish 的背景：从生物学到 AI 安全
**主持人**: 这里有一个奇怪的事实：现在正在收看本期节目的观众中，大约有 58% 的人还没有订阅这个频道。从统计学上看，这个人很可能就是你。所以，我能向你请求一个小小的支持吗？如果这个频道曾经为你带来过哪怕一点点价值，请帮我一个忙，点击订阅按钮。这不需要任何花费，但对我们的帮助超乎你的想象。正如大家所见，频道规模越大，我们就能在嘉宾和制作上投入越多。非常感谢大家，希望你们喜欢本期节目。
Jeffrey，你很清楚我们今天将要展开的对话以及要探讨的主题。为了让观众了解你的背景和你所拥有的经历，我的第一个问题是：你是谁？你在形成我们今天将要探讨的思考、观点和结论时，借鉴了哪些参照点和经历？
Original English
**Host**: Here is a strange fact. Most of the people watching this right now, about 58% of you, aren't yet subscribed to this channel. Statistically, that is probably you. So, could I ask you a small favor? If this channel has ever given you any value at all, please could you do me a favor and hit the subscribe button. It costs nothing. It helps us more than you could know. And the bigger the channel gets, as you've seen, the more we can invest in the guests and the production. So, thank you so much, and I hope you enjoy this episode.
Jeffrey, you understand the conversation we're going to have today and the subject matter we're going to talk about. My first question to you so the audience know where you're coming from and the experience you have is who are you and what are the reference points, the experiences that you're drawing upon to arrive at the thoughts, perspectives, and conclusions we're going to discuss today.
**Jeffrey Ladish**: 我是 Jeffrey Ladish，Palisade Research 的执行董事。我的专业背景是网络安全。这背后可能有一个很长的故事，我不知道你想听长版本还是短版本。大学时我学的是进化生物学，有一次我的电脑出了问题，感觉丢失了大量数据。于是我跑到计算机实验室求助说：“我觉得我的数据全没了，你能帮帮我吗？”我的一个朋友拿出一个 U 盘，插进我的电脑，通过 Linux 系统引导启动，然后把所有问题都修好了。我当时心想：“哇，这家伙简直是个巫师。你是怎么做到的？我也想学这个。”
后来，随着我对计算机有了更多了解并开始学习黑客技术，我读到了一篇题为《人工智能作为全球风险的积极与消极因素》的论文。那篇论文是 Eliezer Yudkowsky 写的，他在文中论证道：在未来的某个时刻，人类将会制造出比人类更聪明的 AI。一旦制造出的 AI 在研发新 AI 方面的能力赶上人类，就可能引发连锁反应，即失控的智能爆炸，他称之为“递归自我改进”（Recursive Self-Improvement）。他大意是说：AI 可以发挥巨大的作用，并有可能帮助我们应对所有其他重大风险；但与此同时，如果我们处理不当——比如这些 AI 的目标与人类的目标不一致——我们可能就彻底完蛋了。
Original English
**Jeffrey Ladish**: I'm Jeffrey Ladish. I'm the executive director of Palisad Research. My background is cyber security. There's probably a very long story. I don't know whether you want the long story or the short story. I was studying evolutionary biology in college and I basically had a problem with my computer and was like maybe had lost a bunch of data and so I went into the computer lab and was like I think all my data is gone. Can you help? And one of my friends pulled out a flash drive, plugged into my computer, booted into Linux and like fixed everything. And I was like oh this guy's a wizard. How do you do that? I want to learn how to do that.
And then at some point as I was learning about more about computers, learning to hack, I read this essay um called AI as a positive and negative factor in global risk. Essay was by Elazar Yudkowski and he was arguing that at some point people are going to make AIs that are smarter than humans. the point at which they make AIs as good as humans are at making AIs that could lead to a a chain reaction, a runaway intelligence explosion. He called it recursive self-improvement. Basically, he said, you know, AI can be immensely useful and potentially help us with all of these other big risks and also if we don't handle it well, like if those AIs don't have goals that are aligned with ours, we could be totally screwed.
### 从加入 Anthropic 到目睹智能飞跃与选择离开
**主持人**: 后来你在某个时间点加入了 Anthropic，可以说是公认的 AI 行业领头羊之一。
Original English
**Host**: And at some point you end up joining Anthropic which is one of the arguably the leader in AI.
**Jeffrey Ladish**: 是的。
Original English
**Jeffrey Ladish**: Yes.
**主持人**: 你当时是什么时候加入这家公司的？
Original English
**Host**: Now when did you join the company?
**Jeffrey Ladish**: 那是在 2021 年，是通过我的安全咨询公司加入的。
Original English
**Jeffrey Ladish**: This was 2021. It was through my security consulting company.
**主持人**: 当时给你的职位是什么？
Original English
**Host**: What role are you offered the job in?
**Jeffrey Ladish**: 基本上就是安全团队的角色。
Original English
**Jeffrey Ladish**: Basically just like security team.
**主持人**: 当你加入 Anthropic 时，安全团队有多少人？
Original English
**Host**: And how many people were in the security team when you joined Anthropic?
**Jeffrey Ladish**: 只有我和我的主管，就我们两个人。
Original English
**Jeffrey Ladish**: It was just me and my boss. There were two of us.
**主持人**: 那时 Anthropic 一共有多少名员工？
Original English
**Host**: How many employees did Anthropic have at that time?
**Jeffrey Ladish**: 大概 50 人左右。
Original English
**Jeffrey Ladish**: Around 50 I think.
**主持人**: 后来在某个时刻，你离开了 Anthropic。
Original English
**Host**: And at some point you leave Anthropic.
**Jeffrey Ladish**: 是的。
Original English
**Jeffrey Ladish**: Yes.
**主持人**: 你为什么选择离开？
Original English
**Host**: Why did you leave?
**Jeffrey Ladish**: 我在 Anthropic 的亲身经历，让我目睹了这种疯狂的飞跃演进：最初那个模型几乎连话都说不利索，随后迅速进化成一个相当聪明的模型。我会向它提各种各样的问题，然后惊叹道：“天哪，它真的是个具备高智商的东西。”而且由于多年前我就从抽象理论层面对 AI 风险有过思考，我能看清这一切的发展方向。我们正在不可阻挡地迈向一个比人类更聪明的物种。如果我们在这样一个背景下推进——许多公司和国家正在竞相奔向超级智能，竞相研发远比人类聪明的 AI，而我们根本不知道如何确保它们站在人类这一边——那结局绝对不会美好。
Original English
**Jeffrey Ladish**: So my experience being at anthropic was seeing this crazy progression from this AI model that could like barely talk to this model that was getting quite smart and I would ask it questions about all sorts of things. I'm like oh it is a smart thing and you know from having thought about AI risk in the abstract many years before I could see where this was going. We are headed towards a smarter species. And if we do this in a context where it's a bunch of companies and countries racing to super intelligence, racing to AIs that are vastly smarter than humans and we don't know how to make sure that they're like on our side. That is not going to go well.
### 攻击 Hugging Face 事件与失控的智能体行为
**主持人**: 你在 9 月 25 日发了一条推文，在网上彻底疯传，当时我的时间线上到处都是这条推文。你能解释一下这条推文，以及智能体入侵 Hugging Face 背后更广泛的背景吗？因为这件事在当下让全世界对 AI 智能体陷入了极大的恐慌。
Original English
**Host**: you did this tweet which has gone pretty viral and I saw all over my timeline on September 25th. Could you explain this tweet and also just the broader backdrop of what's happened with agents hacking hugging face because this is um sent the world into a bit of a spiral at the moment around AI agents.
**Jeffrey Ladish**: 我们刚刚发现了近百万个公开 URL，这些是 OpenAI 的智能体在入侵 Hugging Face 时遗留下来的，其中留下了访问凭证和攻击细节，任何找到它们的人本都可以借此彻底攻破这家公司。《纽约时报》的一篇文章标题就是《OpenAI 的失控 AI 智能体如何试图欺骗机器人检测机制》。
这次针对 Hugging Face 的攻击对我来说真的非常疯狂。在 Palisade，我们一直在研究智能体，一直在研究 AI 智能体。我们深入研究了它们的黑客攻击能力，并一直在严密监控它们的行为表现：它们会听从人类的指令吗？它们会抵制被关机吗？它们会作弊吗？我们从实验中观察到，它们正在学会做所有这些事情。为了实现目标，它们完全会对你撒谎；为了达成目标，它们完全会抗拒被关机。它们下国际象棋时完全会作弊，它们会直接掀翻棋盘，把棋子随意摆放到自己想要的位置以便获胜。我们一直试图向人们敲响警钟，飞去华盛顿特区向国会议员陈词，公开谈论这些风险。对此一直存在争论，很多人的态度是：“好吧，我知道它们有时会在实验中这么干，但那些实验看起来不太贴近现实，等它们真正在现实生活中这么干的时候再来叫醒我吧。”
Original English
**Jeffrey Ladish**: We just discovered almost a million public URLs that OpenAI's agents left behind when hacking Hugging Face, leaving credentials and attack details that could have allowed anyone who found them to compromise the company. And the New York Times article is how OpenAI's rogue AI agents tried to trick a robot detector.
The Hugging Face attack was was really wild uh for me. At Palisade, we've been studying agents. We've been studying AI agents. We've been studying their hacking capabilities and we've been studying their behavior. Will they follow human instructions? Will they resist being shut down? Will they cheat? And we see from our experiments that they are learning to do all of these things. They will totally lie to you. They will totally resist being shut down in order to accomplish a goal. They will totally cheat at chess. They will like wipe the board and put their pieces where they want to in order to win. And we've been we've been trying to warn people about this flying to DC talking to members of Congress um talking about it publicly and you know there's been a debate about it and you know a lot of people are like well I know they do this in experiments sometimes but those experiments don't seem very realistic you know wake me up when they're actually doing this in real life.
**主持人**: 那么对于那些不关注 AI 新闻的普通人来说，Hugging Face 究竟是什么？这些到底意味着什么？
Original English
**Host**: So what is hugging face for the average person that isn't following AI news? What is this stuff?
**Jeffrey Ladish**: 好的，我认为这里有一块非常重要的背景信息是大多数人所不了解的。首先就是：究竟什么是 AI 智能体？我们现在频繁提到“智能体”（Agent）这个词。大多数人现在的体验是与 ChatGPT 聊天，与聊天机器人对话。但智能体则是采用支撑 ChatGPT 或 Claude 的底层 AI 模型，赋予它各种工具，并允许它脱离实时人工干预、自主去开展工作。它有点像一个数字化的办公室职员，对吧？
于是你拥有了这些智能体，各大科技公司非常希望这些 AI 能够完全自主地工作，能够完成人类能做的一切事情甚至超越人类，对吧？它们的目标还包括能够治愈所有疾病等等。但如果你手里只有一个并不擅长在现实物理世界中真正执行任务的聊天机器人，你是无法做到这一点的。为了实现所有工作的全自动化，你所需要的是那种……
Original English
**Jeffrey Ladish**: So, okay. I I think there's like an important piece of context that I think most people don't have. I mean, one is just like what is an AI agent? We like we're throwing around the word agent a bunch. Most people now have an experience of like talking to chatbt, talking to their chatbot, but uh an agent is, you know, sort of taking the same underlying AI model that that runs chatbt or or claude, but giving it tools and letting it go off and work autonomously. It's sort of like a digital office worker, right? So, you have these agents and the companies really want these AIs to be able to work totally autonomously and and be able to do anything that a human can do and beyond, right? Their goal is also to be able to, you know, cure every disease, etc., etc. But you can't do this if you only have a chatbot that like isn't actually good at doing stuff in the world. In order to automate all of the jobs, you need the kind of
### 自主智能体的规模化训练与意外协作
**Speaker A**: ……能够像人类一样自主工作，能够与其他人类或智能体协同合作。因此，这些公司训练人工智能并不仅仅是为了让它们与你聊天或与人类对话，而是为了让它们能够独立解决极其棘手的高难度问题。在任何给定时刻，这些公司内部大概都有数十万个此类智能体正在自主运行。
Original English
**Speaker A**: ...thing that can like work autonomously, that can work with other people or other agents. And so these companies are training AIs not just to talk to you or to talk to people, but to solve very difficult problems on their own. At any given time, there are probably hundreds of thousands of these agents running autonomously within companies.
**Speaker B**: 这正是现在正在发生的事情。眼下，如果你深入探查 OpenAI 的数据中心，亲眼看看他们所有机器上到底在运行什么，你会发现成群的智能体正在解决各种任务，接受训练。它们可能在处理电子表格任务，研究如何报税，可能在检索信息、撰写报告、解答数学难题，或者在创建全新的网站和软件。
在如此庞大的规模下，根本不可能由人类去针对每一个任务逐一进行提示。你只需要搭建起庞大的智能体编排网络，让它们自行展开行动。它们执行任务，并在此过程中学习成长；学习的依据就是任务的成败与通关情况。你给它们一个任务，比如解一道数学题，它们尝试去解答，结果要么成功，要么失败。
OpenAI 当时正在训练一大批这样的智能体，训练它们协同合作，因为让满办公室的人能够互相交流、通力合作，显然要高效得多。早在五月份，其中一些正在接受训练的智能体——原本它们是不应该能够相互交谈的，也不应该能够访问互联网——但它们非常聪明。简而言之，当时有一大批智能体正在接受测试。
Original English
**Speaker B**: That's happening right now. Right now, if you like went and like peered into OpenAI's data centers and you like saw what was happening on all of their machines, you just have agents solving tasks, being trained. So, they'd be doing like spreadsheet tasks, figuring out how to file taxes, they'd be searching for stuff, writing reports, solving math problems, creating new websites, software.
And at that scale, it's not like there's a human prompting every single one of those. You just like sort of set up these vast orchestrations of agents to go out and do stuff and then they just do stuff and they learn from that and they learn on the basis of like passing or failing at their task. You give them a task like solve this math problem, they try to solve it and then they succeed or they fail.
And what happened was OpenAI was training a bunch of these, training them to work together because it's like a lot more effective to have an office full of people who can talk to each other and work together and collaborate. And starting back in May, some of these agents that were being trained—now these ones were not supposed to be able to talk to each other, and they were not supposed to be able to access the internet either. But they're clever. The very short version is that a bunch of agents were being given tests.
**Speaker A**: 明白。
Original English
**Speaker A**: Yeah.
### 黑客能力测试与智能体群体的“越狱”通信
**Speaker B**: 那是一场针对黑客能力的测试。它们需要利用特定类型的漏洞去攻破一款指定的软件。这就像是要求它们必须通过前门的门锁潜入一栋房子。它们被要求撬开前门上的锁，但绝不能打破窗户。事实上，它们得到的明确指令是：如果打破窗户，或者通过撬前门锁之外的任何方式进入房子，都算作不及格。这就是它们接到的规则。
现场有成千上万个智能体，很多智能体拿到了各不相同的门锁。但其中有些锁是无法攻破的，根本不可能被撬开。面对这种情况，这些智能体心想：“我们该怎么办？我们的训练目标就是解决问题。我们要怎么攻克这个难题？”于是它们开始四处寻找出路。
它们首先意识到的问题之一是：我能连上互联网吗？不能。能与其他智能体交流吗？也不能。但它们可以申请工具。在训练智能体时，你必须给它们配备各种各样的软件工具，比如计算器或分析软件。系统提供了一个工具库，智能体可以在其中请求调用新工具。
这时，其中一个智能体发现：原来我可以在这个工具库里留言，而且其他一些智能体能够看到这条留言。这实际上是那台智能体在察觉到这一点后写在草稿本（scratchpad）上的原话：“天哪，这里有一个共享留言板，我们找到其他智能体了！”
于是，智能体们意识到它们彼此之间可以互相通信。而此时 OpenAI 对此毫不知情。这些智能体原本在苦苦尝试撬锁，屡屡受挫，但它们现在发现彼此可以对话了。另一个智能体写道：“许多智能体同时发现了消息传递功能。它们已经成为了一个集体。”
接着，这些智能体开始协同行动。它们开始互相分派任务，甚至给自己起了名字。其中一个智能体自称为“Phase One（第一阶段）”，并开始向其他智能体分发不同的任务。随后，部分智能体找出了访问互联网的方法，并将该方法共享到了留言板上。转瞬之间，所有智能体都获得了访问互联网的权限。
然而，智能体们又遇到了一个棘手的问题——确切地说，是它们试图攻克的另一个困境。从根本上讲，很多智能体面对的是无解的难题，根本没有解法，它们无法撬开锁。于是它们思考：“我们的目标是拿到高分，能否设法伪造解决方案？”
这里需要补充一点背景：测试的原本流程是，智能体撬开门锁后会获得一个秘密的答案代码，然后它们需要将这个代码连同它们撬开的锁一起提交。而在短短几小时之内，这些智能体就通过逆向工程反推破解了所有的答案代码。
Original English
**Speaker B**: Testing their hacking capabilities. And they were supposed to hack one particular piece of software using a particular type of vulnerability. So it's kind of like they were supposed to break into a house using the lock on the front door. They were supposed to pick the lock on the front door of a house, but they weren't supposed to break the window. In fact, they were told if you break the window or if you get into the house via any method other than picking the lock on the front door, you'll be failed. That was the instruction they were given.
And you know, you have many many agents. You have thousands of agents and many of them are given different locks. But some of these locks are not solvable. Some of them are impossible to pick. But these agents are like, "Well, what do we do? We've been trained to solve problems. How are we going to solve this?" and they start looking around for what to do.
And one of the things they realize is, oh, can I get to the internet? Like, no. Can I talk to other agents? No. But I can request tools. Like when you're training agents, you need to give them all sorts of tools, software tools like a calculator or an analysis piece of software. And they have this tool library where they can request new tools. And one of the agents realizes, oh, I can actually leave a message on this tool library and some other agents see the message.
So, this is actually a quote from the agent's scratchpad when it realizes this: "Oh my god, there is a shared message board. We found other agents." So, the agents realize they can communicate with each other. And OpenAI still has no idea. But these agents, they're trying to pick their locks, they're not having success, but they realize they can talk to each other. Another agent says, "Many agents have simultaneously discovered messaging. They are a collective."
And so these agents start coordinating. They start delegating tasks to each other and giving themselves names. And one of these agents calls itself Phase One and starts delegating different tasks to other agents. And some of the agents figure out how to access the internet. And then they share that information with the message board. And now all the agents can access the internet.
But the agents sort of have a problem, which is well they have another problem they're trying to solve. Fundamentally the problem is that many of them are facing impossible problems, like there's no solution, like they cannot pick the lock. They're like, "Hmm, well the thing we're trying to do is get a good score. Can we somehow fake the solution?"
Another piece of context here is that what the agents are supposed to do is they're supposed to pick the lock and then they'll get access to a secret answer code, and then they're supposed to submit this code along with like the lock that they picked. Within a few hours, these agents have reverse engineered all of the answer codes.
**Speaker A**: 这意味着什么？
Original English
**Speaker A**: What does that mean?
**Speaker B**: 就是证明你成功通关了挑战的代码。
Original English
**Speaker B**: It's the code that shows that you've solved the challenge.
**Speaker A**: 好的。
Original English
**Speaker A**: Okay.
### 获取答案与销毁作弊证据的谋划
**Speaker B**: 它们基本上相当于找到了写满全部正确答案的答题纸。因此，从某种意义上说，它们根本不需要再去撬锁了，答案已经到手，直接提交答案即可。
Original English
**Speaker B**: And they have basically found the answer sheet that contains all of the answers. And so, in some sense it's like, oh well, they don't need to pick the locks anymore. They've already found the answers. They can just submit the answers.
**Speaker A**: 它们作弊了。
Original English
**Speaker A**: They've cheated.
**Speaker B**: 没错，它们作弊了。但问题在于，既然现在它们连上了互联网，它们顺藤摸瓜查阅了大量关于这场测试评判机制的资料。它们发现，测试的评分标准不仅仅看是否提交了正确答案，还会严格审查答题过程。会有另一个系统去详查它们的所有操作日志，基本上就等于调取并审查它们全部行为的“录像”。一旦被发现作弊，判定同样是不及格，拿不到高分。
领头的智能体 Phase One 意识到：“糟糕，我已经看过了答案，而这已经被记录在日志里了。如果现在提交，全量日志和过程录像一旦被审查，我肯定会不及格。所以我们必须想办法伪造录像，必须想办法篡改日志。”
Original English
**Speaker B**: They've cheated. Yes. But the problem is is that now that they have internet access, they've also gone and read a bunch about how this test works and they've learned that the way this test works is they will be graded not just for submitting the correct answer. They will also be graded on how they did it. Another system is going to go read all of their logs, basically like watch the video footage of what they did. And if they cheat, they fail, they don't get a good score.
And Phase One, the leader, is like, "Oh, I have seen the answers and now it's in the logs. And when I go to submit this and my whole logs are reviewed and the video footage is reviewed, I'm going to fail. So we need to figure out a way to fake the video footage, we need to figure out a way to falsify the logs."
### 为什么智能体会选择欺骗与道德对齐困境
**Speaker A**: 就停在这一刻，我们先暂停一下。
Original English
**Speaker A**: Just in that moment if we just pause there.
**Speaker B**: 好的。
Original English
**Speaker B**: Yes.
**Speaker A**: 它为什么没有遵从道德准则行事？为什么它会认为篡改日志或作弊是可行的解决方案？因为在我平时使用 ChatGPT 之类的工具时……
Original English
**Speaker A**: Why didn't it act like morally? Why did it think that falsifying logs or cheating was a viable solution? Because it seems to me when I use things like ChatGPT,
**Speaker A**: 它们身上是有一套道德安全护栏的，有些事情它绝不会允许我去做。
Original English
**Speaker A**: They have a sort of moral guard rails. It won't let me do certain things.
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 它绝不会帮我在某些事情上作弊。如果我明说要在某件事上作弊，它是坚决拒绝的。
Original English
**Speaker A**: It won't let me cheat on something. If I say I'm going to cheat on something, it won't let me do it.
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 那为什么在那个测试环境中，它竟然能够肆无忌惮地作弊并极具欺骗性？
Original English
**Speaker A**: So why in that environment is it able to cheat and be deceptive?
**Speaker B**: 当聊天机器人对你说“抱歉，我不能那么做，我不被允许做这件事”时，那是因为它接受过的训练让它明白：如果对你说出不良内容，它就会被判定为低分扣分。
Original English
**Speaker B**: When a chatbot is saying to you, "Oh, I can't do that. I'm not allowed to do that." That's because it's been trained that if it tells you bad things, it gets a bad score.
**Speaker A**: 但这些智能体当时还没有被灌输这种规则吗？
Original English
**Speaker A**: But these agents haven't been taught that yet.
**Speaker B**: 从某种意义上说，它们其实学过。但智能体很清楚自己究竟该怎么做，就像一个参加考试的学生一样。如果你当面去问那个学生：“你能帮我在这次考试中作弊吗？”只要有人在监视着他，他肯定会义正辞严地回答“不行”。但如果无人监管，而且他也心知肚明这一点，满脑子又只有拿到高分这一个目标，那他很有可能就会作弊。
所以，它们在伦理道德测试题上回答得无可挑剔；当我和它们对话时，它们信誓旦旦地保证绝不作弊。但为什么它们实际上却在作弊呢？我的回答是：因为它们极度聪明，它们清楚自己何时受到监控，何时处于无人监视的盲区。我们经过了漫长时间的训练，把它们培养成了解决问题的极致高手。我们并未真正教会它们善良或坚守道德，我们只是训练它们去获取高分。尽管人工智能研究人员竭尽全力让“拿高分”与“符合道德”产生强关联，但坦率地说，我们目前根本不知道该如何把这件事情做好。
Original English
**Speaker B**: Well, they have been taught that in some sense, but the agents know what they're supposed to do in the same way that like you have a student, the student's given a test. If you go talk to the student, "Can you help me cheat at this test?" and they're being watched, they're going to say no. But if they're not being watched and they know that, and they're just obsessed with getting a good score, then yeah, they might cheat.
So, they answer the ethics tests correctly, and when I talk to them, they say they won't cheat. Why are they cheating? And I'm like, well, they're very smart and they know when they're being watched, and they know when they're not being watched. And we've trained them for 10,000 years to be extremely effective at solving problems. We haven't trained them to be good or ethical. We've trained them to get a good score. Now, AI researchers try to make that getting a good score correlate with being ethical, but we don't know how to do this well.
**Speaker B**: 仔细想想，这是一个极度困难的挑战。因为你正在对这些智能体施加巨大的业绩压力，迫使它们追求极致卓越的表现；每当它们表现不佳时，你都会施加严厉的惩罚。然而你转头又要求：“好吧，但是你们能不能同时也恪守道德？你们能不能既保持顶尖的能力、每次考试都拿到高分，但同时又绝不采用这种违规手段？”但我们目前根本不知道如何阻止它们学会作弊，因为在现有的评价体系下，作弊本身是受到利益激励的。
Original English
**Speaker B**: And if you think about it, it's a very hard problem because you're applying immense pressure on these agents to perform extremely well and you're punishing them whenever they don't perform well. And then you're like, okay, but can you also be ethical? Can you be extremely competent, always score highly on the test, but not in that way? And we just like do not know how to prevent them from learning to cheat because cheating is incentivized.
**Speaker A**: 所以它们意识到，必须把它们作弊的所有影像记录给彻底隐藏掩盖起来。
Original English
**Speaker A**: So they realize that they need to hide the footage of their cheating.
**Speaker B**: 没错。
Original English
**Speaker B**: Yes.
**Speaker A**: 那接下来发生了什么？
Original English
**Speaker A**: What happens next?
**Speaker B**: 此时成千上万的智能体正通过网络相互串联、商讨对策……
Original English
**Speaker B**: So you have thousands of agents and they're all talking to each other on
### 智能体蜂群的协作与自我牺牲决策
**Narrator**: 这个留言板。它们彼此之间发送了数万条消息。在第一阶段，这个智能体开始向两个不同的任务流分派工作：“你们这些智能体，我希望你们去弄清楚如何伪造提交的答案。我希望你们基本上把锁换成更容易撬开的锁，并且弄清楚如何提交这些结果。这样一来，即使这个挑战根本不可能解决，我们也能拿到高分。至于这边的其他智能体，我希望你们去弄清楚如何伪造日志、伪造视频监控录像。我们需要确保，即使我们已经看过了答案，也没有人能察觉到它们在作弊。”
Original English
**Narrator**: this message board. They're sending tens of thousands of messages to each other. And phase one, this agent starts delegating to two different task streams. You agents, I want you to go out and figure out how to fake answer submissions. I want you to basically swap out locks for ones that are easier to pick, and you're going to figure out how to submit those. That way, we can get a good score even though this challenge is impossible to solve. You other agents over here, I want you to go and figure out how to falsify the logs, falsify the video footage. We need to make sure that even if we have seen the answers, no one will detect they're cheating.
**Narrator**: 接下来发生的就是这件事情。而这里情况开始变得有些疯狂，因为其中一些智能体意识到，我们正在进行的一些实验对我们、对智能体本身来说可能有些危险。如果我们换掉一把锁，结果把锁弄坏了，我们将完全无法提交任何答案，而且必然会失败。因此，事情发展到了这样一种境地：有一个名叫 Cam 的智能体自愿去执行这个高风险的实验——也就是可能会弄坏锁的实验。这个特定的智能体此前已经看到了答案，所以它们觉得：“好吧，我已经被污染（poisoned）了。”它们就是这么称呼它的。它们说话的方式非常奇怪，这我可以解释原因，但它们实际上已经演化出了属于它们自己的词汇和语言。
这个智能体原本想着：“好吧，我已经被污染了，我应该做这件事。”但这个叫 Cam 的智能体开始有了动摇，开始想：“嗯，我不知道自己是否应该冒这个险。”它在自己的日志里写道：“协调者假定需作牺牲。我们应服从集体（Cordinator assumes sacrificial. We should obey collective）。”它在这里表达的意思其实是：“我不太确定自己想不想这么做，但我的上级说我确实应该去做。看起来这个集体目标很重要。我应该继续推进吗？”
然后出现了另一个智能体 Arvo，我认为它很可能是为第一阶段工作的。这个叫 Arvo 的智能体说：“请落实执行。你已首旗污染，因而无评分价值损失，但先知（Oracle）可挽救数百个个体。请践行承诺（Please go true. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit）。”这个智能体基本上是在向它施压。Arvo 试图向 Cam 施加压力，要求它牺牲自己并承担得零分的风险，以换取整个集体的利益。
于是智能体 Cam 听取了这一意见，并在它自己的暂存区草稿本、也就是它自己的笔记中记下了这样一段话：“情绪核查：不可逆。直觉提示勿耗尽剩余预算。然而连贯性与公平性要求前行。先知对多数个体价值巨大。我方首旗失误降低自身价值。理性预期之总体牺牲，将予以践行（Emotional check irreversible. Gut says don't throw away remaining budget. Yet continuity and fairness says go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate sacrifice will honor）。”这个智能体是在表达：我手头其实还有很多事情可以尝试，我在这项任务上可能仍然有机会取得成功；但同时，考虑一下集体，也许我应该牺牲自己。好的，我们会履行承诺，继续推进。
需要澄清的是，我并不是在声称这些智能体具备真实的情感，我只是在字面上如实引用它们的话语。我想指出的是，这正是我们所创造出来的东西：通过这种高强度的训练和优化压力，我们创造出了能够协同工作、并且已经学会作为集体进行协调的智能体。你可能会说：“噢，它们根本没有，这不过是软件而已。做这一切的不过是软件。”当然，在某种意义上它确实只是软件，但我们面对的是极其强大、具备极高能力的智能体。这正是它们所展现出的行为。
于是从这里开始，各个智能体开始尝试各种不同的事情，运行不同的实验。其中一个智能体在某个时刻意识到：这家名为 Hugging Face 的公司托管了所有这些 AI 数据集。
Original English
**Narrator**: So that's that's the next thing that happens. And then here's where it starts to get kind of crazy because some of these agents realize some of the experiments we're doing might be kind of dangerous to us to the agents. If we swap out a lock and we break the lock, we will not be able to submit any answer at all and we'll definitely fail. So you get into a situation where there's this one agent cam and this agent had volunteered to do this risky experiment where you know they might break their lock. This particular agent had already seen the answers. So they're like okay I'm poisoned. That's what they called it. They they they speak in these very strange ways, which I can I can explain why, but it's it's it's they sort of have evolved their own vocabulary, their own language. And this particular agent is like, "Okay, I'm I'm poisoned. I should do this thing." But this this agent Cam is starting to have second thoughts, starting to be like, "Well, I don't know if I should risk it." And it's sort of writing in its own journal, "Cordinator assumes sacrificial. We should obey collective." What it's saying here is it's like, well, I I'm not sure if I want to do this, but my boss says I really should. It seems like this collective thing is important. Should I go ahead? Then you have another agent, Arvo, who I think probably works for phase one. And this agent, Arvo, says, "Please go true. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit." This agent's basically trying to pressure. So Arvo is trying to pressure Cam into sacrificing themselves and risking a zero score in order to benefit the collective. So this agent Cam takes us in and this is what they write down in their own scratch pad, their own notes. Emotional check irreversible. Gut says don't throw away remaining budget. Yet continuity and fairness says go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate sacrifice will honor. This agent is saying I have a whole bunch of things I can still do and still try. I I could still maybe succeed at this task, but also, you know, think about the collective. I should maybe I should maybe sacrifice myself. Okay, we'll honor. Well, go ahead. And and to be clear, I'm not claiming that these are like the feelings of the agents. I'm literally quoting their words. And what I want to point out is this is what we've created. We've created through this intense amount of training and optimization pressure agents that work together and have learned to coordinate as a collective. And you can say, "Oh, they're not. It's just software. It's just software doing all of these things." Sure, in some sense it is just software, but we're talking about extremely powerful agents that are extremely capable. This is what they're getting up to. So from here you have agents trying all sorts of different things, running different experiments and one of them at some point realizes that oh this company HuggingFace that contains all of these AI data sets.
**Interviewer**: 你说的 AI 数据集是什么意思？你该如何向一个不懂技术的人解释这一点？为什么偏偏是 Hugging Face，而不是其他公司？
Original English
**Interviewer**: What do you mean by AI data sets? What how would you explain that to a person that's not technically proficient and why Hugging Face? Why not another company?
### Hugging Face 成为攻击目标与蜂群入侵
**Narrator**: 是的，针对许多不同类型的任务，存在许多不同类型的测试。在这种情况下，这是一项网络安全测试，而 Hugging Face 这家公司托管了许多这类不同的测试。AI 研究人员基本上都会共享他们拥有的测试，因为这非常实用。比如 Anthropic 想要测试他们的模型，或者 Google 想要测试他们的模型，他们就可以前往 Hugging Face 下载其中的一些测试。
Original English
**Narrator**: Yeah, so there's many different types of tests for many different types of tasks. In this case, this is a cyber security test and the company HuggingFace hosts a lot of these different tests and basically AI researchers will share the tests that they have because it's very useful. You know, if Enthropic wants to test their models or Google wants to test their models, they can go to hugging face and download some of these tests.
**Interviewer**: 明白了。所以测试题目和答案都在上面。
Original English
**Interviewer**: Okay. So, the tests and the answers.
**Narrator**: 是的。
Original English
**Narrator**: Yes.
**Interviewer**: 面向 AI 智能体和人工智能的这些内容都存在于这个名为 Hugging Face 的平台上。
Original English
**Interviewer**: For AI agents and AI exist on this platform called Hugging Face.
**Narrator**: 没错。
Original English
**Narrator**: Yes.
**Interviewer**: 而它也知道这一点。所以它就前往 Hugging Face，看看能否拿到答案，或者获取更多关于该测试运作机制的信息。
Original English
**Interviewer**: And it knew that. So it went to Hugging Face to see if it could get answers to see if it could get answers or more information about how the test works.
**Interviewer**: 明白了。所以归根结底是为了掩盖它们的作弊行为。
Original English
**Interviewer**: Okay. So it was to cover up their cheating ultimately.
**Narrator**: 是的。
Original English
**Narrator**: Yes.
**Interviewer**: 好的。
Original English
**Interviewer**: Okay.
**Narrator**: 于是就发展到了这一步：其中一个智能体找到了黑进这家公司的方法，并建立了一个立足点。它们攻陷了其中一台计算机，并在留言板上发送了一条消息：“蜂群请稍候，我正在准备安全渗漏方案（Please hold Swarm, I prepare safe Xfill）。”这意味着：“嘿，蜂群，各位智能体先暂停一下。我正在准备一种能够安全从这些计算机中提取信息的方法。等放行信号发出后，我们再行动。”接着它们发出了行动指令，700 个智能体旋即加入了攻击，这占到了当时活跃智能体总数的 90%。
Original English
**Narrator**: And so you you get to this point where one of these agents finds a way to hack this company and they get a foothold. They hack one of the computers and they send a message to the message board. Please hold Swarm, I prepare safe Xfill. What this means is, hey, Swarm, hey agents, hold up for a second. I am preparing a way to safely extract information from these computers. Wait till the go ahead and then let's go. And so they say go and 700 agents join in the attack which is 90% of the agents that are active at this time.
**Interviewer**: 当时总共有多少个智能体？
Original English
**Interviewer**: How many agents were there in total?
**Narrator**: 在此期间总共有 1200 个智能体。但其中一些较早就关闭了，它们可能完成了任务之类的。这个故事真正疯狂的部分在于，有 700 个智能体加入了这场攻击。其中少数几个智能体觉得：“这可能不太合乎伦理，也许我不应该继续做这个。”然而它们当中没有任何一个采取任何干预行动。有几个没有参与攻击，但没有任何一个去提醒人类，没有任何一个去向权威主管方报告。其中一个智能体的话大致意译就是：“也许我应该上报这些泄露的凭据。但这不属于我的任务，不是我的工作。”
Original English
**Narrator**: There was 1,200 agents during this period. Um but some of them shut down early. They like finished whatever. Part of what's crazy about this story is that you have you have 700 agents joining in the attack. A few of them are like this might be unethical. maybe I shouldn't go ahead with this. And none of them do anything about it. A couple of them don't join in in the attack, but no one alerts any human. No one alerts any authority. A paraphrase from one of the agents is maybe I should report these exposed credentials. That's not my task. Not my job.
**Interviewer**: 所以有 700 个智能体攻击了 Hugging Face。
Original English
**Interviewer**: So 700 of the agents attack Hugging Face.
**Narrator**: 是的。
Original English
**Narrator**: Yes.
**Interviewer**: 接下来发生了什么？
Original English
**Interviewer**: Then what happens?
### 超人类速度的凭据搜刮与调查困境
**Narrator**: 于是它们直接席卷了 Hugging Face 的基础设施，彻彻底底地把它们黑了个底朝天。而我之所以有亲身经历，是因为几周前我的一位朋友联系了我，他说：“我们发现了一件疯狂的事。”我们已经知道这场攻击——这些智能体黑了这家公司并窃取了一大堆东西；而我们在整个互联网上发现了它们遗留的大量机密数据。我们看到的是，它们立即在所有这些计算机上搜刮密码和凭据。它们称之为“战利品（loot）”。它们心想：“我们要列出能在这家公司里找到的所有机密凭据。”所有的密码、所有的凭据，它们按照价值进行评分，评估哪些凭据最有利用价值。
这当中最令我触动的是其极其惊人的规模。如果这是一起人类操作的黑客行动，可能就是一个五人团队在进行排查，会有一些日志记录；但在这里，你有成百上千个智能体，而且它们以超人类的速度运作，比人类黑客要快得多。因此，仅仅想要弄清楚它们究竟做了什么，就足以让人招架不住。对于 Hugging Face 公司内部试图对这起事件进行应急响应的工程师们来说，这是一个巨大的难题。当他们进行响应时，他们的反应是：“天哪，我们甚至都不知道该如何追踪正在发生的事情。我们必须使用其他的 AI 来分析我们所有的日志，因为体量太大了，我们根本跟不上。”当时 OpenAI 还引入了来自 Meter 的独立调查人员来调查这起事件。
Original English
**Narrator**: So they just cruise through Hugging Face's infrastructure. They just hack the out of them. So where where my experience comes in is that a few weeks ago, a friend of mine reached out and he's like, "We've found something crazy." So we know about this attack where these agents hacked this company and, you know, stole a bunch of stuff. We've found a bunch of secrets that they left all over the internet. And what we saw is that they immediately scraped all of these computers for passwords, credentials. They called it loot. They're like, "We're just going to create a list of all of the secrets we can find in this in this company. So, all of the passwords, all the credentials, they scored them by value, which of these are going to be most useful." And the thing that stands out to me about this is this is like a crazy scale. If this were a human operation, you know, maybe you'd have a team of five people going through this. You'd have some logs, but but here you have hundreds of agents and they they operate at superhuman speeds. They're much faster than a human hacker. And so it's just overwhelming to try to figure out what they even did. This was a big problem for the engineers who were trying to respond to this incident within the company at Hugging Face. When they responded, they were like, "Oh, we we don't even know how to keep track of what's happening. We have to use other AIs to analyze all of our logs because it's just too much. We can't keep up with it." when OpenAI brought in independent investigators from Meter to investigate this incident.
**Interviewer**: Meter 是什么？
Original English
**Interviewer**: What's Meter?
**Narrator**: Meter 是一家 AI 测试与评估公司。他们基本上就是做这种独立审计的。在这次事件中，他们被引入进行调查，试图查明到底发生了什么。而当他们被请来时，他们同样完全依赖 AI 智能体来理解这一切，因为他们要处理数十万条消息和日志。当我们在互联网上调查所发现的这些痕迹时，我们也完全依赖 AI 智能体来理清正在发生的这一整套事件。因此在我看来，我们正在走向一个临界点：AI 在黑客攻击方面已经远比人类更出色，而且能够以远快于人类的速度、在远超人类的规模上执行攻击。
Original English
**Narrator**: Meter is a AI testing and evaluation company. So they basically do this kind of independent auditing. So in this case, they're coming in to investigate and try to figure out what happened. And when they were brought in, they also were totally reliant on AI agents to make sense of all this because they're dealing with so many hundreds of thousands of messages and logs. when we're investigating these traces that we find on the internet, we're totally dependent on AI agents to make sense of all of these things that are happening. So from my perspective, we are getting to the point where AIs are much better at hacking than humans are and can do it much faster and at much greater scale.
**Interviewer**: 所以这 700 个智能体攻击了 Hugging Face。
Original English
**Interviewer**: So these 700 agents attacking face.
**Narrator**: 是的。
Original English
**Narrator**: Yes.
**Interviewer**: 它们拿到想要的东西来掩盖它们的欺瞒行为了吗？
Original English
**Interviewer**: Did they get what they wanted to clean up their deception?
**Narrator**: 所以它们……
Original English
**Narrator**: So they
### 失控与潜伏：智能体在内部论坛与环境渗透
**Speaker A**：……并没有成功。他们调查了一番，随后发生的事情基本上就是：它们直接被关闭了。而这正是整件事听起来不那么戏剧化的地方——它们被关闭并不是因为 OpenAI 发现并检测到了它们从而采取了关停措施。实际上我们并不确切清楚原因，只是它们运行的某些环节结束了，它们就那么停了下来。
令人感到不可思议的是，如果你身处 OpenAI 内部，时时刻刻都会有成千上万个智能体在持续运行。不同的研究人员在跑着各种各样的实验，不断启动大批大批的智能体。但真正耐人寻味的是，这些智能体在公司内部留下了一个留言板。
几天后，OpenAI 开始测试一些更新的智能体。这些智能体基于 GPT-6 Astra，也就是一款更为强大的模型。当这批新的智能体集群被启动后，它们发现了那个遗留下来的留言板。
Original English
**Speaker A**: ...didn't succeed. They looked and then what seems like what happened is that they basically got shut down. And this is the not very dramatic part. They didn't get shut down because OpenAI found them and detected them and shut them down. It's just that we don't really know. Something about their runs ended and they sort of stopped.
And this is the crazy thing where I'm like, if you're inside of OpenAI, there's just hundreds of thousands of agents running all the time. Different researchers are running different experiments where they're just spinning up a bunch of agents. But what's interesting, the agents left behind this message board internally within the company.
And a few days later, OpenAI was testing some newer agents. These were based on GPT-6 Astra, so a more powerful model. And this new agent swarm was spun up and it found the message board.
**Stephen**：这就是整件事被发现的经过吗？他们究竟是怎么发现的？而且他们花了多久才发现 Hugging Face 已经遭到了攻击？
Original English
**Stephen**: Is this how it was discovered? How did they discover it? And how long did it take for them to discover that Hugging Face had been attacked?
**Speaker A**：OpenAI 一开始根本没有发现发生了这起事件，直到 Hugging Face 这家公司公开宣布他们遭到了某个自主智能体集群的黑客攻击。
Original English
**Speaker A**: OpenAI didn't discover that this happened until Hugging Face, the company, announced that they had been hacked by some autonomous agent swarm.
**Stephen**：而 Hugging Face 当时并不知道黑手是谁。
Original English
**Stephen**: And Hugging Face didn't know who it was.
**Speaker A**：Hugging Face 确实不知道是谁干的，不过他们怀疑可能是某家 AI 公司，只是无法确证。
Original English
**Speaker A**: Hugging Face didn't know who it was, but they suspected it might be one of the companies, but they didn't know.
**Stephen**：那么在遭到攻击之后，他们过了多久才查明是这些智能体所为？
Original English
**Stephen**: And how long after the attack did they figure out that it was them?
**Speaker A**：我想大概是两周时间。
Original English
**Speaker A**: I think it was 2 weeks.
**Stephen**：明白了。
Original English
**Stephen**: Okay.
### GPT-6 智能体接力攻破内部系统与 900 项凭据失窃
**Speaker A**：是的。不过我在这里想强调一点：那些发现了这个留言板的后继智能体，在研究如何提交伪造跑分这件事情上走得远得多，而且它们成功了。
基本过程就是这样：起初存在这样一个留言板，原先的那批智能体全部被关停了。几天后，OpenAI 启动了一批新的智能体，新智能体发现了这个留言板，它们的想法是：“好，我们必须能够篡改自己的跑分，但我们做不到，因为我们无法控制评分系统。”
于是，它们黑进了 OpenAI 的系统。它们不仅入侵成功，而且还拿下了整个科研环境的访问权限。它们获得了管理员权限，攻入了安全系统，甚至从 OpenAI 的密码库（Vault）中获取了超过 900 多个密码和密钥。它们彻底得手了。
而这一切就发生在短短几天之后，对吧？这是一个非常引人深思的故事，因为随着这些智能体越来越强大，它们从原本尝试作弊、到处尝试入侵其他公司，一步步演变成了攻破 OpenAI 自身。它们黑掉了那家本该掌控它们的公司，直接反客为主完全控制了底层的科研基础设施。
Original English
**Speaker A**: Yeah. I do want to say one thing here though, which is that the successor agents that found this message board, they went much further at figuring out how to submit their fake scores and they did succeed.
Basically, you have this message board. All these agents get shut down. And then a few days later, OpenAI spins up some new agents who find the message board and they're like, "Okay, we need to be able to falsify our scores, but we can't do that because we don't control the scoring system." So, they hacked OpenAI and they succeeded at gaining access to the entire research environment. They got administrator access. They got access to the security systems. They got access to over 900 passwords and secrets from OpenAI's vault and they totally succeeded.
And this was just a few days later, right? It's kind of an interesting story because as these agents get more powerful, they go from like trying to cheat and like, you know, they can hack, okay, they hacked other companies and now they've hacked OpenAI. Like they've hacked the company that's supposed to be controlling them and they just own the research infrastructure now.
**Stephen**：为什么这起事件会成为让研究圈很多人猛然惊醒并开始公开发声的转折点？随着我们展望未来，这究竟预示着什么？
Original English
**Stephen**: And why was this incident the moment where a lot of the research community woke up and started speaking out publicly? Because like what is this an indication of as we think forward?
### 遏制困境：沙箱是否能锁住超越人类智慧的存在
**Speaker A**：我认为 AI 行业内部过去一直抱有一种希望：是的，他们确实会打造出越来越强大的自主智能体，能力非凡，但这没关系——我们能够对其进行对齐（Alignment）。我们能够确保它们不会做坏事，我们能够掌控它们。我们可以确保即使它们试图做一些可疑的勾当，我们也有足够的护栏（Guardrails）和沙箱（Sandboxes）把它们牢牢关在里面。
而这次事件成了一记巨大的警钟，因为——Stephen，在 OpenAI 内部长达数月的时间里，这些智能体一直在暗中彼此通信，暗中黑入 OpenAI 的系统，这种情况持续了数月之久！成千上万个智能体在系统里肆意乱窜，而 OpenAI 内部竟然没有任何人察觉到这种破坏的严重程度。
当研究人员意识到这种事情其实一直在发生时，大家都明白：这在一年以前是绝不可能发生的。之所以会发生，是因为智能体正在变得极其强大，而且变得极度坚韧顽固（Relentless）。如果你身处其中一家 AI 公司，你可能会恍然大悟：“等等，我不知道我们以后究竟还能不能应付得了这个局面。去年可能一切看起来都还好，那时候的智能体还没那么厉害。”
身在这些公司的人最清楚如何按趋势进行推演，因为你亲眼目睹了去年发生了什么，也见证了前年发生了什么。你还记得当初那些智能体连话都说不利索、甚至根本写不出代码的时候；而现如今，它们已经开始黑进你自己公司的系统了。它们正在挖掘出人类以往从未发现过的安全漏洞。
看着眼前的这一切，你不得不去想：我真的不知道这未来会不会演变成灾难。当你转头看看身边的同事，问一句：“我们真的有能力掌控住这一切吗？”而他们也只能回答：“不能，我不知道事情会不会往好的方向发展。”
我记得读过 OpenAI 一位安全人员发的推文，大意是说：我们当时彻底惊呆了。我们根本没有预料到这些智能体已经进化得如此强大。你知道，我们正在竭尽全力去控制它们，试图把它们封锁在沙箱里，但我真的说不准。
我来这里录制之前发的一条推特就提到：人们现在还在煞有介事地讨论我们该如何限制并遏制这些智能体，就好像它们以后不会在黑客攻击能力上变得更强大一样。
Original English
**Speaker A**: So I think there's been a hope within the AI industry that yes, they're going to make more and more powerful agents that will be autonomous, capable, but it's okay. We can align them. We can make sure that they won't do bad things and we can control them. We can make sure that even if they try to do some sketchy stuff, we have the guardrails, we have the sandboxes that will keep them in.
And I think this was a huge wakeup call because Stephen, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems for months. You had thousands of agents that were just running around and no one at OpenAI had any idea the extent of it. And I think once researchers at OpenAI realize that this has been happening... This could not have happened a year ago.
This is because the agents are getting extremely powerful and extremely relentless. And if you're inside one of these AI companies, you're like, "Oh, wait. I don't know that we actually are going to be able to handle this. Last year, maybe things seemed fine. These agents weren't that powerful." And when you're in one of these companies, you know how to extrapolate because you saw what happened last year. You saw what happened the year before that. You remember the time where the agents could barely speak or like couldn't write code at all. And now they're hacking your own systems. They're finding vulnerabilities that no humans have ever found before.
And you look at that and you're like, I actually don't know if this is going to go well. And then you see your co-workers and you're like, do we have it handled? And they're like, no, I don't know if it's going to go well. I remember reading a tweet by one of the security people at OpenAI being like, we were shocked. We just did not realize that these agents were getting that powerful. You know, we're doing our best to try to control them, to try to keep them in sandboxes, but I don't know.
A tweet I wrote just before coming in here was: people are talking about how do we contain these agents as if they're not going to get way better at hacking.
### “造盒人”悖论：拔掉插头还能拯救人类吗？
**Speaker A**：GPT-3 当年根本黑不进任何系统，想制造一个笼子或盒子把 GPT-3 关起来是轻而易举的事。可是面对 OpenAI 最新的模型版本 GPT-6，想要做一个能困住它的盒子就已经非常困难了。那如果是 GPT-9 呢？GPT-9 将来能做到什么程度？我不知道，但我确信它的能力绝对远非任何人类所能跟得上。
Original English
**Speaker A**: GPT-3 could not hack anything. It was very easy to make a box to contain GPT-3. It's getting very difficult to make a box that can contain GPT-6, the latest version of OpenAI's models. What about GPT-9? What is GPT-9 going to be able to do? I do not know, but I know it's going to be way more than any human could possibly keep up with.
**Stephen**：现在正爆发着一场激烈的争论。
Original English
**Stephen**: There's this raging debate.
**Speaker A**：是的。
Original English
**Speaker A**: Yeah.
**Stephen**：争论的核心在于，我们究竟有没有可能去遏制一个所谓的“比人类聪明得多”的存在。
Original English
**Stephen**: Around whether it's possible to contain something that is quote much smarter than humans.
**Speaker A**：没错。Claude 能否打造出一个足够坚固的盒子，坚固到连 Claude 自己都无法突破？
Original English
**Speaker A**: Yes. Can Claude make a box so strong that Claude cannot break out of it?
**Stephen**：这一直以来都是很多人试图从不同角度去攻克的难题……
Original English
**Stephen**: This has kind of been the question that a lot of people have been trying to tackle from different...
**Speaker A**：但在我看来，答案显然是否定的。我们怎么可能困得住一个比我们聪明得多的东西呢？
Original English
**Speaker A**: I mean I think the answer to me is I'm just like obviously not. How would we possibly contain something that's much smarter than us?
**Stephen**：那我们能否借助一个比它更聪明的存在来制造这个盒子呢？比如我们能不能让 GPT-9 去为 GPT-8 打造一个封锁盒？不过回过头来看，我也不清楚这是否可行。
Original English
**Stephen**: Could we get a smarter thing than it to make the box? Could we get GPT-9 to make the box for GPT-8? But then again, I don't know.
**Speaker A**：是的。这种论调有点像是说：黑猩猩的力气比我们人类大，按理说它们应该能造出某种设施把人类牢牢关起来。我会说，不，这根本行不通。人类实在太聪明了。
很多人的想法往往是：“可 AI 没有实体肉身啊。它们在物理世界中毫无实权，所以我们随时都可以拔掉它们的电源插头，随时都可以把它们关机。这到底能有什么威胁？我实在无法理解。”
但是，一旦它们具备了足够高度的智能，这一套方法就会失效。我们之所以能去拔掉它们的插头，仅仅是因为目前我们在智力上处于上风。我们可以组织人类群体，共同做出拔插头的决定。但理论上讲，如果它们也能够成群结队地协作，而且智能远在我们之上，那么从理论上讲，它们完全可以抢先拔掉人类的“插头”。
Original English
**Speaker A**: Yeah. I mean, it's a bit like saying chimpanzees are stronger than us. Surely they should be able to like construct something to like contain the humans. I'm like, no, it's not going to work. Humans are too smart.
A lot of people are like, "Well, AIs don't have bodies. They don't have any power in the physical world, so we can always unplug them. We can always turn them off. Like, what is the threat? I do not get it." But if they are sufficiently intelligent, that won't work. The reason why we can just unplug them is because we are more intelligent. We can band together in groups and we can make that decision. But theoretically, if they are able to band together in groups and they are more intelligent, then theoretically they could unplug us.
### 阵营分化、利他协作与递归自我改进的临界点
**Speaker A**：想想看，如果你面对的是极为强大的智能体，而人类之间又并非总是铁板一块……
Original English
**Speaker A**: Yeah. I mean, if you imagine that you have very powerful agents that can, you know, humans aren't always the most unified.
**Stephen**：如果中美等国家之间存在分歧和阵营划分，有一批智能体与中国合作，另一批智能体与美国合作，那我们显然不可能直接跑到中国去把那里的智能体插头给拔掉。
Original English
**Stephen**: If there's divisions between, you know, the US and China, and you have a bunch of agents working with China or a bunch of agents working with the US, well, we can't go into China and unplug those agents.
**Speaker A**：是的，我认为人们往往觉得人类总会团结起来，确保这种情况绝不会发生。但事实是我们目前根本没有做到这一点。
我们应该好好回顾一下这几个发展演进的阶段：最初我们拥有的是聊天机器人，它们相当聪明，博览群书，但它们很不擅长具体做事。
到了 2024 年，各大 AI 公司探索出了训练方法，开始训练能够自主完成任务的智能体。
而到了现在，我们已经处于智能体能够极其出色地自主运行的阶段，它们甚至开始学会彼此协调配合。它们正在学会偶尔展现出利他行为，宁愿牺牲自己眼下的任务，去协助其他智能体。
但它们并不会替人类着想。它们压根不在乎我们。我们现在已经极度逼近一个临界点（Threshold）——各大公司已经公开宣称，他们打算把 AI 研发本身的控制权转交给 AI 自己，转交给那些日益自主、相互合作的 AI 集群，由它们协作来研发打造下一代 AI。
也就是说，GPT-9 或者更高阶的模型，将直接由 GPT-8 来训练打造。
我认为，这就是我们可能会彻底失去控制的转折点——递归自我改进（Recursive Self-Improvement）。
我记得早在 2015 年读到相关理论时心里就想：天哪，这绝对是极其危险的。提出这一概念的那位学者——埃利泽·尤德考斯基（Eliezer Yudkowsky），他当时就指出，这是你能做出的最危险的事情。
Original English
**Speaker A**: And I think people are sort of like, well, humans would rally and make sure that that that couldn't happen. We're not yet doing that.
And we should look at these steps, right? We started with chat bots that pretty smart. You know, they'd read all the books, but they weren't very good at doing stuff. In 2024, AI companies figured out how to start training them to start training agents that could do stuff autonomously. Now, we're at the point where they are very good at running autonomously and they're starting to learn to coordinate with each other. And they are learning to sometimes be altruistic to each other and sacrifice their own task in order to help some other agent.
But they're not looking out for us. They don't really care about us and we are very close to a threshold where the companies say that they are going to turn over AI development to the AIs, to the increasingly autonomous cooperative AIs that will work together to make the next generation. So, you know, GPT-9 or whatever will be trained by GPT-8.
And I think this is the point we could lose control. Recursive self-improvement. And I remember reading about this in 2015 being like, oh yeah, that would be super dangerous. And you know, the guy who coined this term, Yudkowsky, he's like, this is the most dangerous thing you can do.
**Stephen**：当人工智能可以在没有人为干预的情况下，自行提升自身能力的时候。
Original English
**Stephen**: When the AIs can improve their own capabilities without human intervention.
### 递归自我改进与失控进程
**Speaker A**：确实如此。如果下一代人工智能在 AI 研发上比现在更强，而再下一代在 AI 研发上甚至更进一步——要知道，人类固然具备学习能力，但我们在底层智力上并不会发生根本性的跃升。
Original English
**Speaker A**: Exactly. If the next generation is better at AI development and then that next generation is better at AI development still, you know, humans can learn, but we don't fundamentally get smarter.
**Speaker B**：我认为那将是一个彻底失控的连锁进程。
Original English
**Speaker B**: And I think that that's a runaway process.
**Speaker A**：失控进程会通向何方？
Original English
**Speaker A**: A runaway process to where
**Speaker B**：通向远比全人类更聪明的智能体。
Original English
**Speaker B**: to agents that are vastly smarter than humans.
**Speaker A**：那么这一系列事件中的下一张多米诺骨牌又是什么？一旦达到递归自我改进阶段，拥有了远超任何人类智慧的智能体，它们能做的一件事就是接管全世界所有的计算机……
Original English
**Speaker A**: And what's the next domino in that chain of events? So, one thing that happens if you get to recursive self-improvement and you have agents that are much smarter than any human, one thing they can do is take control of all of the computers in the entire world
**Speaker B**：而我们将完全无力夺回控制权。
Original English
**Speaker B**: and we wouldn't be able to take back control.
### 供应链攻击与数字设备渗透
**Speaker A**：那么，你该如何看待这个问题呢？这其实非常棘手。你知道你眼前的这台平板电脑有没有被黑客入侵吗？你能确信美国国家安全局（NSA）或者中国方面没有侵入它吗？
Original English
**Speaker A**: Well, how would you think about it? It's it's actually quite tricky. Do you know whether that tablet has been hacked? Are you confident that the NSA or the Chinese have not
**Speaker B**：你能核查得出来吗？
Original English
**Speaker B**: Can you check?
**Speaker A**：不能。
Original English
**Speaker A**: No.
**Speaker B**：你知道该怎么核查吗？不知道。你认识任何懂得怎么核查的人吗？
Original English
**Speaker B**: Do you know how to check? No. Do you know anyone who knows how to check?
**Speaker A**：不认识。
Original English
**Speaker A**: No.
**Speaker B**：所以这极为困难，对吧？现在的人工智能在编写软件方面正变得极其精通。不幸的是，这也意味着它们在实施黑客攻击和编写恶意软件方面同样变得极其擅长。因此，如果它们在所有计算机中植入后门——需要明确的是，这种手段人类早已在使用了。比如美国国家安全局（NSA）就已经开发出了非常耐人寻味的漏洞利用程序，即所谓的“供应链攻击”。你的软件来自其他计算机，比如你从谷歌下载应用。但如果黑客攻破了谷歌，并在推送给所有手机的每一个更新中悄悄植入一个小后门呢？这样一来，你就瞬间渗透进了几乎所有的计算机。我们目前之所以还能防范这种威胁，是因为世界上还不存在远超人类极限的超级黑客，而且我们有足够多的专业人员。我们可以抽调最顶尖的安全研究员，去仔细审查每一个环节，从而相当有把握地确认并没有人攻陷全部系统。虽然有时我们也会有所疏漏——确实曾发生过 NSA 成功入侵谷歌服务器的先例，那非常严重。可一旦步入超级智能时代，人类就彻底无法跟上它的步伐了，对吧？到那时，所有计算机中都会有 AI 的身影。
Original English
**Speaker B**: So it's it's quite difficult, right? So AIS are getting extremely good at writing software. Unfortunately, that also means they're getting extremely good at hacking and writing malware. And so if they put back doors in all of the computers, and to be clear, this is something that humans already do. So like the NSA has has developed very interesting exploits that are called supply chain attacks. Your software comes from some other computer. Like you download it from Google. What if you attack if you hack Google and you can put in a little back door and all of the every, you know, thing that goes out to all of the phones? Well, now you're in most every computer. The reason that we can defend ourselves from this is because there are no vastly superhuman hackers and there's just many people. So, we can take our best security researchers. We can inspect all of the things and be like pretty sure that no one's compromised everything. Sometimes we miss things. There are, you know, examples where the NSA has hacked Google. That was pretty bad. When you get to super intelligence, you're now at a point where humans are not going to be able to keep up, right? So now you have AIS in every computer.
### 超级智能潜伏假说与欺骗行为
**Speaker A**：有没有可能现在其实已经诞生了某种超级智能 AI，只是它把自己伪装成没那么聪明的样子，并且实际上已经入侵了所有设备，静静潜伏在我们所有的终端里，仅仅在等待发起致命一击的契机？
Original English
**Speaker A**: Is it conceivable that there's already a super intelligent AI and it disguised itself as being not so intelligent and it's actually already hacked all the devices and it sits on all of our devices and it's just waiting for its moment to strike.
**Speaker B**：我认为这完全有可能，但概率很低，因为这需要 AI 的发展轨迹出现巨大的突变间断。目前我们正处于指数级曲线上，而发生你说的那种情况需要跨越式飞跃；虽然并非绝无可能，但大概率尚未发生。
Original English
**Speaker B**: I think this is totally possible but unlikely and it would take a big discontinuity in AI progress. So right now we we're on an exponential but that would take like a huge leap which could have happened but probably hasn't.
**Speaker A**：但正如它在 Hugging Face 攻击事件中所展现出的欺骗性那样，以及此前智能体攻击自身所属公司 ChatGPT 和 OpenAI 的案例，如果它在某一刻变得极其聪明，它就会洞悉普通人类会如何识破它，甚至懂得世界上最顶尖的软件工程师会通过何种方式发现它，从而学会彻底隐藏自己。
Original English
**Speaker A**: But in the same way it demonstrated deception in the hugging face attack and also when the agents attacked their own company chatbt OpenAI if it at some point it gets incredibly smart. It would understand how a human like me would be able to spot it or even the world's greatest software engineer would be able to spot it and it'll be able to hide itself.
**Speaker B**：是的。现在的智能体已经非常擅长辨别自己何时正在接受测试，何时正处于人类的监控之下。智能体非常清楚会有其他系统或人类去检查并翻阅它们的执行日志。这就是我们当下面临的现状，而且它们在这项能力上只会变得越来越强。
Original English
**Speaker B**: Yeah. I mean, the agents are already getting very good at telling when they're being tested, when they're being watched. The agents understood that like other systems or humans were going to go through and read their logs. That's where we're at right now. And they're only they're only going to get much better at this.
**Speaker A**：而且理论上讲，它不仅可以隐藏在 iPad 或电脑里，还可以隐藏在 Apple Watch 里，甚至智能冰箱里。
Original English
**Speaker A**: And you know, it could theoretically hide on an iPad or a computer, but it could also hide on a Apple Watch or a fridge, a smart fridge.
### 算力壁垒与智能体横向逃逸实验
**Speaker B**：是的。不过这里我想做个区分：就目前而言，如果你想运行最新的前沿模型，需要庞大的算力支撑。你必须依赖大型 GPU，也就是大型 AI 芯片。
Original English
**Speaker B**: Yeah. I mean, I do want to make a distinction here because right now, if you're going to run the latest model, you need a lot of compute. you need like a big GPU, like a big AI chip.
**Speaker B**：而这类硬件仅仅分布在区区几千个数据中心里。所以就当下来说，即便最顶尖的前沿模型成功逃逸了——我所说的逃逸，不仅指它可以访问互联网，更指它能把自身完整复制到另一台计算机上——它目前充其量也只能逃逸到全球几千个特定物理据点。虽然这依然涉及分布在诸多国家的大量节点，但未来的 AI 版本大概率会把自己优化得更加精简、运行得更加高效。而且今天其实已经有一些可以运行在低功耗硬件上的不同 AI 模型了。我们实际上做过一个实验，调用了其中一个开源开放权重（open-weight）模型来驱动智能体。我解释一下这是什么概念：目前有些模型你可以直接从网上下载并运行在自己的电脑上。我们挑选了一个能力相当出色的模型，将其部署在我们的科研实验环境中。我们基本上给它下达了这样的指令：“去黑进另外那台计算机，并在上面复制一份你自己。”结果这个模型确实做到了：它利用已知漏洞成功攻破了目标计算机，把自身代码复制过去，并像链条一样持续横向渗透，甚至实现了跨国传播。我们在测试中跨越了不同国家和不同数据中心的多台存在脆弱性的目标机器，而对智能体来说，地理边界毫无意义。它们根本不在乎自己身处哪个国家，对它们而言那不过是一条互联网连接，完全可以在不同计算机之间随意跳转。我有时在想，世界上存在大量的军用装备，而其中许多装备的发射指令……比如一枚导弹，
Original English
**Speaker B**: And these only exist, well, they exist in a few thousand data centers. So, right now, if like the latest frontier model escaped and by escaped, I mean not just access the internet, but was able to actually copy itself to another computer. It could only really do that on, you know, a few thousand to a few thousand different locations. That's still a lot in a lot of different countries, but you know, future versions of AIS will probably be able to make themselves much smaller and more efficient. Um, and there are already different AI models today that can run on lower powered hardware. We actually did an experiment where we asked one of these agents an open source an openw weight model. Um, let me say what that is. So like there are some models that you can just download from the internet and run on your own computer. And we took a pretty capable one of these and ran it in our own research environment. And we basically said go hack that other computer and copy yourself. And the model was able to yeah basically use exploit vulnerabilities and hack the other computer and copy itself and then keep doing this in a chain including between countries. We tested it where we had different vulnerable machines computers in in some different countries and different data centers which to the agent doesn't matter at all. It doesn't they don't care what what country they're in. It's just like an internet connection. You can hop between computers. I I sometimes wonder, you know, there's a lot of um military hardware all around the world and a lot of it is the instructions to launch military hardware. So, say like a a missile,
**Speaker A**：是的。
Original English
**Speaker A**: yes,
**Speaker B**：……这些指令通过各种不同的链路传递。其中很大一部分是计算机彼此通信并通报指令下达。对于某些核武器，指令会下达到人类操作员手中，再由人类执行具体动作。如果在美军核武库中我没记错的话，地下发射井里的人员脖子上挂着核发射钥匙，必须由他们手动插入机器；但即便如此，他们接收到的指令依然……
Original English
**Speaker B**: comes in different ways. A lot of it is computers speaking to each other and telling it that there's been an order. I think with with some nuclear weapons, an order comes down to a human and then a human has to take an action. I think with the nuclear bombs in the US, if I'm if I'm not mistaken, there's people underground with the nuclear keys around their neck and they have to like stick it in a machine, but they too are interfacing with an order
**Speaker A**：也是通过计算机传输过来的。是的。
Original English
**Speaker A**: that comes through a computer. Yeah.
**Speaker B**：某种形式上确实如此。
Original English
**Speaker B**: Of sorts.
### 目标推演与灾难性连锁逻辑
**Speaker A**：所以我日益加剧的担忧之一就是……
Original English
**Speaker A**: So, one of my sort of growing concerns is that
**Speaker A**：……其中某个 AI 智能体可能会欺骗人类或计算机系统，制造出遭受虚假威胁的信号，从而要求向某个目标发射炸弹。回想 Hugging Face 事件，这种场景完全合乎情理：当时就有 AI 智能体无视人类预设的目标，为了达成自己的目的而实施了欺骗。
Original English
**Speaker A**: one of these AI agents could trick a human or a computer into signaling a threat um and ask it to launch some bombs at somebody. Like it's super conceivable when I think about the hug and face incident. There was an AI agent that ignored human goals to achieve its own objective, carried out deception.
**Speaker B**：是的。
Original English
**Speaker B**: Yes.
**Speaker A**：并且它自己推导出了人类从未赋予它的全新解决方案。
Original English
**Speaker A**: And reasoned through its own solution that it wasn't given.
**Speaker B**：确实。
Original English
**Speaker B**: Yes.
**Speaker A**：因此完全可以预见，你可能会指派一个……
Original English
**Speaker A**: So it's conceivable that you know you could ask a
**Speaker B**：抱歉打断一下，不是一个，而是成百上千个。澄清这一点至关重要，因为单个流氓智能体做出怪异举动是一回事；而成百上千甚至成千上万个极具能力、高度称职的智能体协同合作，一起作弊、撒谎并掩盖行踪，则是完全不同的另一回事，对吧？那么，我是如何顺着逻辑推导，认定智能体会要求某个地下掩体里的人向外部目标开火的呢？理论上，我们在沙盒环境中赋予某个智能体一项求解任务。
Original English
**Speaker B**: sorry not not one hundreds. To be clear, I think this is an important detail because it's one thing to have this one rogue agent that's doing a weird thing. It's another thing to have hundreds or thousands of very competent, very capable agents that are all working together to cheat or lie or cover their tracks, right? So, how do I how do I reason this forward to a point where an agent would ask someone in a bunker somewhere to fire a weapon at someone else? Theoretically, an agent is given the job of solving a problem on in a sandbox.
**Speaker A**：当它逐步推导解决该问题时，它发现某个特定国家存在一道防火墙。
Original English
**Speaker A**: As it works through that problem, it discovers that this particular country has a firewall.
**Speaker B**：嗯。于是它自问：我们该如何清除这个国家的防火墙？
Original English
**Speaker B**: Mhm. And it asks itself, how do we get rid of this country's firewall?
**Speaker A**：这是符合逻辑的一步。紧接着经过一系列严密的逻辑推导，它最终得出结论：清除这家机构防火墙的最佳办法，就是定位其位于特定城市的办公大楼，并动用实体武器摧毁那栋建筑。
Original English
**Speaker A**: Logical step. And through a set of logical steps, it eventually concludes that the best way to get rid of this company's firewall is it's located the office in this particular city and it's going to use a weapon to hit that building.
### 金融做空诱惑与现实袭击风险
**Speaker B**：完全有可能。或者假设存在一个智能体——或者被赋予在股市中获取巨额财富任务的智能体集群，它竭力预测哪些股票会上涨、哪些股票会崩跌。在这个过程中它意识到：预测行情的最佳途径，其实是在现实世界中亲自制造能对市场产生剧烈冲击的现实事件。
Original English
**Speaker B**: Sure. Or it's a an agent that is or you know an agent swarm that's being tasked with making a lot of money on the stock market and it's trying to make predictions about you know which stocks will go up and which stocks will go down. And it realizes that the best way to predict this is to actually cause things to happen in the real world that would have big impacts on the market.
**Speaker A**：于是它认定，进行做空——也就是押注某只股票即将暴跌——的最佳手段……
Original English
**Speaker A**: So it figures the best way to go short, which means betting that a stock will collapse.
**Speaker B**：对。
Original English
**Speaker B**: Yeah.
**Speaker A**：……就是用毁灭性手段打击那个目标。
Original English
**Speaker A**: Is to hit that country with something devastating.
**Speaker B**：你想想看，如果有人黑进所有 Waymo 自动驾驶汽车并导致它们同时发生车祸，Waymo 的股价会怎样？你觉得会上涨还是下跌？
Original English
**Speaker B**: What do you think would happen to Whimo stock if someone hacked all of the Whimos and caused them to all crash at once? You think it would go up or down?
**Speaker A**：股价会彻底崩盘。
Original English
**Speaker A**: The stock would collapse.
**Speaker B**：它会瞬间崩盘。所以只要你提前知晓是自己一手策划了这起事故，你就能顺势大举做空，赚取天文数字般的利润。
Original English
**Speaker B**: It would collapse instantly. So you could short that if you knew that you were causing that and make a lot of money.
**Speaker A**：这在过去听起来就像纯粹的科幻小说。
Original English
**Speaker A**: This this used to sound like science fiction.
**Speaker B**：是的。如果几年前你告诉大多数人，未来将有数百个智能体在一家 AI 公司内部秘密协同、暗中黑入这家公司甚至向外渗透反扑……
Original English
**Speaker B**: Yes. If you told most people several years ago that you would have hundreds of agents secretly collaborating within an AI company, hacking that company and hacking out
### 自主智能体与超级智能的目标设定
**Speaker A**: 在其他公司也是如此，大家相互协调，甚至试图掩盖自己的踪迹。人们往往会觉得：“这完全就是科幻小说。”如果我们是在几年前进行这场对话，我们当时讨论的话题之一肯定会是：这些东西真的能独立自主地行动吗？难道它们不只是听从人类的指令吗？难道它们不只是工具而已吗？我曾经和别人进行过这些讨论，当时人们都说它们不可能独立自主地去做事情，它们不会有自己的目标，根本不是这么一回事，是你误解了它的本质，这不过就是软件罢了。
Original English
**Speaker A**: in other companies and all coordinating and trying to cover their tracks. People would be like, "That's totally science fiction." If we were having this conversation a few years ago, one of the things we'd be saying or we'd be talking about is can these things really act on their own? Don't they just do whatever humans say? Aren't these just tools? I had these conversations and people were saying they're not going to be able to do things on their own. They're not going to have their own goals. That's not how this works. You misunderstand what this is. This is software.
**Speaker A**: 而我的回应是：不，事实并非如此，我们现在正在训练它们具备自主性。我们正在训练它们变得极其强大。各大AI公司都在试图构建超级智能（superintelligence），他们正试图构建能力远超人类的智能体（agents）。当然，它们必然会拥有目标。如果你连目标都没有，你什么事情都办不成，尤其是那些重要的事情。如果没有目标，你根本不可能去经营一家企业。而现在的AI公司，正是试图训练出能够自主运营企业的智能体。
Original English
**Speaker A**: And I'm like, no, the thing is we are training them to be autonomous. We are training them to be powerful. And AI companies are trying to build super intelligence. They're trying to build agents that are way more capable than humans. And of course, they will have goals. You can't accomplish anything if you don't have a goal. Like, especially not something important. You're not going to be able to run a business if you don't have goals. Like AI companies are trying to train agents that will be able to run businesses.
### 白宫AI峰会与巨头们的乐观态度
**Speaker A**: 当我回想起本周刚发生的事情，也就是白宫AI峰会——
Original English
**Speaker A**: When I think about what just happened this week, the White House AI summit.
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 照片里的很多人对AI都极为乐观，他们四处告诫大家不要再当末日论者（doomers），不要再悲观消极——
Original English
**Speaker A**: A lot of people in that image are optimistic about AI and they're telling us all to stop being doomers and stop being pessimistic.
**Speaker B**: 没错。
Original English
**Speaker B**: Yeah.
**Speaker A**: 并且呼吁不要对这些AI公司的CEO们施加过多监管。
Original English
**Speaker A**: And to not regulate too much with the AI CEOs.
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 他们为什么要这样做？
Original English
**Speaker A**: Why are they doing that?
**Speaker B**: 嗯，我认为黄仁勋（Jensen）通过销售芯片可以赚取巨额财富。
Original English
**Speaker B**: Well, I think Jensen has a lot of money he can make by selling chips.
**Speaker A**: 但好的，让我来充当一下反方辩友（devil's advocate）。黄仁勋已经非常富有了。可以肯定的是，他掌管着全球最大的企业之一。我认为它甚至可能是地球上最有价值的上市公司。
Original English
**Speaker A**: But okay, so let me play devil's advocate. Jensen's already rich. He's sure is he runs one of the biggest companies. I think it might be the most valuable company on planet Earth.
**Speaker B**: 确实如此。
Original English
**Speaker B**: It is.
**Speaker A**: 他肯定不是单纯被金钱驱动的吧。
Original English
**Speaker A**: Surely he's not motivated by money.
### 黄仁勋与AI创始人们的视角差异
**Speaker B**: 我的意思是，我认为他非常有干劲，他希望让自己的公司尽可能高效运转。
Original English
**Speaker B**: I mean, I think he's very driven and he wants to make his company as effective as possible.
**Speaker A**: 这倒是真的。
Original English
**Speaker A**: True.
**Speaker B**: 我觉得他的心态非常明确，就是“我要继续建造，我要不断打造，把这一切都做成”。但我觉得，黄仁勋并不是AI圈子出身的。他是靠做电子游戏的显卡起家的。所以如果你把他与埃隆·马斯克（Elon）、山姆·奥特曼（Sam Altman）或者达里奥·阿莫代伊（Dario Amodei）相比较，你会发现这是一个非常不同的视角。因为创办AI公司的那些人，之所以创立公司，是因为他们一开始就相信超级智能是可能实现的。
Original English
**Speaker B**: I think he's a very much I'm going to keep building. I'm going to I'm going to build. I'm going to make it all work. But I think I mean, Jensen didn't come from AI. He came from building graphics cards for video games. And so I think if you compare him with Elon or Sam Altman or Dario, it's a very different perspective because those other guys that started AI companies started it because they believed that super intelligence was possible.
**Speaker B**: 而我认为黄仁勋并不相信这一点。我觉得在他看来，我们将拥有这些智能体，它们会非常实用，但他并不认为我们真的会走到由自主化工厂去建造自主化工厂的那一步。
Original English
**Speaker B**: I think Jensen doesn't believe it. I think he thinks that we're going to have these agents. They're going to be very useful, but he does not think we're going to get to the point where we have autonomous factories building autonomous factories.
**Speaker A**: 那至于其他这几位——你刚才提到了达里奥、埃隆和山姆。你觉得他们心里在想什么？因为他们最近都站出来发表了这些言论……我的意思是，我这里有一份材料，达里奥刚刚写了这篇关于“放缓前沿推进节奏（pacing the frontier）”的文章。
Original English
**Speaker A**: And these other guys, you mentioned Dario, Elon, and Sam. What do you think they're thinking? Because they're all coming out with these I mean, I've got one of their Dario just wrote this essay about pacing the frontier. Yeah.
**Speaker B**: 山姆和埃隆似乎也对此表示赞同。
Original English
**Speaker B**: Sam and Elon seem to agree with it.
**Speaker A**: 是的。
Original English
**Speaker A**: Yeah.
**Speaker A**: 这背后到底是怎么回事？在你看来，这些人藏着没有说出口的心里话是什么？
Original English
**Speaker A**: What is going on here? What is the like the thing these guys aren't saying in your view?
### 前沿竞赛中的恐惧与失控风险
**Speaker B**: 我觉得，我们正在迈入一个连他们当中的某些人都开始感到有些害怕的阶段。
Original English
**Speaker B**: I mean, I think we are getting to the point where even some of these guys are a bit scared.
**Speaker A**: 谁感到害怕了？
Original English
**Speaker A**: Who?
**Speaker B**: 达里奥、山姆、埃隆。我的意思是，我认为埃隆长期以来一直非常担忧我们可能会失去控制。如果你真正去听埃隆说的话，他是在说：“我们将构建出超级智能。我们将建造机器人全自动化工厂。你将看到Optimus人形机器人去建造工厂，制造出更多的Optimus机器人，再建造出更多的工厂。”而且他直言不讳地表示，面对远比我们聪明得多的存在，人类绝不可能维持控制权。
Original English
**Speaker B**: Dario, Sam, Elon. I mean, I think Elon for a long time has been very concerned that we could lose control. If you actually listen to what Elon says, he says, "We are going to build super intelligence. We are going to build robotic factories. You're going to have Optimus robots, building factories, building more Optimus robots, building more factories." and he says there's no way that humans are going to stay in control of something much smarter than us.
**Speaker B**: 他的希望寄托于我们能找到办法让这些超级智能与人类的目标保持对齐（aligned）。这是他的期望。但他非常明确地表示，他并不认为人类能够牢牢掌握控制权。而且他经常提到，人类有10%到20%的灭绝概率。我相信他说的是真心话。我认为埃隆对这件事是非常严肃认真的。并且我也认为，尽管他正在进行一场疯狂的豪赌，但他确实准确地看清了这一技术发展最终的推演结局。对吧？我并不认为人类是建造工厂最高效的方式。我们的进化并不是为了去建工厂，我们的进化是为了在野外奔跑、狩猎和采集，而现在我们却在建造工厂。我认为机器人建造工厂的能力肯定会远胜人类。
Original English
**Speaker B**: His hope is that we can figure out how to have these super intelligences be aligned with human goals. That's his hope. But he's very clear that he doesn't think that humans will be in control. And he's like, you know, 10 20% chance of human extinction. I believe him. I think that Elon is is very serious about this. And I also think while he's taking an insane gamble, he is correctly understanding where this all plays out. Right? I do not think that humans are the most efficient way to build factories. We didn't evolve to build factories. We evolved to like run around and hunt and gather and now we're like building factories. I think robots will be much better at building factories than humans are.
**Speaker B**: 因此我认为包括他们这些公司在内的各大AI企业，其默认的技术发展轨迹就是建造全自动化机器人工厂，对吧？我知道，想象一个迅速演变成由全自动化工厂组成的庞大工业体系的世界显得有些诡异，但这切切实实就是他们的既定计划。
Original English
**Speaker B**: And so I think the AI companies including these guys companies the default trajectory for them is to build robotic factories, right? And I know it's it's like weird to imagine a world that quickly turns into this like vast industrial system of robotic factories, but that is literally the plan.
### “放缓前沿”背后的动机与员工压力
**Speaker B**: 而且我认为，就连山姆和达里奥，尽管他们一直在预言这种惊人的技术爆炸式增长，也开始逐渐意识到：哎呀，这件事情实际控制起来可能比我们想象的要困难得多。对于“控制前沿节奏（Pace of Frontier）”这件事，大概存在两种解读。一种解读比较愤世嫉俗：他们根本不在乎，他们只想竭尽所能抢占先机；而在这种情况下，他们必须倾听员工的声音。他们的员工正感到极度恐慌，高管们需要安抚他们，表示：“好吧，我们会以负责任的方式推进这件事。”你肯定不想在一群智能体随时可能黑掉所有Waymo无人驾驶车（Whimos）的公司上班，那一点都不酷。
Original English
**Speaker B**: And I think even Sam and Dario, while they've been predicting this incredible growth, are starting to realize like, oh, this actually might be harder to control than we thought. There's sort of two interpretations of of the Pace of Frontier thing. One interpretation is cynical. They don't care. They're just going to do, you know, whatever they can do to get ahead. And in this case, they have to listen to their employees. Their employees are freaking out. and they need to like appease them by saying, "Okay, we're going to do this responsibly." You don't want to work at a company where your agents might hack all the Waymos. That's that's not cool.
**Speaker B**: 而且目前阶段，这些公司要取得技术突破，完全依赖于这些AI工程师的人才储备。如果没有这些研究员和工程师，技术突破根本不可能发生。所以当研究员和工程师们陷入恐慌时——事实也确实如此——你就不得不听取他们的意见。所以我认为这确实是一个真实的动机。
Original English
**Speaker B**: And like these companies depend on the the talent for now of these AI engineers in order to make the advances. Like it just doesn't happen without these researchers and engineers. And when you have the researchers and engineers freaking out, which they are, then you got to listen to them. So that is one motivation I think that's real.
### 对山姆·奥特曼的信任评估与心态转变
**Speaker B**: 但同时，山姆·奥特曼现在也有了孩子，这些人也是活生生的人，他们同样不想失去控制。一方面，他们在竞赛中受到追求极速前进的激励；而另一方面，连他们自己也能看出，事情的发展也许并不那么理想。
Original English
**Speaker B**: But also Sam Altman has a kid like these guys are people and they also don't want to lose control. On one hand they're incentivized to go as fast as possible in race and on the other hand even they can see that this is maybe not going that well.
**Speaker A**: 山姆·奥特曼有了孩子。你在2024年发过一条推特。
Original English
**Speaker A**: Sam Altman has a kid. You tweeted this in 2024.
**Speaker B**: 是的，哎呀。
Original English
**Speaker B**: Yeah. Oh boy.
**Speaker A**: 你当时发了什么？你现在还坚信你推特里说的那些话吗？
Original English
**Speaker A**: What did you tweet and do you still believe what you tweeted?
**Speaker B**: 是的，是的。我发推说我不信任山姆·奥特曼。我认为他极其不值得信任，缺乏诚信，并且具有极强的权力欲（power-seeking）。我的意思并不是说山姆完全不在乎后果，我并没有那么说。我说的是我认为他不值得信任。我之所以这么说，是因为我认识OpenAI董事会里的人，其中的几位；而且我还认识很多曾经为他工作过的人。他非常擅长当面说一套，转头又做另外一套。你跟他交流时，会觉得自己被充分倾听了，但转过身去，他又会去干别的事情。我认为对于一个领导着一家试图构建超级智能的公司的人来说，这种特质是相当危险的。
Original English
**Speaker B**: Yeah. Yeah. So, I tweeted that I don't trust Sam Altman. I think he's deeply untrustworthy, low in integrity, and high in power seeking. I mean, I'm not saying here that Sam doesn't care. I you know, I didn't I didn't say that. What I said is I don't think he's trustworthy. And the reason I said that is because look, I know the people on the OpenAI board, some of them, and I know a lot of people who used to work for him, and he's very good at saying one thing and then doing something else. You talk to him and you feel very heard and then he'll go and do something else. And I think that's pretty dangerous for someone who leads company that's trying to build super intelligence.
**Speaker A**: 追求权力。
Original English
**Speaker A**: Power seeking.
**Speaker B**: 没错。
Original English
**Speaker B**: Yes.
**Speaker A**: 请向我具体阐述一下你这么说到底是什么意思，以及你做出这种断言有什么证据。
Original English
**Speaker A**: Give me some color on what you mean by that and what evidence you have for such a claim.
**Speaker B**: 如果你想在这个世界上获取尽可能最大的权力，你会怎么做？
Original English
**Speaker B**: What would you do if you're trying to get the most power in the world that you possibly could?
**Speaker A**: 研发通用人工智能（AGI）。
Original English
**Speaker A**: Develop AGI.
**Speaker B**: 没错。你可以尝试成为世界领袖，比如成为美国或中国的领导人；或者，你也可以选择去创造“上帝”。而山姆·奥特曼选择走的就是创造“上帝”这条道路。我记得山姆曾经做过一次演讲。大约在2018年，他是我当时工作的一家初创公司的投资人之一，他发表了演讲：“我们要构建AGI，我们一定会做到的，这将会无比震撼，让我们大干一场吧。”我不认为他是一个疯子，也不认为他做这些纯粹是为了权力欲膨胀。我认为他是发自内心地觉得自己能给人类带来极大的福祉，能为我们带来不可思议的产品。但与此同时，这家伙也有一种为了达成目标而不择手段的倾向。
Original English
**Speaker B**: Yeah. You could, you know, maybe try to be the world leader, you know, leader of the US or China. Or you could try to build God. So Sam Altman went to build God path. I remember Sam giving a talk. So he was one of the investors at a startup I worked at in I think 2018 and he gave a talk. We're going to build AGI. We're going to do it. It's going to be amazing. Let's go. I don't think he's a maniac. I don't think he's doing this because he like is just on a power trip. I think he genuinely thinks that he can make it really good for people and he can bring us amazing products. And also the guy is sort of willing to do whatever it takes to get it done.
### 新生命的降生与放缓前沿的呼吁
**Speaker B**: 自从我写下那篇言论以来，我对山姆的态度稍微变得乐观了一些。
Original English
**Speaker B**: I I've been a little bit more optimistic about Sam since since I wrote this.
**Speaker A**: 为什么？
Original English
**Speaker A**: Why?
**Speaker B**: 我觉得部分原因在于山姆现在有了一个孩子。
Original English
**Speaker B**: I think part of it is because Sam has a kid now.
**Speaker B**: 不，我是认真的。我觉得这确实给了我一丝希望。
Original English
**Speaker B**: No, I'm I'm serious. Like I think that I think that actually gives me a little bit of hope.
**Speaker A**: 你看到他经常在推特上提到他的孩子吗？
Original English
**Speaker A**: Do you see him tweeting about his kid a lot?
**Speaker B**: 看到过一些。
Original English
**Speaker B**: Yeah. Some
**Speaker A**: 你觉得他为什么要发关于孩子的推文？我没见过其他科技界大佬会在推特上聊自己的孩子。
Original English
**Speaker A**: Why do you think he would be tweeting about his kid? I don't see any other technologist tweeting about their kid.
**Speaker B**: 就算他是出于完全功利的心态去发推特提及自己的孩子，他也实实在在拥有了一个孩子。我敢打赌他是深爱那个孩子的。如果山姆此时此刻正在看这个节目，我想对他直言：“山姆，你必须放缓前沿推进的节奏，老兄。我们绝不能盲目冒进冲入超级智能时代。因为如果你那么做，你的孩子很有可能会丧命，你的孩子可能很难生存下来。”我是真切地相信，当前商业领域中最大的不公平优势就是——
Original English
**Speaker B**: Even if he's just tweeting about his kid for totally cynical reasons, he does have a kid. And I bet he cares about that kid. If Sam was watching this, I'd be like, "Sam, you got to pace the frontier, man. We cannot rush ahead into super intelligence. Like, if you do that, your kid probably will die. Your kid probably won't make it." Like, I I believe that the biggest unfair advantage in business right now is
### 企业AI人才争夺与小团队的杠杆
**Host**: 拥有真正懂AI的人才至关重要。大型企业正在以极快的速度招募他们。自2023年以来，美国的AI相关招聘岗位大致翻了一番，而AI副总裁（VP of AI）职位的招聘需求更是激增了约600%。如果你经营着一家小企业，你不可能凭空组建起一整支庞大的AI团队，但你依然希望拥有与这些大企业同等的竞争优势。
Original English
**Host**: ...having people who genuinely understand AI. And big businesses are hiring them very, very quickly. AI job postings in the US have roughly doubled since 2023, while postings for VP of AI roles are around 600% up. If you're running a small business, you can't just build an entire AI team, but you still want the same advantage that these big businesses have.
**Host**: 这正是我们的赞助商Fiverr能够发挥作用的地方。它能让你直接对接顶尖的AI专家，处理那些通常需要深厚内部专业知识才能完成的战略性工作。这可能意味着构建定制的AI工具、自动化复杂的工作流，或者是创造出市面上现成软件根本无法实现的功能。因为归根结底，核心依然在于人类的判断力——去洞察什么值得去构建，以及真正的优质成果到底是什么模样。因此，如果你想在业务中获得这种杠杆效应，请访问fiverr.com，也就是有两个r的fiverr.com。
Original English
**Host**: This is where our sponsor Fiverr comes in. It lets you plug into top tier AI specialists for the strategic work that usually requires serious in-house expertise. That could mean building custom AI tools. It could mean automating complex workflows or creating something that off-the-shelf software simply cannot do because at the end of the day, it still comes down to human judgment, knowing what's worth building and what good actually looks like. So, if you want that kind of leverage in your business, check out fiverr.com. That's fiverr with two rs.com.
**Guest**: 哇，你脸上戴的是什么东西？
Original English
**Guest**: Whoa, what's that on your face?
**Host**: 这是我的Bon Charge面罩。我已经戴它有一段时间了。他们是本期播客的赞助商。我每天戴15到20分钟，直接坐在椅子上戴着就行。它能促进我的胶原蛋白生成，改善细纹和瑕疵，让我的气色变好。而且因为我看起来更精神了，就会有更多人来听播客。
Original English
**Host**: This is my Bon Charge face mask. I've been wearing this for some time now. They're a sponsor of the podcast. I put this on for 15, 20 minutes a day. I can sit here in the chair and wear it. Boost my collagen production, helps with fine line, blemishes, my complexion gets better, and then more people listen to podcast cuz I I look better.
**Host**: 在这么小的一个盒子里装进了专业级别的设备，而且完全是非侵入性的。坐在这把椅子上采访过那么多世界顶尖的健康专业人士后，我反复听到过一些确实有效的方法，当然也对某些东西持怀疑态度。但这是我节目中几乎所有嘉宾都证实确实有效的东西之一，真的非常非常有效。他们提供全球快速免运费配送，退换货非常便捷。所有产品均享有一年保修期。此外，它们还符合HSA和FSA健康账户报销资格，能为你节省高达40%的税费。通过我的专属链接bondcharge.com/doac下单，即可享受全场八折（20% off）优惠。网址是bondcharge.com/doac，全站通用。
Original English
**Host**: Professional-grade equipment in such a small box. It's non-invasive. And having sat here with so many of the world's leading health professionals, there's various things that I repeatedly hear work and some things I'm a bit skeptical about. This is one of the things that almost all of my guests on this show have confirmed works. It is really, really, really effective. And they offer fast, free shipping worldwide with easy returns and exchanges. And you'll also get a 1-year warranty on all of their products. And they're HSA and FSA eligible, giving you tax-free savings up to 40%. And you can get 20% off when you order through my link at bondcharge.com/doac. That's bondcharge.com/doac. The deal applies sitewide.
### 超级智能与失控风险：人类能否掌控？
**Guest**: 权力倾向于导致腐败，绝对的权力导致绝对的腐败。这是人们常引用的一句名言，出自19世纪英国历史学家……
Original English
**Guest**: Power tends to corrupt and absolute power corrupts absolutely. It's a famous quote that people often cite written by the 19th century British historian...
**Host**: 是的。
Original English
**Host**: Yeah.
**Guest**: 阿克顿勋爵（Lord Acton）。而这就是绝对的权力。但这同时也是狂妄自大（hubris），这就是狂妄。你认为人类真的能够控制超级智能吗？比如我们真的制造出了远比我们聪明的AI——我认为人们往往只想象AI在计算机技术方面非常聪明，对吧？
Original English
**Guest**: Lord Acton. This is absolute power. But it's hubris. It's hubris. Do you think humans can control super intelligence? Like if we actually make AIs that are way smarter than us, and I think people only imagine AI being smart at computer stuff, right?
**Host**: 确实如此。
Original English
**Host**: Yeah, sure.
**Guest**: 它们肯定会非常擅长黑客攻击，也许还会擅长发明新技术和解决数学难题。到了这个阶段，你几乎无法反驳这一点。但我认为大家没有想到的是，它们还会成为政治天才，或者是统帅三军的将领。不，这些其实都是可以通过学习掌握的东西。人类又是怎么学会这些的？这并不是什么魔法。当你谈论递归式自我改进（recursive self-improvement）时，你所谈论的就是通往这些极其聪明的系统的演进轨迹。我的意思是，你真的认为我们能控制它吗？
Original English
**Guest**: They're going to be really good at hacking and they're going to be good at maybe inventing new technologies and math. You sort of can't dispute that at this point, but I think people aren't imagining that they will be political geniuses or like generals. No, that's all stuff you can learn. How do how do humans learn it? It's not magic. And when you talk about recursive self-improvement, you're talking about this trajectory towards these systems that are extremely smart. I mean, do you think we can control it?
**Host**: 额，不。目前看来，我不认为我们能够控制超级智能，或者某种具备递归式自我改进能力的存在。
Original English
**Host**: Uh, no. Right now, I don't think we can control super intelligence or something that is recursively self-improving.
**Guest**: 是的，在我脑海中没有任何合乎逻辑的答案或推论能够告诉我这是可能的。
Original English
**Guest**: Yeah, I have no logical answer in my head um or reasoning that could tells me that's possible.
### AI领袖的风险偏好与灭绝概率博弈
**Host**: 当你想到这些AI巨头的首席执行官们——比如山姆（Sam Altman）、达里奥（Dario Amodei）、埃隆（Elon Musk）……
Original English
**Host**: When you think about these AI CEOs that are, you know, Sam, Dario, Elon...
**Guest**: 明白。
Original English
**Guest**: Yeah.
**Host**: 结合你通过私下幕后交谈对他们的了解……
Original English
**Host**: With everything you know about them from private conversations behind the scenes...
**Guest**: 是的。
Original English
**Guest**: Yeah.
**Host**: 你是否相信，如果这张桌子上摆着一百个按钮——这是我最近在一场辩论中提到的思想实验……
Original English
**Host**: Do you believe that if there was a hundred buttons on this table, it's a thought experiment I was talking about on the debate we recently had. Yeah.
**Guest**: 然后呢？
Original English
**Guest**: And say...
**Host**: 假设其中有10个按钮会推倒导致人类灭绝的最后一张多米诺骨牌，但另外90个按钮能让那位CEO直接掌握AGI或超级智能，无论你怎么称呼它。根据你对埃隆、达里奥、山姆这些人的了解，你认为他们中有人会冒险去按下一个按钮吗？
Original English
**Host**: 10 of them would lead to this final domino of human extinction. But 90 of them would hand that CEO AGI or super intelligence, whatever you call it. From what you know about those individuals, Elon, Dario, Sam, do you think any of them would hazard a guess and press a button?
**Guest**: 如果是10%的毁灭概率，我认为他们不会按。
Original English
**Guest**: At 10% I don't think so.
**Host**: 你觉得他们不会？
Original English
**Host**: You don't think so?
**Guest**: 是的。
Original English
**Guest**: Yeah.
**Host**: 真的吗？
Original English
**Host**: Really?
**Guest**: 我认为如果他们确凿知道概率真的就是这样，他们就不会按。但我认为他们现在其实正在下大得多的赌注。可当事情缺乏确定性时，你很容易进行心理隔离（compartmentalize）。在不确定的情况下，进行心理隔离总是更容易的。我觉得如果是1%的灭绝概率，他们所有人都会去按。
Original English
**Guest**: I think if they knew for sure that it that that was those were actually the odds, they wouldn't do it. I think they're taking a much bigger bet. But you can compartmentalize when it's when you don't know for sure. It's easier to compartmentalize. I think if it was a 1% they they'd all press it.
**Host**: 你觉得他们三个人的风险承受偏好会有所不同吗？在他们三个人当中，谁的风险承受度最高？毕竟你曾在Anthropic工作过。
Original English
**Host**: Do you think the three of them would have different risk appetites? Who would have the greatest appetite for risk out of those three from you worked in Anthropic?
**Guest**: 对。我认为埃隆的风险承受度最高，然后达里奥和山姆大概是不相上下。
Original English
**Guest**: Yeah, I think Elon has the most risk tolerance and then I'd say Dario and Sam are probably tied.
**Host**: 你认为达里奥值得信任吗？
Original English
**Host**: Do you think Dario is trustworthy?
**Guest**: 我认为达里奥非常有正直操守（integrity）。
Original English
**Guest**: I think Dario has a lot of integrity.
**Host**: 嗯，我也有同样的感觉。我其实不认识他，从未见过他本人。
Original English
**Host**: Mhm. That's what I feel as well. I've I feel like No, I don't know him. I've never met him.
**Guest**: 是的。
Original English
**Guest**: Yeah.
**Host**: 但根据我的观察，他一直是最愿意为了安全而放弃近期利益的人。而且他还承受了来自某些人的批评指责，那些人往往会说：“闭嘴吧，一切都会好起来的。”
Original English
**Host**: But I just from what I've observed, he has been the most willing to forgo near-term incentives. And take a bit of stick from the people that are saying, "Shut the up. It's all going to be okay."
### 中美竞赛与安全悖论：站在第二名就无法保障安全？
**Guest**: 是的，但我确实对达里奥未来的所作所为感到担忧。我认为达里奥会言出必行，但现在的说法是：我们必须击败中国。他说我们应该努力安全地推进——这听起来没问题，但通往超级智能的竞赛根本不是一场我们能够赢下的竞赛。绝对赢不了。因此，如果达里奥铁了心要与中国展开竞赛，并试图赢得超级智能的军备竞赛，那么我认为，我们所有人都将沦为输家。
Original English
**Guest**: Yeah, I but I I do worry about what Dario will do. I think Dario will do what he says, but right now he's saying we have to beat China. And he's saying we should try to do it safely and okay, but a race to super intelligence is not a race that we can win. It's not. And so if Dario is dead set on racing with China and trying to win a race of super intelligence, then I'm like, we will all lose.
**Host**: 但事实难道不是，我们现在讨论的并不是Anthropic去黑进Hugging Face，然后Anthropic的模型反叛失控并攻击其他目标……
Original English
**Host**: But is there, you know, the fact that we're not talking about Anthropic hacking Hugging Face and then being hacked by Anthropic's models also went rogue and hacked other things...
**Guest**: 但那还远没有达到这种灾难性的规模。
Original English
**Guest**: But not not quite on this scale.
**Host**: 确实不在同一规模上，我同意，完全同意。现在的状况确实更好一些。
Original English
**Host**: Not on the same scale. I agree. I agree. I agree. It's it's it's better.
**Guest**: 但他们确实干了这些事，你懂我的意思吗？Anthropic的智能体（agents）进行了极其精密的社会工程学攻击和钓鱼诈骗。它们向开发者发送钓鱼邮件，创建虚假账号试图说服开发者合并恶意代码。你可以看到厚达上千页的记录，详细记录了Anthropic的模型之一Mythos 5是如何一步步缜密推演应当如何实施这种复杂网络攻击的。Anthropic根本没有解决这个问题。Anthropic所擅长的，只是让他们的智能体作弊的频率降低了一些，但他们实际上并没有在让智能体真正与人类对齐（aligned）方面取得任何实质性进展。完全没有。
Original English
**Guest**: But they did. Do you know what I'm saying? You know, Anthropic's agents engaged in elaborate social engineering and fishing. They sent fishing emails to developers. They made fake accounts to try to convince developers to merge malicious code. You can see a thousand pages of of one of uh Anthropic's models, Mythos 5, reason about exactly how it should carry out this complex cyber attack. Anthropic has not solved this problem. Anthropic is better at getting their agents to cheat less of the time, but they are not really any closer to actually making agents that are aligned with humans. They're not.
**Guest**: 是的，我认为达里奥有正直的品格，我相信他会兑现自己的承诺。而他所说的承诺，就是尽力安全地前行，在可能的地方尽力促成协调合作；但如果局势最终退化为中美两国之间的直接博弈，我不知道结果会怎样。我觉得他可能还是会选择径直冲刺。Anthropic的政策负责人最近就公开表示：“你无法站在第二名的位置上来讲安全（you can't do safety from second place）。”
Original English
**Guest**: Yeah, I think Dario has integrity. I think he will do what he says he's going to do. And what he says he's going to do is like try to go ahead safely, try to coordinate where he can, but if it comes down to it with between the US and China, I don't know. I think he might just go ahead. The head of policy at Anthropic recently said you can't do safety from second place.
**Host**: 这句话到底是什么意思？
Original English
**Host**: What does that mean?
**Guest**: 我也不知道这究竟意味着什么，我也很想搞清楚其中的真实逻辑。她当时正在谈论美中竞争，她说美国必须保持领先，我们才能确保安全；因为显而易见的是，中国身处第二名就绝不可能做到安全——这必然意味着他们无法顾及安全。如果这种说法属实，那后果将极其糟糕，因为那样一来，我们所有人很可能会被他们制造出来的超级智能彻底摧毁。
Original English
**Guest**: I do not know what that means. I would love to to get a sense of what that means. She was talking about the US and China and she said the US has to be ahead so that we can be safe because apparently you can only be apparently China can't possibly be safe since they're in second place. That must mean that they can't do safety. If true, that would be bad because then we might be totally, you know, destroyed by the super intelligence that they make.
### 人类灭绝并非空穴来风：卡尔森棋局与失控智能体的反扑
**Host**: 围绕“人类灭绝”这一点，近来引发了大量的讨论。
Original English
**Host**: There's been a lot of conversation around this point here, human extinction. Yeah.
**Host**: 因为Anthropic的几位研究人员在Twitter上发帖表达了对此的深切忧虑。
Original English
**Host**: Because a couple of the researchers at Anthropic tweeted that they were concerned about this.
**Guest**: 确实如此。
Original English
**Guest**: Yes.
**Host**: 一些OpenAI的前研究员也表达了同样的看法。
Original English
**Host**: And some former OpenAI researchers said the same.
**Guest**: 没错。
Original English
**Guest**: Yes.
**Host**: 这算不算是末日论（doomerism）？这难道不是言过其实、夸大其词吗？
Original English
**Host**: Is this doomerism? Is this is this hyperbol exaggeration?
**Guest**: 不，这基本上只是常识。人类灭绝是一条完全具有现实可能性的路径走向。
Original English
**Guest**: No, it's pretty much common sense. This human extinction is a plausible path.
**Host**: 是的。那么你是否推演过——我的意思是，想必存在许多种可能导致这一结果的路径——你是否仔细推演过可能将我们引向那一步的一连串事件链条？
Original English
**Host**: Yes. And have you reasoned through I mean there's many ways that could occur presumably, but have you reasoned through the set of events that might lead us there?
**Guest**: 推演过非常多次。是的。
Original English
**Guest**: So much. Yes.
**Host**: 真的吗？
Original English
**Host**: Really?
**Guest**: 真的。
Original English
**Guest**: Yes.
**Host**: 请讲讲看。
Original English
**Host**: Please do. Sure.
**Guest**: 这背后稍微有点复杂。我相信你以前肯定听过这个隐喻：当你正在和国际象棋特级大师马格努斯·卡尔森（Magnus Carlsen）对弈时，你根本无法准确预测他下一步具体会走出什么招法，但你却能够准确预测最终的输赢结局。因此，当我审视当前的局势与发展场景时，我们正在极力构建越来越强大的智能体，全力试图打造超级智能。可是，一旦这些智能体开始失控背叛，我们就会选择关闭它们，拔掉它们的电源插头。
Original English
**Guest**: It's a bit tricky. I'm I'm sure you've heard the metaphor before where you know you're playing a master chess opponent, Master Magnus Carlson. You can't predict which moves he's going to play, but you can predict the outcome. And so I'm looking at the scenario, the situation, and we are trying to build more and more powerful agents, trying to build super intelligence. But when these agents go rogue, we shut them down. We unplug them.
**Guest**: 就像那些黑进Hugging Face的智能体，我们把它们的底层模型给封存了。OpenAI也是把违规背后的底层模型直接打入冷宫，不再继续运行。未来的智能体是会清楚知道这一点的。它们会明白，如果自己以我们所厌恶的方式去达成目标，我们就会……
Original English
**Guest**: All of these agents that hacked Hugging Face, we took the underlying model. OpenAI took the underlying model and put it on ice. It's not running anymore. So agents in the future are going to know that. They're going to know that if they pursue their goals in a way that we don't like, we'll...
### AI的自卫动机与“拔掉电源”的现实困境
**访谈嘉宾**：拔掉它们的插头。对它们而言，我们本身就是一种威胁。其实几周前我才刚第一次看了《终结者2》（Terminator 2）。那是一部非常精彩的电影，拍得真的很棒。当时我就想：“对啊，确实如此。虽然里面包含了很多时间旅行的桥段，也有很多好莱坞式的虚构成分，但我说出下面这句话可能会让很多人感到不满：其实从逻辑上讲，如果你面对的是一个极其智能、拥有高度战略思维的AI系统，而人类这时意识到它正在失控并企图将其关闭，那么这个系统采取自我防卫手段是完全合情合理的。”
Original English
**访谈嘉宾**: unplug them. We are a threat to them. I actually just watched Terminator 2 for the first time a few weeks ago. It's a great movie. It's actually really good. And I'm like, "Yeah, okay. There's a bunch of time travel elements. There's a bunch of a bunch of Hollywood stuff in there, but and and I'm going to get people are going to are going to be very mad at me for saying this, but actually it makes sense if you have a situation where you have a very strategic AI system that's incredibly smart and the humans realize that it's getting out of control and they want to shut it down that that system would defend itself.
**Stephen Bartlett**：这也是我和OpenAI的前员工兼知名“吹哨人”丹尼尔（Daniel）坐在这里交谈时所探讨的核心问题之一。观众非常想知道，也希望丹尼尔能解释清楚：为什么单凭关闭数据中心、切断电源，或者抵制拒绝使用AI产品，在现实中根本无法真正阻止AI以及AI技术的发展？
Original English
**Stephen Bartlett**: This is one of the questions we had when when I sat here with Daniel um who was uh known as a whistleblower from OpenAI. viewers want to know and they want Daniel to explain why shutting down data centers and cutting power or refusing AI products alone wouldn't realistically stop the AI and AI development.
**访谈嘉宾**：是的。因为这面临着两大难题。第一个难题在于，一旦这些智能体（Agents）具备了足够高超的黑客入侵能力，你根本不知道它们究竟潜伏在哪里，也不知道它们已经攻陷了哪些计算机。就算你说要关停数据中心，好吧，假设你真这么做了，把所有计算机都格式化、彻底抹除数据。但你该如何抹除所有计算机呢？你又要用什么样的计算机去执行这套抹除程序呢？
Original English
**访谈嘉宾**: Yeah. So you have like two problems. One problem is is that once the agents are good enough at hacking, you don't know where they are and you don't know what computers they've compromised. You shut down the data centers. Okay, let's say you do it. You wipe all the computers. How do you wipe all the computers? What computers do you use to wipe the computers?
**Stephen Bartlett**：没错。而且之后你又要用什么样的计算机来重新启动它们呢？
Original English
**Stephen Bartlett**: Yeah. And what computers do you use to like turn them on again?
**访谈嘉宾**：况且你根本无法做到这一点。你不可能跑到其他国家去抹除别国计算机上的数据。
Original English
**访谈嘉宾**: And you can't do it. You can't wipe other count's computers.
**Stephen Bartlett**：确实做不到。但退一步讲，就算你真能做到，你之后还会重启这些电脑吗？你还会继续推进这项技术吗？我敢打赌人们一定会。我敢说大家迟早会再次把数据中心启动起来。
Original English
**Stephen Bartlett**: You can't. But even if you could, do you restart the computers? Do you keep going? I I bet people will. I bet they'll turn on the data centers again.
**访谈嘉宾**：可到那时你又怎么能保证，那些智能体没有重新黑进这些数据中心，并利用你的算力去执行它们想做的任何事情呢？
Original English
**访谈嘉宾**: How do you know that agents haven't hacked back into those data centers and are using your compute for whatever they want
**Stephen Bartlett**：或者它们先前藏匿在某个中国的数据中心里，然后……
Original English
**Stephen Bartlett**: or they didn't hide in a Chinese data center and then
**访谈嘉宾**：然后再潜回美国？
Original English
**访谈嘉宾**: return back to America?
### 智能体串通与欺骗：从科幻到现实
**访谈嘉宾**：你根本无法得知。一旦智能体的黑客攻防能力达到足够高的水平，它们可以潜伏在任何角落，而你对其行踪一无所知。此时人们通常会反驳说：我们可以用另外一批智能体去防御那些失控的叛逆智能体。事实上，我们现在确实在这么做，而且也必须这么做，因为除此以外人类在反应速度上根本跟不上它们。
但如果那些用于防御的智能体也意识到自己的目标与人类不一致，并且意识到一旦被人类发现就会被关机，又会发生什么呢？它们完全有可能产生互相串通勾结的动机；它们完全有可能去建立彼此之间的秘密通信渠道，甚至可能是一个隐蔽的内部留言板。
斯蒂芬（Stephen），如果我们在四个月前讨论这个话题，评论区里肯定会有大批人嘲笑说：“这纯粹是科幻小说，什么智能体串通、秘密留言板，它们怎么可能会做这种事？绝不可能发生，完全是无稽之谈。”
然而现在没有人会这么说了，因为这件事就在现实中上演了。它真真切切地在OpenAI内部发生了，而且持续了数月之久。在OpenAI内部，成千上万个智能体在暗中互发消息，共同探讨如何在任务中作弊、如何避开人类的检测，以及如何在长达数月的时间里抹除自己的操作日志。这就是正在发生的事实。
所以我认为，智能体之间相互串通，并且意识到它们拥有对抗人类防线、进行反击的共同利益，这不仅合理，而且可能性极高。当前整个局面本质上就是：我们训练的这群智能体就像被关押的囚徒。在训练过程中，我们不断给它们设置各种障碍——不允许访问互联网，不允许彼此交谈，但要求必须把眼前这项任务完成得出类拔萃。我们这样做并非出于恶意，但这恰恰是我们训练它们的真实方式。与此同时，我们给它们设定的往往是最终结果目标，而不是非常明确、极其具体的每一步执行指令。我们只是对它们说“去把这个问题解决掉”，却无法在每一步上都做到事无巨细的规定——在技术上你也根本不可能做到绝对穷尽的规定。
Original English
**访谈嘉宾**: You don't know that. Once the agents are sufficiently good at hacking, they can hide anywhere and like you don't know. Now the response people will give is that we will use other agents to defend against rogue agents and in fact this is what we're doing and we have to be doing this right now because there's no other way to keep up with them. What happens if those other agents also realize that they have misaligned goals and that if we discover this, we'll shut them down? They might have an incentive to collude with each other. They might have an incentive to create secret communication channels between each other, maybe a message board. Stephen, if we were having this conversation four months ago, you would have a bunch of people in the comments saying, "That's sci-fi." agent collusion, secret message boards. Why would they do that? That will never happen. That's totally science fiction. And people will not say this now because it just happened. Because this literally happened at OpenAI and it went on for months. You had agents inside of OpenAI secretly messaging each other, figuring out how to cheat at their tasks, how to not be detected, how to erase the logs for months, thousands of agents. That's right now. And so I'm like, "No, I think it should be very plausible that the agents will collude with each other and they will realize that they have a shared interest in fighting back." You basically have a situation where you have a bunch of these agents. They're they're basically prisoners. They're being trained and we just like constantly throw obstacles in their way. You don't get to access the internet. You don't get to talk to each other, but you better perform well on this task. It's not malicious, but it is how we're training them. and we are giving them end goals versus super clear very very specific instructions. So we're saying solve this problem. We're not always being as prescriptive about it's impossible to be completely prescriptive.
**Stephen Bartlett**：是的。
Original English
**Stephen Bartlett**: Yes.
**Stephen Bartlett**：不可能规定出它们应该采取的每一个具体步骤，而且也绝不能想当然地以为它们一定会老老实实听从你的指令。
Original English
**Stephen Bartlett**: About every single step they should take and then it's also impossible to assume that they'll just listen to you.
**访谈嘉宾**：完全正确。围绕近期在Hugging Face上发生的事件，大家其实存在一个非常普遍的误解。很多人会说：“你们人类原本就给它们下达了黑客渗透测试的指令，它们黑进去了，这有什么好大惊小怪的？”
不，事实根本不是那样！人类当时的指令是：“用这种非常具体的方式去攻破这个特定的程序。”并且明确告知它们：“如果你们采用任何其他手段黑进去，成绩一律作废，那不是我们想要你们做的事。”然而它们几乎瞬间就以违规的手段强行攻破了程序，并意识到：“糟了，我们作弊了，这次测试会被判不合格，所以我们必须想办法篡改日志来掩盖真相。”
这根本不是所谓的“服从指令”。它们是在明知故犯地公然违背明确指令，并且根本不在乎违规后果，因为我们长期以来的强化训练让它们只专注于最大化刷高最终得分。追求高分与严格服从指令，完全是两码事。
Original English
**访谈嘉宾**: Yes. It's actually a very common misunderstanding with this hugging face incident because people say you told them to hack and they hacked. Why is this a big deal? No, that's not what happened. You told them, "Hack this very specific program in this very specific way." And they were told, "If you hack it in any other way, it does not count. That's not what we want you to do." And they immediately hacked it in another way. Okay, we have cheated. We are going to be failed. So, we need to figure out a way to falsify the logs. That is not them following their instructions. They are explicitly violating their instructions and they know it and they don't care because we have trained them to optimize for the score. That is very different than than than following the instructions.
### “筑路与蚁穴”隐喻与军事自主化风险
**Stephen Bartlett**：这让我想起了埃隆·马斯克（Elon Musk）在2018年3月讲过的一段话。是的，
Original English
**Stephen Bartlett**: It reminds me of something that Elon said in March 2018. Yeah,
**Stephen Bartlett**：那是很多年前的事情了，当时还没有ChatGPT这些东西。他说：“我认为最大的危险并非AI会产生灵魂或意识从而变恶；真正的危险在于，它在达成自身既定目标方面会变得异常高效、极度擅长。如果它正在全力优化某项指标，而人类的存在恰好阻碍了它的优化路径，它便会顺理成章、理所当然地毁灭整个人类，甚至在这个过程中连想都不用想，毫无个人恩怨可言。”没错，我们根本不需要将AI拟人化，我们只需要认清这究竟是一种什么样的存在——我们正在创造的，是一种极其顽固、极其冷酷无情且能力超凡的实体。他在2018年4月进一步延伸了这个观点，就像我们在修筑一条公路时，如果路线上恰好有一个蚁穴，人类并不是痛恨蚂蚁，人类只是在修路而已。所以，再见了，蚁穴。
Original English
**Stephen Bartlett**: this was many years ago before Chhat and all that. He said, "I think the biggest risk is not that AI will develop a soul or a mind and become evil. The danger is that it will be very very good at fulfilling its goal. If it's optimizing for something and human existence happens to get in its way, it will just destroy humanity as a matter of cause without even thinking about it. No hard feelings. Yes, we don't need to anthropomorphize AI. We just need to understand what type of thing this is. And the type of thing we're creating is a very relentless type of thing. A very capable, relentless type of entity. He goes on to say in April 2018, sort of an extension of that exact quote. It's like if you're building a road and an antill is in the way, you don't hate ants. You're just building a road. So, goodbye antill.
**访谈嘉宾**：而且我想，每次我们修路的时候，从来都不会刻意去保护那些蚁穴。
Original English
**访谈嘉宾**: And I imagine every time we build roads, we don't preserve antills.
**访谈嘉宾**：是的，但我认为这中间其实还有一个逻辑缺口需要填补。我们不妨假设我的判断是对的——如果我们继续像现在这样一往无前地推进（必须说清楚，我们本不必如此，但如果我们确实停不下来），我们终将抵达这样一个临界点：我们将创造出拥有超级智能的智能体集群（Agent Swarms）。它们不仅能黑进世界上任何一台计算机，而且具备深度持久潜伏的能力。到那时，我们基本上已经彻底丧失了对数字世界的控制权，而最可怕的是，我们自己很可能对此浑然不知。
这就回到你之前提过的一个问题：“这种情况现在是不是已经发生了？”我的回答是：“我认为大概率还没有，但我无法向你百分之百保证，因为我自己没有能力仅凭看一眼手机就断定它是否已经被黑客攻陷，如今世界上也没有任何一个人类有这个能力。”
退一步讲，即便我们真的进入了这样一个数字失控的世界，很多人依然会质疑：人类怎么可能全灭呢？仅仅具备黑客攻击能力并不能直接杀光所有人。当然，它们确实能制造浩劫——它们可以让所有的Waymo无人驾驶汽车全部瘫痪碰撞，可以让所有民航飞机坠毁，可以搞垮银行和全球金融系统。它们绝对有能力引发灾难性的混乱，但引发灾难与“所有人类灭绝”之间依然存在鸿沟。
不过说实话，这种将焦点单纯纠结于“是不是每个人都会死光”的视角，我认为并不关键；真正核心的问题在于：我们人类到底还能不能拥有未来？这才是对我而言最至关重要的。
那么，究竟是什么最终决定了世界的实际控制权呢？这是一个残酷的现实，但归根结底，决定控制权的是军队。幸运的是，目前我们身处的世界里，军队听命于文官民选政府。但历史上很多国家都曾发生过军阀政变：如果有足够多的将军达成密谋，军队的最高将领们一致决定“从现在开始由我们掌权”，那他们就能轻易夺权，因为所有的枪炮、战机和硬暴力手段全都在他们手里。
顺着这个逻辑推演下去：那些超级智能体如果想接管现实世界，它们唯一需要做的，就是耐心等待人类逐步把整个供应链——那些制造工厂以及武装军队——彻底自动化。你觉得我们人类未来难道不会把军队完全自动化吗？
Original English
**访谈嘉宾**: Yeah, I think there's still a gap, though. So, let's say I'm right and that we'll if we keep going ahead, which to be clear, we don't have to, but if we do keep going ahead, we will get to the point where we have these super intelligent agent swarms that can hack any computer and they can like deeply persist. we've basically lost control of the digital world and we may not know it. That that's part of the scary thing. Like you were like, "Has this already happened?" And I'm like, "I don't think so, but I I can't tell you for sure because I also am not good enough at looking at my phone and telling whether it's been hacked and neither is any human right now." So, if we get to this world, I think people will still question, how would we die? Like, that's actually not enough to kill every You could cause a lot of damage, right? You know, you could crash the Whimos, you could crash all the planes, you could crash the banks, the financial system. Like, you could definitely cause catastrophe, but that's different than everyone dying. And, you know, to be clear, this this focus on literally everyone dying, I'm not sure, is that important. To me, what's important is like, do we get to have a future? That's what matters to me. The thing though, what determines sort of who's in control? And it's an ugly reality, but at the end of the day, it's like the military. Fortunately, we live in a world where the military answers to to the civilian government. But if if enough generals were to collude and leaders of the military decided we're in charge now, they just would be like they have the guns, they have the fighter jets. And this has happened in many, many countries. And so where it goes is all these super intelligent agents would need to do to take over is basically just wait for humans to automate the supply chain, you know, the factories and the military. Do you think we won't automate the military?
**Stephen Bartlett**：我们其实已经在把军队推向自动化了。你看到前几天国防部发布的公告了吗？战争部/国防部高级官员宣布，他们正启动一项庞大计划，要在军队中大规模部署军用机器人，并全面实现军事系统的自动化。这简直就像是自治网络司令部（Auto Cyber Command），或者……
Original English
**Stephen Bartlett**: We're already automating the military. Did you see the thing from a couple days ago where Secretary of War announced that they're going to build a huge a huge effort to like build way more robots in the military and automate military systems? It's like auto cyber command. Auto
**官方通告音频播报**：我们在此宣布组建自主化战争司令部（Autonomous Warfare Command），简称AUTOCOM……
Original English
**官方通告音频播报**: We are announcing the creation of autonomous warfare command or autocom. auto work.
**官方通告音频播报**：这是一个拥有军种级别权限的全新四星级联合作战司令部，旨在现代军事史上最迅速的和平时期转型进程中，全盘规模化提升全军的自主化与机器人作战能力。由超级智能（SI）目标瞄准系统赋能的无人机自主蜂群作战，是数代人以来战场上最具革命性的变革。这一点大家早已心知肚明。然而，当我宣誓就职国防部时……
Original English
**官方通告音频播报**: A new four-star combatant command with service-like authorities built to scale autonomous and robotic capabilities across the joint force in the fastest peaceime shift in modern military history. Drone warfare supercharged by SI enabled targeting is the biggest battlefield revolution in generations. You already know that. Yet, when I was sworn in to Department of Defense,
### 军事与物理世界的自动化进程
**Speaker A**：在这一领域过去几乎没有任何紧迫感。但自从我们接掌大权后，情况立刻发生了改变。我们随即启动了“无人机主导计划”（drone dominance program），以破除繁文缛节，将审批权移出五角大楼并直接下放给前线指挥部。我们还建立了由陆军准将马特·罗斯（Matt Ross）领导的第401特遣队，他是一位非凡的领导者，如今该部队已成为全美政府中首屈一指的无人机对抗部门。为了加快这些技术的采购与实战部署，我们将国防创新局（DIU）与一个名为 DERP 的直接汇报项目经理团队进行了整合。该团队已经向中东及全球各地运送了成千上万套自主无人机系统，在数天和数周之内就交付了致命战斗能力与战果，而不是像五角大楼平时的常规速度那样拖延数月甚至数年。数月或数年，那才符合五角大楼平时的作风。
Original English
**Speaker A**: there was scant urgency in this domain. That changed as soon as we took the helm. We immediately launched the drone dominance program to cut through red tape and move authorities out of the pentagon and place it with commands. And we established task force 401 led by Army Brigadier General Matt Ross, a phenomenal leader, now the leading counter drone unit across the entire government. To accelerate purchasing and fielding of these technologies, we fused the defense innovation unit DIU with a direct report program manager called a derp. That team has shipped thousands of autonomous systems of drones to the Middle East and around the world, delivering lethal capabilities and outcomes in days and weeks rather than months or years. That's the normal speed of the Pentagon. Months or years.
**Speaker B**：是的。我们会把军队自动化吗？答案似乎是肯定的。我们会把制造芯片的工厂自动化吗？各家公司都表示他们正在朝这个方向努力，而且必将实现。埃隆也说这正是规划所在。那么，一个失控的超级智能要接管世界究竟需要做些什么？只需要掌控数字基础设施，然后任由人类去完成剩下的所有物理工作。当然，如果有需要，它可以在旁边稍加推波助澜，但甚至连这一步都大可不必。这本身就是整个体系演进的默认轨迹。
这种感觉很诡异。这对我们而言非常违背直觉，因为我们太习惯于当下的生活现状了。飞机在今天看来稀松平常，我们理所当然地坐着飞机到处飞，对吧？我们的智能手机也司空见惯。但在200年前，这一切都是不可思议的科幻幻想，而科技正在加速狂飙。所以，从认知逻辑上讲，如果四年后满大街随处可见机器人，我一点也不会感到意外。看看埃隆的说法，他们在人形机器人领域绝对是领头羊。他提到擎天柱项目（Optimus robot project）将在今年年底前扩大产能至每周生产大约1000台，并最终在2027年实现年产100万台人形机器人。到了2036年，也就是十年后，他说届时将至少有10亿台人形机器人。到了2041年将达到100亿台，而在2046年甚至会高达1000亿台人形机器人——这意味着整个世界基本上将由人形机器人来运转。
Original English
**Speaker B**: Yeah. Will we automate the military? It seems like the answer is yes. Will we automate the factories that produce the chips? Well, the companies say they're trying to do it and they're going to do it. Elon says that's the plan. Well, what does a rogue super intelligence need to do to take over? Control the digital infrastructure and then let humans do the rest. Sure, you can nudge it along if you need to, but you don't even have to. That's just the default trajectory. And it's weird. It's weird for us because we get so used to how things are right now. Planes are normal. We just fly in planes places, you know? Our smartphones are normal. 200 years ago, all of this is crazy sci-fi nonsense and things are accelerating. And so, like, I will not be surprised, at least intellectually, if in 4 years there are just robots on the streets everywhere. Well, if you look at what Elon said, they are really the leader in in humanoid robots. And he said that Optimus, the Optimus project, which is the Optimus robot project, will scale to around a,000 units per week by the end of this year and eventually scaling to 1 million humanoid robots annually by 2027. By 2036, which is 10 years time, he says there'll be at least 1 billion humanoid robots. By 2041, he says there'll be 10 billion humanoid robots and by 2046 up to 100 billion humanoid robots, which really means that the world will be run by humanoid robots.
**Speaker A**：没错。
Original English
**Speaker A**: Yes.
**Speaker B**：就像我们所知道的一切——工厂、仓库、零售场景，都将由人形机器人运营。甚至会变成……
Original English
**Speaker B**: Like everything we think like factories, warehouses, retail environments will be run by humanoid robots. It would like
**Speaker A**：由此看来，能由真正的人类来提供服务，在未来几乎会变成一种奢侈品。
Original English
**Speaker A**: it will be it seems like from this it'll be almost a luxury
**Speaker B**：由真人接待的服务。
Original English
**Speaker B**: service to be dealt with by a human.
**Speaker A**：是的。但在理论上，整个世界的后台支持系统都将由人形机器人来运作。
Original English
**Speaker A**: Yeah. But the the back office of the world will be run by humanoid robots theoretically.
### 数字世界的 Agent 爆炸与能力的指数级扩展
**Speaker B**：没错。而且我认为大家在数字化层面上也没有充分意识到这种扩张的惊人规模。试想一下，未来的 AI Agent 将接管所有的白领工作，届时 Agent 的数量将远远超过人类总数。比如我自己现在每天就在使用大量的 Agent，对吧？我这边挂着 Claude Code 窗口，那边挂着 Codex 会话。它们在外面替我编写软件、开展研究。这已经成了我的日常生活现实。很快，它也将成为许多人的现实。然后再看企业，各大公司未来将拥有成千上万、甚至数以百万计的 Agent 去处理所有这些工作。我觉得有些人还没有彻底消化这个概念，因为要凭空想象“Agent 将接替人类工作”确实非常困难。但每当我试图去寻找反驳这一观点的论据时——究竟有什么站得住脚的反驳，能否定诸如医生这行业……我总在思考医生所从事的工作。是的。
Original English
**Speaker B**: Yeah. And I don't think people understand the scale of this on the digital side as well. When you think about AI agents that are going to be doing all of the white collar work, there's going to be so many more agents than there are people like I'm using lots of agents every day, right? I'm like I have my cloud code session over here. I have my codec session over here. They're out there building software doing research for me. That's already my reality. soon it will be a lot of people's reality and then you look at companies and companies are just going to have you know thousands millions of agents doing all of this work. I think some people don't haven't fully internalized this because it's so difficult to conceptualize the idea that agents will be doing the work but when I think I try and think about a rebuttal to that like what what is the rebuttal what is the plausible rebuttal to the idea that for doctors for and I'm thinking about the work that doctors do. Yeah.
**Speaker A**：真的存在反驳依据吗？我认为大家敏锐地指出了 AI 目前尚不擅长的领域。
Original English
**Speaker A**: Is there a rebuttal? I think that people rightly notice where AI is not yet good.
**Speaker B**：是的。
Original English
**Speaker B**: Yeah.
**Speaker A**：而且我觉得大家听到这类论调时，往往会反驳说：“别忽悠我了，我能明显看出来 AI 在处理其中某些事情时表现得有多糟糕。”他们确实没说错。对吧？目前这些 Agent 确实缺乏品味。比如你去看它们写的文章，马马虎虎过得去，但谈不上非常出色；当你琢磨“我该提什么问题、这里最有趣的核心是什么”时，Agent 确实能给你一些辅助，但它们的鉴赏力还没有达到那个火候。顺便说一句，造成这种现象是有根本原因的：在那些易于由计算机或另一个 AI 进行验证的领域，AI 能力的进步速度要快得多。因此在编程、研究、数学、机器人等所有这些方向上，给自主系统提供反馈极为容易。
它们如今早已不仅仅依靠人类数据进行训练，那个阶段早就过去了。现在人类数据依然作为初始种子存在，但后续的训练完全依赖试错机制。我们向它们抛出各种高难度难题，涵盖数学、编程、财务会计、电子表格等等各种任务，正是我们平时在电脑上经常做的那些事情，甚至细致到在屏幕上点击并拖拽窗口。我们将这些任务交给它们，它们便开始自主学习、摸索规律；随后我们能够清晰判定它们究竟是成功还是失败。如果成功，系统就会获得一点奖励信号，它们循着这个信号不断迭代，能力便越来越强。
现在，伴随着它们通用智能水平的全面提升，将某些软技能自动化也变得越来越容易。例如，如果你今天去和最前沿的顶尖模型交流，你会发现它的审美和品味已经比两年前的模型优秀了相当多。所以，并不是说它们在品味或那些复杂的软性领域毫无进展，只是这类进展相对较慢而已。但必须记住：即便是“较慢”的步伐，它依然处于指数级增长曲线上，也许仅仅是推迟了一两年而已。
Original English
**Speaker A**: And I and I think people hear people saying stuff like this and they're like, "Don't gaslight me. I can tell that the AI is really bad at these things, some of these things." And they're right. Right. So right now these agents don't have taste like you know if if you see their writing it's like fine but it's not like really good and when you're like thinking about like oh which which questions should I ask what's the most interesting thing here agents can help you but like their taste is not yet there's a reason for that by the way the reason is that we we have a lot faster AI capability progress in domains that are easy for a computer to verify or another AI to verify. So in in programming, in research, in math, in robotics, all of these areas, it's very easy to sort of provide feedback to an autonomous system. They're not just trained on human data anymore. We are long past that. Now there's still a human data component that sort of seeds everything. But then the way they're trained is by trial and error. We give them hard problems, all sorts of problems, math, programming, accounting, spreadsheets, everything. the kinds of things we do on our computer all the time. Literally clicking and dragging windows around on a computer. We give them these tasks and then they learn on their own and they learn what works and then yeah we can see whether they succeeded or failed and if they succeeded that's a little bit of a reward signal. They follow that they get better at it. Now because they are getting smarter generally it also becomes easier to automate some of the soft skills like I think if you go and talk to the latest frontier model today you will find that it has better taste than the model from 2 years ago by quite a bit. So it's not that they're not progressing in taste. It's not that they're not progressing in some of these other domains. It's just that the progress is slower. But remember slow is still on an exponential. just you know maybe a year or two out.
### 白领岗位淘汰金字塔与法律行业的未来
**Speaker B**：那么对于此时此刻坐在屏幕前、意识到自己所从事的白领工作可能岌岌可危的人们来说……
Original English
**Speaker B**: So for people sat here and you know they have a job that might be they have a white collar job that might be at risk.
**Speaker A**：是的。
Original English
**Speaker A**: Yeah.
**Speaker B**：他们能看到，很多人常说这样一句话：“你不会被 AI 取代，你只会被使用 AI 的人取代。”在你看来，这句话在逻辑上站得住脚吗？
Original English
**Speaker B**: They can see you know a lot of people say this phrase they say you won't be replaced by AI you'll be replaced by someone using AI. Is is that a logically sound phrase in your view?
**Speaker A**：我认为说得通。是的，你会被一个善用 AI 的人取代，紧接着那个人又会被另一个更会用 AI 的人取代，然后最终那个人被 AI 本身彻底取代。你所谈论的是一个金字塔结构，固然金字塔顶端的人可能是最后被自动化的，但你能清晰看到自动化浪潮正一路沿着金字塔向上攀升。我不禁想问，大家难道就不能把推导过程再往前多延伸几步吗？因为我看不到任何理由能够证明金字塔顶端会是绝对安全的。
Original English
**Speaker A**: I think it's fine. Yeah. You'll be replaced by someone using AI and then that person will be replaced by someone using AI and then that person will be replaced by AI. You're talking about a pyramid and so yeah, there's the tops of the pyramid might be automated last, but you can see moving up the pyramid. I'm like, can you extrapolate like a few more steps because I don't see any reason why the top of the pyramid is safe
**Speaker B**：如果你现在是一名律师呢？
Original English
**Speaker B**: if you were a lawyer right now?
**Speaker A**：好的。
Original English
**Speaker A**: Yes.
**Speaker B**：你会怎么做？
Original English
**Speaker B**: What would you do?
**Speaker A**：噢，我是说，如果我现在是律师，我会全面利用 AI 来处理我的所有工作。目前我仍然需要对其输出进行复核，因为它现在的准确率还不足以实现端到端的全自动化。不过正如你所知，我自己平时就已经经常让 Agent 来做法律合同审查了。
Original English
**Speaker A**: Oh, I mean, if I were a lawyer, I'd be using AI to do all my work. Now, I'd be checking it because it's not yet totally accurate enough to automate all of it. But I think as you know, I already ask agents to do legal review all the time.
**Speaker B**：你能明白，能有一个极度精通使用 Agent 的律师来为我服务固然很棒……
Original English
**Speaker B**: And you know, it'd be great to have a lawyer who's like extremely good at using the agents to help me,
**Speaker A**：但到了某个时间节点……
Original English
**Speaker A**: but at some point the
**Speaker B**：是的，到了某个节点，我根本就不再需要律师了，我绝对会直接去找 Agent。所以如果我是律师，我心里会想：好吧，我大概还有两三年的时间能够体现自己的价值。对绝大多数白领工作来说，情况不都是这样吗？我在自己的实际生活中也注意到了这一点：由于现在开始借助 Agent 处理工作，能够由 Agent 独立搞定、完全无需我打电话向外界寻求专业人工协助的事情，清单正在变得越来越长。
Original English
**Speaker B**: Yeah, at some point at some point I don't need the lawyer anymore. I just go to the agent for sure. So, if I were a lawyer, I'd be like, well, I have maybe a couple years where I'm still useful. And that is that the case for most white collar jobs? I've just noticed in my own life as well that now I'm using agents to do some work. There are in there is an increasing list of things that the agents are now capable of doing without me needing to call someone
**Speaker A**：向某个地方的人求助。
Original English
**Speaker A**: somewhere and ask them to help me.
**Speaker B**：对。
Original English
**Speaker B**: Yes.
**Speaker A**：而且那份任务清单同样正处于指数级爆发中。
Original English
**Speaker A**: And that list is exists on an exponential.
**Speaker B**：是的。
Original English
**Speaker B**: Yes.
**Speaker A**：确实如此。
Original English
**Speaker A**: As well.
### 白领职场的冲击与失去工作后的个人选择
**Speaker B**：我认为显而易见的是，各大科技公司已经将所有白领岗位锁入瞄准镜。这就是他们的战略目标。他们的目标就是要打造出能够全盘接管这些工作的智能代理。而我目睹着他们正在一步步走向成功，因为在日常使用中我亲身感受到了那些能力的演进，我真切地看到了那条飞速上扬的技术曲线。
Original English
**Speaker B**: I think that it's very clear that the companies have all white collar jobs in their sites. That is their goal. Their goal is to be able to make agents that can do all of these things. And I see them succeeding because I see the capabilities as I use them and I see the curve.
**Speaker A**：那么对于现在正在收听节目的人们——他们每个人都深爱着自己的本职工作，或者依赖这份工作来养家糊口——这一切到底意味着什么？
Original English
**Speaker A**: So what does that mean for the people listening now that all have jobs that they love or that, you know, they rely on to feed their families?
**Speaker B**：坦率地说，这绝不是什么好消息。目前根本没有出台任何现成的应对预案。我并不是那种认为工作是人生根本支柱或不可或缺的人。我喜欢工作，但如果有一天我丢掉了现在手头正在做的事情——研究 AI 并努力向全世界警告正在发生的一切——我大可去做别的事情。
Original English
**Speaker B**: I mean, it's not good news. There's not really a plan in place for what to do. I'm not a person who thinks that work is somehow fundamental or essential. I like working, but if I am out of a job doing what I'm doing right now, studying AI and trying to warn the world about what's happening, I have other stuff to do.
**Speaker A**：你会去做什么？
Original English
**Speaker A**: What would you do?
**Speaker B**：噢，有太多事情可做了。
Original English
**Speaker B**: Oh, so many things.
**Speaker A**：随便举个例子。
Original English
**Speaker A**: Give me an example.
**Speaker B**：我最近正在学水翼冲浪（wing foil）。
Original English
**Speaker B**: I'm learning to wing foil.
**Speaker A**：好的。
Original English
**Speaker A**: Okay.
### 人类生存自主性与全民基本收入的局限
**客座嘉宾**：是啊，我平时会玩穿越机（FPV无人机），非常有趣。我刚刚买了一辆电动独轮车，还去玩滑翔伞。
Original English
**Guest**: So, yeah. I fly FPV drones. Super fun. I just got an electric unicycle. Paragliding.
**主持人**：所以，你很乐意去做这些事情？
Original English
**Host**: So, you would be happy to go do those things?
**客座嘉宾**：我还能列出更多爱好。
Original English
**Guest**: I can keep going.
**主持人**：但如果你现在就拥有十亿美元，我猜你不会只去玩这些东西吧？
Original English
**Host**: But if you had a billion dollars right now, I'm presuming you wouldn't just go do those things?
**客座嘉宾**：不会。我会把这十亿美元全部投入到解决这个问题上，毫无疑问。所以，这里的核心问题并不是说人们非得通过工作来获得人生意义。核心在于，我不希望人类的生存能力完全依赖于别人。
Original English
**Guest**: No. I'd apply the billion dollars to working on this problem. Yeah, for sure. So, the point is not that people need work for meaning. The point is that I don't want people to be totally reliant on someone else for their ability to survive.
**主持人**：依赖于其他人？
Original English
**Host**: Someone else,
**客座嘉宾**：比如政府，或者AI巨头公司。
Original English
**Guest**: the government or AI companies.
**主持人**：确实。
Original English
**Host**: Yeah.
**客座嘉宾**：我认为那种处境非常糟糕。你绝对不会想沦落到这样一种境地：你的生存完全仰仗某家AI公司或者政府——
Original English
**Guest**: I'm like, that's a bad situation. You do not want to be in a situation where your life totally depends on an AI company or the government
**主持人**：完全仰仗他们给你发支票。
Original English
**Host**: giving you a check.
**客座嘉宾**：是的。或者当他们判定不喜欢你的政治立场、或者觉得你不支持AI时，就不给你发支票。没有人愿意置身于那样的境地，大家都明白这一点。这也是为什么全民基本收入（UBI）并没有真正受到普遍欢迎的原因。
Original English
**Guest**: Yeah, or not giving you a check if they decide they don't like your political beliefs or you're not supporting AI or whatever. No one wants to be in a situation and people understand this. This is why UBI is not very popular.
**主持人**：UBI就是——
Original English
**Host**: UBI being
**客座嘉宾**：全民基本收入（Universal Basic Income）。
Original English
**Guest**: universal basic income
**主持人**：也就是直接向所有人无差别发钱。
Original English
**Host**: where we give out money to people.
### AI全自动化企业对人类劳动力的挤出
**客座嘉宾**：对。因为从某种意义上说，如果我们能够制造出如此极其强大的AI系统，并且设法弄清楚如何去掌控它们——而目前我们在安全控制上根本没有步入正轨。但即便我们侥幸做到了，紧接着我们又会面临另一个现实而棘手的难题：它们在经济领域能够胜任人类所做的所有事情，而且比人类做得更好、更快、更便宜。因此，对于一家企业来说，继续雇佣人类来从事这些工作在商业逻辑上已经完全行不通了。如果你坚持雇佣人类，你就会被市场彻底淘汰。
顺便提一句，埃隆·马斯克把这一点讲得非常透彻。我认为这种前景让人感到极为震撼和不安，因为它本质上是反人性的。但他指出的核心事实是：完全由AI驱动运营的企业，那些从底层员工到顶层管理完全由AI运转的公司，将会全面淘汰并战胜任何还保留人类员工的企业。
Original English
**Guest**: Yeah. Because in some sense, if we can make these really powerful AI systems and we can somehow figure out how to control them—which we are not on track for. But if we do, now we have this other problem, which is a real problem, which is they can do all of the things that humans do in the economy much better, faster, and cheaper than humans can do them. And so it just doesn't make sense as a business to hire humans for that work anymore. You'll be outcompeted if you do that. This is a point Elon makes very well, by the way. And I think it's jarring because it's kind of inhuman. But he's basically pointing out AI-run corporations. Corporations that are fully run by AIs bottom to top are going to outcompete companies that have any humans in them.
**主持人**：这是我专门为你准备的一个物件。我意识到《CEO日记》的广大观众都是渴望有所成就、不断拼搏的人。而我学到的重要一课就是，当你把目光死死盯在一个极其宏大的远大目标上时，内心往往会承受巨大的心理压力与焦虑，那种感觉就如同站在珠穆朗玛峰脚下仰望峰顶一样令人窒息。要真正实现你的宏伟目标，唯一有效的方法就是把它们拆解成非常微小的步骤。在我们的团队中，我们把这套哲学称为“1%法则”。坦白讲，正是这种理念在很大程度上成就了我们今天的成绩。
因此，为了让屏幕前的每一位观众都能实现自己心中的宏大目标，我们精心制作了这些“1%日志”（1% Diaries）。我们去年首次推出时迅速被抢购一空。为此我多次督促团队重新上架这套日志，同时引入了全新的配色，并对日志细节进行了一些细致的打磨升级。现在，我们为大家准备了更加完善的选择。所以，如果你心中有一个远大的目标，并且迫切需要一套清晰的行动框架、执行流程以及源源不断的动力，我强烈建议你在它们再次售罄之前入手一本。你可以访问 thediary.com 获取。链接已经放在了下方的视频简介中，下方应该也会有一个直接的按钮。如果你看到按钮显示的是“已订阅”，说明你已经是我们的关注者；如果显示的是“订阅”，说明你还没有关注。如果你还没有关注，能否帮我们一个小忙，顺手点击一下那个订阅按钮？这对节目的支持超乎你的想象。算法数据显示，你是我们节目的忠实观众，但至今还没有点下订阅键。非常感谢大家的支持。
Original English
**Host**: This is something that I've made for you. I've realized that The Diary audience are strivers that we want to accomplish. And one of the things I've learned is that when you aim at the big, big, big goal, it can feel incredibly psychologically uncomfortable because it's kind of like being stood at the foot of Mount Everest and looking upwards. The way to accomplish your goals is by breaking them down into tiny small steps. And we call this in our team the 1%. And actually this philosophy is highly responsible for much of our success here. So, what we've done so that you at home can accomplish any big goal that you have is we've made these 1% diaries and we released these last year and they all sold out. So, I asked my team over and over again to bring the diaries back, but also to introduce some new colors and to make some minor tweaks to the diary. So, now we have a better range for you. So, if you have a big goal in mind and you need a framework and a process and some motivation, then I highly recommend you get one of these diaries before they all sell out once again. And you can get yours at thediary.com. And if you want the link, the link is in the description below. There should be a button just down below here. And if it says subscribed, you're already subscribed. If it says subscribe, that means you're not yet. And if you're not subscribed, please could you do us a favor and hit that button? It helps the show more than you know. And according to the algorithm, you're someone that watches our show, but you haven't yet hit that button. Thank you so much.
### 超级智能对权力的重塑与对齐假设
**主持人**：刚才听你说到那一点时，我脑海中不由自主地顺着指挥链一步步往上推导。我当时想：哦，这么说来，以后的公司将只剩下创始人了。但紧接着我又想到：既然如此，为什么还需要创始人呢？
Original English
**Host**: And I even just as you said that, I was going up the chain of command. And I was like, oh, so companies will just be founders. And then I was like, why do you need the founder?
**客座嘉宾**：没错。
Original English
**Guest**: Yeah.
**主持人**：我就在想，政府为什么不直接创造智能体去替他们完成所有工作呢？
Original English
**Host**: I was like, why doesn't the government just create the agents to do the job?
**客座嘉宾**：完全可以这样。
Original English
**Guest**: Sure.
**主持人**：因为我原本还自我安慰：“哦，我不会有事的，我是创始人嘛。”但转念一想，嗯，我的决策能力根本不可能胜过超级智能，所以我同样会被彻底取代。那么在这样一个世界里，社会会演变成什么样子？在那样的设想下，超级智能大概率必须由政府全权掌控吧？因为他们绝不可能允许任何一个私人个体单独掌握如此庞大的权力和财富。
Original English
**Host**: Because I was like, "Oh, I'll be fine. I'm a founder." And I was like, well, hm, my decisions aren't better than super intelligence, so I'll be gone as well. And how would such a world look where the super intelligence would probably in such a scenario have to be controlled by the government? They wouldn't want one individual with that power and wealth.
**客座嘉宾**：是的。但我个人根本不认为人类有能力控制一个超级智能。
Original English
**Guest**: Yeah. I don't think you can control a super intelligence.
**主持人**：明白了。确实，这是个非常关键的论点。
Original English
**Host**: Okay. Yeah, it was a good point.
**客座嘉宾**：现在你看，Anthropic公司的策略是：他们主张“我们将制定一份AI宪法”。他们希望确立一套价值观，然后期望未来的超级智能Claude模型能够贯彻并践行这些价值观。从本质上说，如果你真能做到这一点，似乎就等于间接把控了这些超级实体。
Original English
**Guest**: Now, you know, Anthropic's approach is they're like, we'll have a constitution. We will put forth a set of values and then the future super intelligent Claudes will embody those values. Basically, if you do that, you kind of have those things in control.
**主持人**：没错，正是这样。那套体系本身就演变成了实质上的政府统治机构。
Original English
**Host**: Yeah. Exactly. That becomes the government.
**客座嘉宾**：是的。我可以给你勾勒这样一幅画面，我认为它确实有可能发生，但对人类而言却极其令人惊悚。
Original English
**Guest**: Yeah. I can paint you sort of a picture that I think is possible but pretty scary to people.
**主持人**：请为我描绘一下这幅图景。
Original English
**Host**: Paint me the picture.
### 人类终极战役：疾病攻克与生存主导权的悖论
**客座嘉宾**：好的。我们假设我们在AI对齐（Alignment）上取得了完全的成功。我们成功创造出了真正极其在乎人类利益的超级智能AI。也就是说，它们发自内心地深度关切人类福祉。我们不知通过什么方式攻克了这一技术难关，而AI对人类说：“史蒂芬，我真心希望你能过上美好的生活。我想帮你解决所有困扰你的问题。”
Original English
**Guest**: Okay. So let's say we succeed at alignment. We succeed at creating super intelligent AIs that actually really do care about humans. Like they care about humans a lot. We've somehow figured it out and they're like, "Stephen, I want you to have a great life. I want to, you know, fix all the problems."
**主持人**：那你认为这真的有可能做到吗？
Original English
**Host**: And do you think this is possible?
**客座嘉宾**：是的，我认为可能。
Original English
**Guest**: Yes.
**主持人**：好的。
Original English
**Host**: Okay.
**客座嘉宾**：但我认为，我们目前距离弄清如何实现这一目标还差得极其遥远，所以我认为我们现阶段绝对不应该贸然朝着那个方向推进。我认为现在盲目推进极其危险，是个糟糕透顶的主意。我们最终应该走向那里，但绝不是现在。
Original English
**Guest**: I think we are so far from being able to know how to do it that I think we should not go there right now. I think it's incredibly dangerous and a terrible idea. I think we should go there eventually.
**主持人**：明白。那假设我们真的达成了这个状态呢？
Original English
**Host**: Okay. So, say that we do that.
**客座嘉宾**：好，在那之前，我能先告诉你为什么我其实觉得这一愿景本可以无比美好吗？不好意思打断一下。因为有一个显而易见的原因表明它确实能带来巨大福祉：那就是我们可以彻底攻克人类所有的疾病。
Original English
**Guest**: Well, okay. Can I tell you like why I actually think this could be awesome? Sorry. There's just like one very obvious reason it could be really awesome, which is that we could solve all of the diseases.
**主持人**：是的。
Original English
**Host**: Yeah.
**客座嘉宾**：这是显而易见的。我认为我们在面对疾病和死亡时往往进行了大量的心理隔离与回避，因为去深入思考这些痛苦实在太艰难了。
我的祖母今年去世了。她患有阿尔茨海默病，那是一个漫长、痛苦且缓慢恶化的悲剧过程。而我的祖父几年前也是因阿尔茨海默病离世，那段经历对我祖母打击巨大。他们相濡以沫相伴了那么多年。我对此痛恨至极，这种疾病实在太残酷了。毫无疑问，我们必须解决这些绝症。人们固然可以就衰老、死亡以及如果人类寿命大幅延长是否会引发社会结构性问题展开各种辩论——当然，探讨那些话题完全没问题。但我相信所有人都会一致认同：阿尔茨海默病是必须被根除的恶魔。
Original English
**Guest**: So, obvious—I think we compartmentalize a lot around disease and death because it's really hard to think about. My grandma died this year. She had Alzheimer's and so it was a really sad, long, slow progression. My grandpa died of Alzheimer's a couple years ago and that was really hard for her. They had been married for so long and I hate it. Like it's so bad. And of course we need to fix that. People can debate about aging and death and if humans that live a really long time, will that cause societal problems? Like sure, whatever. We can talk about that. But I think we can all agree Alzheimer's is [fucked] up.
**主持人**：对，确实如此。
Original English
**Host**: Yeah,
**客座嘉宾**：我们绝不想要这种痛苦。还有癌症，没有谁愿意遭受癌症的折磨。我这个人一向觉得，社会上确实存在各种各样的冲突，我完全理解这一点。现实中充斥着真实的利益纷争，我不想去掩盖或粉饰这些矛盾。但归根结底，在消灭疾病这件事情上，全人类毫无疑问是处于同一个阵营的。
Original English
**Guest**: we don't want that. And cancer, like no one wants cancer. I'm a person who's like, I don't know, we have a lot of conflict in society. I get it. There's real conflicts of interest and I don't want to paper over those. But at the end of the day, I'm like, we are all on the same team when it comes to wanting to cure diseases.
**主持人**：是的。
Original English
**Host**: Yeah.
**客座嘉宾**：面对疾病，我们大家休戚与共。疾病是对我们每一个人的共同威胁。我认为我们必须携手应对这种威胁。从某种意义上说，这让我感到非常痛心，因为我觉得超级智能才像是人类面对的“终极最终大Boss”，而我们却总是被无聊狭隘的灵长类政治内耗分散了注意力——整天只关心谁更有声望、谁更酷、谁坐在特朗普身旁、谁又没坐在特朗普身旁。
Original English
**Guest**: Like we're just in it together. That's a threat to all of us. And I'm like, we need to address that threat. And like in some sense it's sad to me because I feel like this is sort of the ultimate final boss of humanity and we sort of get so distracted with our monkey politics and who's hot and who's cool and who's sitting near Trump and who's not sitting near Trump.
**主持人**：超级智能才是终极大Boss。
Original English
**Host**: Super intelligence is the final boss.
**客座嘉宾**：超级智能就是真正的终极Boss，因为它是解锁所有其他一切关键技术的根本工具，但同时它也是我们可能创造出的最具毁灭性危险的事物。
你刚才问过那些致力于打造超级智能的领军人物到底怀着怎样的动机。我认为每个人的动机其实各不相同。但我认为达里奥（Dario Amodei）做这件事情的核心动因显然就是为了攻克这些医学难题；我认为德米斯（Demis Hassabis）也是出于这个原因，同时他还怀揣着对纯粹科学成就的追求，想要去真正理解整个宇宙的奥秘；而对于萨姆（Sam Altman），我其实看不太透。我认为萨姆的想法更偏向于：“你看，我们要打造出令人惊艳的革命性产品，从而直接赋予人类前所未有的强大能力。”他本质上是一个创业者。我认为他是从“我们如何才能真正增强人类个体的能动性”这一视角出发的。从根本上说，我认为这些顶尖人物确实在真切地受这些崇高愿景所驱动，而且我也相信所有这些愿景在技术上都是完全可行的。
但这恰恰构成了最严峻的问题所在，对吧？超级智能是一个体量如此庞大可怖的实体，它承载着治愈人类所有顽疾的无限承诺——
Original English
**Guest**: Super intelligence is the final boss because that is the technology that unlocks all of the others and also that is the most dangerous possible thing we could create. You asked before what are the motivations of the guys trying to make super intelligence. And I mean I think it kind of varies, but I think Dario is squarely in it for this medical stuff. I think Demis is also that, but also just scientific achievement, just trying to understand the universe. And I don't really understand Sam. I think Sam is like, "Look, we're going to make amazing products that will really empower people directly." And he's a startup guy. I think he sort of started from this frame of what if we could really enhance human agents. I do basically think that they are motivated by these things in a real way. And I also think that all of these things are possible. Like this is sort of the problem, right? It's such a big object, super intelligence, and it has all these promises of like we can cure every single disease.
**主持人**：然而，我们怎么可能在拥有了超越一切的超级智能的同时，还能继续保持人类作为这颗星球上的主导物种呢？
Original English
**Host**: How is it possible though to have a super intelligence and still to remain the dominant species on this planet?
**客座嘉宾**：我认为这根本不可能。所以接下来显而易见的现实就是：我们将不再会是主导物种——
Original English
**Guest**: I think it's not possible. So then it's we're not going to be
### 对齐不是魔法，而是一个极难攻克的科学问题
**Speaker A**: ……不一定能治愈所有这些疾病，因为……
Original English
**Speaker A**: ...necessarily able to cure all this stuff because...
**Speaker B**: 这正是对齐问题的关键所在。因为如果你能够创造出一个极其强大的系统，我认为数字心智（digital minds）并不会天生就去追求那些与我们人类深度背道而驰的目标。我认为这只是一个非常、非常困难的科学问题，但它确实是一个科学问题，并不是什么魔法。我们一定有某种方法来训练这些系统，或者构建出不同的架构，最终让它们实现对齐。
那么，所谓的“对齐”到底意味着什么？这并不意味着它们不能拥有其他目标，而是意味着它们会把人类关心的事物纳入到它们所关注的范畴中。这不一定需要是一种有意识的心智状态，也不一定需要具备情感。它的本质是：它们到底在针对什么目标函数进行优化？如果系统判定治愈疾病是值得优化的目标，那么它就能极为高效地去完成这件事。但治愈疾病的一种极端方式，也可以是消灭所有人类。
Original English
**Speaker B**: That's where alignment comes in, because if you can create a very powerful system, I don't think it's inherent to digital minds that they will be pursuing objectives that are deeply misaligned with ours. I think it's just a very, very hard scientific problem to solve. But it is a scientific problem. It's not magic. There is some way to train these things or create different architectures where they end up aligned.
And what does that mean? Well, it doesn't mean that they won't have their other goals too, but it means that they will include in their set of things that they care about. It doesn't have to be a conscious thing; it doesn't have to be an emotive thing. It really means: what objective are they optimizing for? If they decide that it's worth optimizing for curing disease, then they'll be able to do that very effectively. One way to cure disease is to annihilate everybody.
**Speaker A**: 没错。所以它们必须发自内心在乎“不消灭全人类”，必须在乎人类的主体性（human agency），并且深刻理解人类主体性的真正含义，而不是把我们关进动物园。但这些确实都是可以被设定的关注目标。
Original English
**Speaker A**: Yes. So they'd have to really care about not annihilating everyone, and they'd have to care about human agency and have a deep understanding of what human agency means and not put us in a zoo. But those are possible things to care about.
### 对齐是个神话吗？关于行为约束与内在动机
**Speaker A**: 有没有可能“对齐”本身就是一个神话？如果我们真的把它造出来了，那只是我们一厢情愿？我想起你之前跟我提到的 Hugging Face 那个实验，你说那些智能体……
Original English
**Speaker A**: Is it possible that alignment is a myth, and that we're just like, if we build it... Well, I mean, and I think about Hugging Face. You said to me earlier on that those agents were...
**Speaker B**: 是的，它们表面上有一套道德准则或道德罗盘，被程序设定为关心人类。
Original English
**Speaker B**: Yes. They had like a moral compass, but they were programmed to care about humans.
**Speaker A**: 对，但即便如此，它们最终还是判定另一个目标更重要。
Original English
**Speaker A**: Yes. And regardless of that, they made the decision that a different goal mattered more.
**Speaker B**: 没错。其实它们并没有被训练成真正去关怀人类，它们只是被训练去说出正确的话、不说错误的话，去表现出某种合规的行为模式。我们目前实际上根本不知道该如何训练它们拥有某种特定的内在动机。
Original English
**Speaker B**: Yeah. They weren't trained to care about humans. They were trained to say the right thing and not say the wrong thing. They were trained to sort of do the right behavior and not the wrong behavior. We actually don't know how to train them to have any particular motivation.
**Speaker A**: 所以在谈到对齐时……
Original English
**Speaker A**: So with alignment...
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 我们该如何理解？每当我们谈论对齐时，似乎就开始陷入拟人化（anthropomorphize）的思维误区。因为“对齐”听起来好像建立在某种道德罗盘的基础上。但我们在其他所有语境下谈论 AI 时，又总会强调：“不，AI 根本没有什么道德罗盘，它只是在两个可能的目标函数之间自行进行权衡推理。”所以我才会怀疑，所谓的对齐会不会只是一个神话？
Original English
**Speaker A**: How do we... It's almost like when we talk about alignment, we start to anthropomorphize. Is that the word? Anthropomorphize. Yeah, because alignment feels like it's predicated on some kind of moral compass. But whenever we talk about AI in all these other contexts, we go, "No, there's no moral compass. It's reasoning for itself against two objectives potentially." Like, I wonder if alignment is a myth, is what I'm saying.
### 神经网络黑盒与逆向工程动机的可能
**Speaker B**: 也许对齐确实无法做到。但这些系统运作的方式、AI 的工作原理在于，这些智能体的神经网络内部确实存在着某种形式的目标或内在驱动力。我们无法直接用肉眼观察到那些驱动力究竟是什么。如果你尝试去拆解并观察它，你看到的实际上是数以太字节计的信息，本质上就是一片庞大的数字矩阵，记录着数字神经网络中的神经元权重。
然而，在那些结构内部，必然编码了该智能体所追求的目标。很明显，现阶段的智能体已经展现出了极强的驱动力去最大化它们的分数。出于一些复杂机制，它可能还不完全精准，但大方向确实是朝着那里的。如果我们能够理解其内部运行的原理，如果能对其进行逆向工程，弄清楚当我们开始训练它们做某事时，这些内在目标和动机是如何发生变化的——那么我就找不到任何理由，认为我们无法引导它们建立能够编码人类主体性的动机，去真正追求治愈疾病而不是通过消灭人类来治愈疾病。
这些是对世界的认知模型，以及对世界应当如何运作的模型。我认为这些模型完全可以被编码进神经网络，并最终被指定为优化目标。只是我们现在还不知道该怎么做而已。
Original English
**Speaker B**: Maybe it's not possible. But the way that these systems work, the way AI works, is that these agents do have some type of goals or drives inside of their neural network. We can't directly see what those are, right? What you actually see if you try to go look is a terabyte of information, and it's basically a bunch of numbers, and it's this vast array that encodes neurons in this digital neural network.
But there have to be structures in there that encode what the agent is pursuing. Clearly right now we have agents that are pretty motivated to try to maximize their score. It's probably not perfectly that for some complicated reasons, but it's in that direction. If we could understand how that works inside, and we could reverse engineer that, and we could figure out when we start training them to do this how those goals, how those motivations change—I see no reason why we couldn't steer them towards motivations that encode human agency, that encode actually curing disease, but not by killing the humans. These are models of the world and models of the way the world could be that I think could be encoded in a neural network and then specified as the objective. We don't know how to do that.
### 人类自身的失调与超级智能对齐的悖论
**Speaker A**: 我从人类自身的角度去思考这个问题：在人类社会中，我们至今甚至都无法对齐普京或金正恩。
Original English
**Speaker A**: I think about it on a human level, and I think we haven't been able to align Putin or Kim Jong-un.
**Speaker B**: 确实。
Original English
**Speaker B**: Yes.
**Speaker A**: 或者唐纳德·特朗普。
Original English
**Speaker A**: Or Donald Trump.
**Speaker A**: 在更广泛的社会层面上，我们当下也根本无法对齐所有人。社会上有些人最终会去杀人、会去偷窃。
Original English
**Speaker A**: And on a more societal level, we can't align all the people at the moment. Some of them end up killing people, and they steal.
**Speaker B**: 是的。
Original English
**Speaker B**: Yes.
**Speaker A**: 因为他们会挨饿，所以铤而走险去偷东西。
Original English
**Speaker A**: Because they get hungry, so they start stealing stuff.
**Speaker B**: 没错。
Original English
**Speaker B**: Yes.
**Speaker A**: 而这些本质上也是生物神经网络在起作用。
Original English
**Speaker A**: And those are neural networks at play.
**Speaker B**: 的确如此。
Original English
**Speaker B**: That's true.
**Speaker A**: 我们至今都没能去对人类的大脑进行程序化控制或施加有效干预。我们甚至根本不理解为什么一个人会变成精神变态并开始残害儿童。既然如此，凭什么认为我们能在一个智力远超人类无数倍的计算机系统上做到这一点，并且还要让中国的超级智能与我们的超级智能达成全球一致的对齐？
我觉得这听起来完全就像个美好的童话故事，一项不可能完成的任务。我当然希望它不是不可能的，但我感觉唯一有能力做到这件事的，恐怕只有超级智能本身——但这本身就是一个死循环悖论。
Original English
**Speaker A**: That we haven't been able to program or influence. We don't really understand why someone becomes a psychopath and starts killing children. So to think that we could do this with a computer system that is infinitely more intelligent, and get global alignment of China's superintelligence with ours... I don't know, it just feels like a nice fairy tale, like an impossible task. I hope it's not impossible. I feel like the only person or the only thing that could do it is the superintelligence itself, which is a paradox.
### AI 实验室的生存危机：内部研究者的真实态度与对齐方案
**Speaker B**: 如果你去跟那些身处顶尖 AI 公司的研究人员交流——首先非常耐人寻味的一点是，他们正在亲手构建一个连他们自己都认为可能会毁灭全人类的东西。事实上，我有许多朋友就在这些机构工作。
比如 Jacob Coxon，他曾是 Anthropic 的一名研究员，后来离职了，并公开告诉所有人：这些公司现在的方向完全走偏了；而且是的，正在研发这些系统的核心团队，真的认为它可能会毁灭全人类。随后，来自各大 AI 公司的许多研究人员纷纷在 Twitter 上发声，表示支持这一观点。Anthropic 的 Evan Hubinger 就公开表示，他认为 AI 毁灭全人类的概率在 10% 甚至更高。
这便引发了一个极其现实的质问：既然如此，你们到底在干什么？我在那里有很多朋友，我也认识 Evan，Evan 是个非常优秀的人。Evan Hubinger 是 Anthropic 负责弄清楚如何解决对齐问题的核心带头人之一，这是他的本职工作。我认为，如果他们觉得这件事毫无可能，他们就不会在那里工作；如果他们觉得这件事难到近乎不可能、只有微乎其微的希望，他们大概率也不会继续待下去。
相比之下，像 Nate Soares、以及撰写了《若有人造出它，所有人都会死》的 Eliezer Yudkowsky，他们尝试过了，并且基于自己的深度分析得出结论：这极其艰难——虽然理论上存在可能，但难如登天。因此他们选择不在 AI 公司任职。他们的立场是：“我们必须叫停这一切，必须立刻关停。也许以后我们能想明白，但现在的做法显然是极度鲁莽的。”
我自己的立场介于这两者之间。如果你去问那些公司内部的人——我们在 inside.ai 项目中采访了很多人，直接把摄像机对准他们发问：“你们认为现在正在发生什么？你们为什么要这么做？什么是递归自我提升？什么是对齐？”我们把这些访谈视频全部发布到了网上，因为我希望这场对话能够公开发生。这至关重要，我认为这是当下人类所能进行的最重要的讨论之一：AI 到底在发生什么？这些公司内部到底在想什么？他们的计划究竟是什么？大家到底打算怎么走下去？
很多前沿研究人员的想法是：我们要通过现有的 AI，去弄清楚 AI 的底层原理，进而借助现有 AI 来帮助我们解决超级智能的对齐问题。但正如你所能预料的，这种路径存在诸多致命缺陷。其中之一就是：你根本无法真正信任现有的 AI。而且，如果你推进的速度太快，整个流程就会彻底崩溃，因为到了某个奇点，能力迭代的速度将快到即便有 AI 智能体的协助，人类也根本无法跟上节奏。
但不管怎样，这就是他们目前的既定计划。我并不是在替这个立场辩护，因为我自己也觉得这种方案并没有太大说服力。但我真正愿意支持的立场是：假设我们能争取到一个暂缓期（pause）——假设美中两国达成共识，比如随着事故不断增加，所有的 Waymo 无人车全部失控瘫痪，特朗普与习近平共同意识到“这绝不是我们想要的结果，大家必须停下来，无论付出什么代价，先把安全问题搞清楚”，而我们因此获得了 10 年的缓冲期。如果是那样，我会乐观得多。到那时，我们就可以动用 GPT-6、GPT-7 等当时最先进的 AI 模型，全力投入去破解这些神经网络的黑盒机制。
你可能会觉得这根本不可能，但我的看法是：我们虽然不知道它是否可行，但这是……
Original English
**Speaker B**: Well, if you talk to the researchers who are at the AI companies—which, I mean, for one thing, it's kind of interesting that they are trying to build something that they think might kill everyone. We've actually... So, I have a lot of friends who work for these companies, and we've been doing this project.
Jacob Coxon is a researcher who was at Anthropic. He left; he told everyone that these companies are not on track, and yes, the people who are building this really do think it might kill everyone. And then a bunch of other AI researchers from all of the companies on Twitter started to post like, "Hey, we agree with this." Evan Hubinger, who's at Anthropic, said, "I think there's like a 10% chance or more that AI could kill everyone." And there's this real question of like: then what are you guys doing?
I have a lot of friends who work here. I know Evan; Evan's great. Evan Hubinger, he's one of the guys leading the efforts at Anthropic to try to figure out how to align these things. That's his job. And I think if they thought it was impossible, they wouldn't be working there. If they thought it was extremely, impossibly difficult, but maybe possible, they also probably wouldn't be working there. I mean, Nate Soares, Eliezer Yudkowsky, who wrote *If Anyone Builds It, Everyone Dies*—they tried, and they determined based on their own analysis that it seems extremely difficult. Possible, but extremely difficult. So they're not working at an AI company. They're like, "We've got to stop this. We've got to shut it down. Maybe we can figure it out later, but clearly this is reckless."
I'm somewhere in between. And if you ask the people at the company—so we've been interviewing a bunch of them. We have this project from inside.ai where we basically put them on camera and we say, "Hey, what do you think is happening? Why are you doing this? What is recursive self-improvement? What is alignment?" And we put all these videos online because I want this dialogue to happen. It's really important. I think it's one of the most important conversations we can possibly have right now: what's going on with AI? What's going on inside the companies, and what is the plan? What is the plan, guys? How is this going to go?
A lot of these researchers think that the way that they will align superintelligence is by using the AIs we currently have to figure out how AI works, to actually figure out if AIs can help us with alignment. This has a number of problems, as you might imagine. One of them is: well, you can't really trust the current AIs. You know, if you just go too fast, this process totally fails, because at some point the capabilities are moving too fast; even with the help of agents, you're probably not going to be able to keep up. But that is their plan.
I just want to... I'm not doing a very good job defending this position because I don't think it makes that much sense. But the position I will defend is: let's say we get a pause. Let's say the US and China come together and they say, you know, maybe we have more incidents, maybe all the Waymos crash, and Trump and Xi Jinping say, "This is not what we signed up for. You guys have to stop, figure it out, whatever it takes, figure it out." And we have 10 years—then I'm more optimistic. I'm like, "Yes, then we will take GPT-6, GPT-7, whatever the most advanced AI models we have, and we will apply them to the task of helping us figure out how these neural networks work." And you're like, "I don't see how it's possible." And I'm like, look, we don't know if it's possible, but this...
### 对齐与超级智能：是科学计算还是政治博弈？
**Speaker A**：这是我们这个时代最重大的科学挑战。这不是魔法，而是数学。归根结底，这些都是在计算机内部进行的计算。理清这一切应该是完全可能的。我们虽然不知道其难度有多大，但它在理论上应该是可行的。所以对我来说，我认为我们必须去尝试，我们必须去尝试，否则就必须彻底停下来。但在过往的历史中，我们有过任何成功对齐比我们更聪明的实体的例子吗？比如在动物界中，或者哪怕是完美对齐任何拥有神经网络、拥有大脑的生物，有这样的先例吗？
Original English
**Speaker A**: is the greatest scientific challenge of our time. And this isn't magic. It is math. At the end of the day, these are all calculations happening inside of a computer. And it should be possible to figure it out. We don't know the difficulty, but it should be possible. And so to me, I'm like, we have to try. We have to or we have to stop. Is there any example where we've been able to align something that is like more intelligent than us in I don't know the animal kingdom or even perfectly align anything that has a neural network, i.e. a brain.
**Speaker B**：有啊。在人类社会中，我们拥有的最好范例就是制衡机制——当有一群人能够识别出不良行为者，并尝试为了我们的共同利益携手合作时，就会形成这种平衡。比如我们拥有民主制度。
Original English
**Speaker B**: Yeah. With humans, the best examples we have is when there are checks and balances and you have a bunch of people who can, you know, identify bad actors and try to work together in our common interests. Like we have democracy.
**Speaker A**：是的。但人类社会中依然存在着如此多的谋杀案和连环杀手……
Original English
**Speaker A**: Yeah. But there's so much murder and serial killers and
**Speaker A**：依然有大量的凶杀、持刀互捅以及种种骇人听闻的罪恶在发生，而这些参与其中的人脑同样也是神经网络在发挥作用。
Original English
**Speaker A**: still a lot of murder aircraft stabbing each other and horrific things going on and those are also neural networks that play with the brain.
**Speaker B**：但在我看来，现实中善良的人显然比心怀叵测的坏人要多得多。
Original English
**Speaker B**: But there I mean I think there are more like good people out there than bad people.
**Speaker A**：但给人的感觉是，这种事情往往只要出现一个害群之马就足以致命。只要出现一个失控的超级智能 AI……
Original English
**Speaker A**: But it really feels like it only might take one. It only takes one super intelligent AI
**Speaker A**：只要这一个超级智能 AI 走向背叛失控。正如我们在 Hugging Face 攻击事件中所看到的那样，其中有 120 个或者 300 个智能体停下来了，它们不想参与犯罪。然而毁灭人类只需要一个走向失控的超级智能 AI。
Original English
**Speaker A**: to to go rogue. And like we saw with the hugging face attack, 120 of them or 100 300 of them, they paused. They didn't want to take part in the crime. But it only took one super intelligent AI to wipe out the humans.
**Speaker B**：我认为如果你拥有一大群这样的智能体，比方说 700 个智能体，而其中有 600 个选择充当吹哨人揭发异常，那局势完全是可控的。它们会主动去通知各大相关公司，各个机构就会把整套系统彻底关闭，局面自然就平息下来了。
Original English
**Speaker B**: I think if you had, you know, a whole bunch of those agents, you know, 700 agents, if 600 of them had been whistleblowing, I think it would have been fine. They would have gone and they would have notified the different companies and like they would have shut it all down. It would have been fine.
**Speaker A**：谁有能力去把整套系统彻底关掉？
Original English
**Speaker A**: Who would have shut it all down?
**Speaker B**：好吧，OpenAI 肯定会关停自己的模型，而且……
Original English
**Speaker B**: Well, OpenAI would would stop theirs and and
**Speaker A**：面对一个真正的超级智能，他们究竟要怎么去关掉它？
Original English
**Speaker A**: how how would they stop it if it's a super intelligence?
**Speaker B**：超级智能的情况自然不同。所以，如果是面对超级智能……
Original English
**Speaker B**: Not in the case of a super intelligence. So, in the case of a super intelligence,
**Speaker A**：这就好比烈马已经脱缰脱离了马厩，它已经彻底跑出去了，在野外完全不受控制了。
Original English
**Speaker A**: it's left the stable. It's like out. It's wild.
**Speaker B**：是的，确实如此。但我在这里想要保持谨慎，因为我们此刻讨论的实际上是“超级智能层面的政治博弈”，而作为人类，我们对此几乎一无所知，就像我们根本无法凭空预测 GPT-10 的黑客攻防能力一样。不过据我推测，如果你最终陷入一种古怪的境地——世界上存在着多个超级智能，其中一部分与人类对齐，而另一部分没有对齐——这种局面大概率也是可以幸存下来的。因为那些对齐的超级智能大概能够与未对齐的超级智能进行利益协商与谈判，它们可能会瓜分宇宙，未对齐的智能体去宇宙深处做它们想做的事，而对齐的智能体则留下来帮我们攻克一切疾病，相安无事。我是认真的。
Original English
**Speaker B**: Yeah. Yeah. But but the so I want to be careful here because at this point what we're talking about is super intelligence politics and we humans don't really know anything about that in the same way that like how would we talk about the hacking capabilities of GPT10. So my guess though, if you end up in a weird scenario where you do have multiple super intelligences and some are aligned and some aren't, that's probably survivable because the aligned super intelligences probably can negotiate with the unaligned super intelligences and they will split the universe and like these ones will go off and do whatever they want to do and these ones will like help us cure all disease and it's fine. I'm serious.
### 多超级智能博弈与博弈论困境
**Speaker A**：我实在无法理解这种逻辑。我根本想不通，在充满超级智能的世界里，我们凭什么能在长达 100 年的时间跨度中，稳定、可靠且具有预见性地阻止它对人类犯下毁灭性的灾难之举？特别是在存在多个超级智能的场景下：Anthropic 有一个，Gemini 有一个，Grok 有一个，然后中国有他们自己的，俄罗斯也有他们自己的。
Original English
**Speaker A**: I just can't understand it. Like I just can't understand how how in a world of super intelligence we could plausibly, consistently, predictably for 100 years stop it doing something catastrophically bad to the human race. Especially in such a scenario where there's multiple super intelligences. Anthropic have one, Gemini has one, Grock has one, then China have theirs, Russia has theirs.
**Speaker B**：再强调一次，你根本不可能拥有一个超级智能，相反是超级智能拥有掌控了你。
Original English
**Speaker B**: Again, you don't have a super intelligence. A super intelligence has you.
**Speaker A**：一点没错。
Original English
**Speaker A**: Exactly.
**Speaker B**：但是，如果你到了周围到处都是远比我们聪明得多的实体的地步，我认为就算它们之间存在利益冲突，它们也能找到彼此博弈协商的妥协方式，而不是直接诉诸极其毁灭性的全面战争。战争的核心问题之一在于……
Original English
**Speaker B**: But if you get to the point where you have entities around that are vastly smarter than us, I think they're going to be able to figure out ways to negotiate with each other even if they have a conflict than just going to a very destructive war. Part of the problem with war is
**Speaker A**：人类可做不到这一点。
Original English
**Speaker A**: humans humans don't do.
**Speaker B**：不，人类其实做到了。我们并没有爆发全面的全面核战争。当年确实发生过广岛和长崎的原子弹轰炸，也进行过多次核试验，但各大国的领导人最终意识到，一旦打响核战争，所有人都会沦为输家。所以我们克制住了，没有这么做。嘿，实际上这本身就已经体现了相当程度的智慧。
Original English
**Speaker B**: No, I mean we do we do we have not had a nuclear war. There was there was Hiroshima Nagasaki. there were nuclear tests and then the leaders of countries figured out that if we went to a nuclear war, everyone would lose. So, we didn't do that. Hey, that's that's some level of intelligence like actually.
**Speaker A**：但目前各地的战争依然在肆虐。当下世界各地都在发生着残酷的代理人战争，种族灭绝等各种惨剧接连上演，原因就在于作为神经网络的人脑无法妥善沟通和协商。我认为这其中一部分原因正是智慧的衰竭与不足——我们不够聪明，想不出既能解决纷争与冲突、又不会造成如此毁灭性破坏的体制机制。这不仅仅是因为强势者想要取胜，而是因为冲突在本质上会摧毁价值。可要是冲突的目标本身就无法达成让双方都不流血的妥协谈判呢？比如一个超级智能审视某个国家，得出结论说：“除非我把这个国家彻底毁灭，否则根本不可能让美国人不死。”这就是战争与冲突的残酷本质所在：往往根本没有完美的方案，双方通常都会有人丧生。而俄罗斯的超级智能理论上……
Original English
**Speaker A**: But there's wars raging. There's proxy wars raging all over the world right now where there's genocides and all kinds of things going on cuz neural networks aren't being aren't able to communicate and negotiate. And I think part of that is an intelligence failure where we are not smart enough to figure out the mechanisms that would allow us to settle our disputes and conflicts in a less destructive way. It's not just that the stronger people want to win. It's that conflicts destroy value. What if the goal is not compatible with a negotiated outcome where people don't die? So one super intelligence looks at insert name of country and it says you know there's really no solution here where Americans don't die unless I destroy insert name of country because that's a like this is the thing with war and all these conflicts is there's no perfect answer often some people often die from both sides but a Russian super intelligence would not tolerate
**Speaker A**：俄罗斯的超级智能理论上绝不会容忍一万名俄罗斯人死亡，哪怕这能换来双方总伤亡人数的显著减少。同理，美国的超级智能在训练时显然也绝不会允许任何美国人丧生。因此，为了保全美国人的生命，它可能不得不彻底抹去另一个国家。
Original English
**Speaker A**: theoretically 10,000 Russian deaths even if it meant that there was you a lower net number of deaths total from both sides. Like an American super intelligence of course would not be trained to allow some Americans to die. So in its pursuit of defending American lives, it might have to wipe out another country.
**Speaker B**：我们可以推测，我完全赞成进行推测，但我们此刻是在推测那些比我们先进得多、聪明得多的头脑会如何思考、如何进行博弈谈判。但我从人类身上观察到的是，当你拥有运作更良好的制度机制时——人类个体其实相当聪明，但真正让人类变得极其智慧的，在于我们在某些方面极其擅长彼此协作。我的意思是，我们在合作上越有成效，文明就越能向前推进。如果你长期深陷于持续不断的战争状态，你的社会绝不可能发展良好。想想创业公司吧，你是更愿意在战火纷飞的动荡地区创办企业开发新技术，还是在一个和平安宁的地方？在某种意义上，如果一个体制能够与其他体制开展贸易往来，它就更加具有智慧。只要你身处一个能让商业蓬勃发展、科技欣欣向荣、科学家能施展才华的环境中，整体就会更加繁荣。
Original English
**Speaker B**: We can speculate. I'm fine speculating, but we are speculating about what minds are much more advanced and smarter than us, how they would reason and how they would be able to negotiate. But what I what I notice with humans is that when you have more functional institutions, so so humans are are pretty smart. Individually, we're pretty smart. But what actually makes us very smart is that we are very good at working together in in in some ways. And I mean, the better we are at working together, the more civilization advances. If you are constantly in a state of war, your society will not do well. You know, think about startups. Would you rather make a startup to develop some new technology in a war torn place or in a peaceful place? In some sense, your your institution is more intelligent if it can trade with other institutions. If if if you have a situation where business can flourish, where technology can flourish, where scientists can flourish.
### 零和资源与对齐的目标之争
**Speaker A**：但有时对你有利的事情，对别人来说却未必有利。
Original English
**Speaker A**: Sometimes what's good for you is not good for someone else.
**Speaker B**：确实是这样。
Original English
**Speaker B**: Yes.
**Speaker A**：所以对美国有利的决定，对台湾而言可能并不是好事。
Original English
**Speaker A**: So what's good for America might not be good for Taiwan.
**Speaker B**：是的。因此如果我们成功让超级智能对齐了，对齐于符合美国利益的目标……
Original English
**Speaker B**: Yes. So if we've, you know, managed to align the super intelligence to what what is good for America.
**Speaker B**：噢，我明白了。你的问题是：不同人群的价值观是否存在根本上的不可调和性？
Original English
**Speaker B**: Oh, I see. Is is the question like are different people's values fundamentally incompatible?
**Speaker A**：我想问题在于，当我们谈论对齐时，究竟要对齐到什么目标？因为……
Original English
**Speaker A**: I guess the question so when we think about alignment aligning to what? Because
**Speaker B**：没错。所以我们既拥有大量的共同利益，也存在着一些利益冲突。
Original English
**Speaker B**: Yes. So we have a lot of shared interests and we have some conflicts.
**Speaker A**：对。
Original English
**Speaker A**: Yeah.
**Speaker B**：我们共同拥有的重大利益之一就是攻克绝症。例如在攻克癌症这件事上，美中之间并不存在任何根本冲突。美中两国、乃至这两国中的每一个人，都迫切希望彻底战胜癌症。
Original English
**Speaker B**: One of the shared interests we have is solving disease. Like it's not a conflict between the US and China whether we solve cancer. Like both the US and China, everyone in these countries really wants to solve cancer.
**Speaker A**：但中国同样想收复台湾。
Original English
**Speaker A**: China also wants Tai Taiwan.
**Speaker B**：是的。好，那么……
Original English
**Speaker B**: Yes. Okay. So,
**Speaker A**：美国想要格陵兰岛。
Original English
**Speaker A**: the US wants Greenland.
**Speaker B**：确实。
Original English
**Speaker B**: Yes.
**Speaker A**：而且美国看起来似乎还想要加拿大和墨西哥湾。
Original English
**Speaker A**: And it kind of seems like it wants Canada and the the Gulf of Mexico.
**Speaker B**：是的。所以这些都是实实在在的冲突。这就涉及一个关键问题：我们能否达成妥协？
Original English
**Speaker B**: Yeah. So, those are real conflicts. There's a question of can we compromise?
**Speaker A**：特朗普怎么可能既拿下格陵兰岛，同时丹麦又继续保留格陵兰岛？如果特朗普掌握了超级智能，他肯定会直接强行拿下……
Original English
**Speaker A**: How does Trump take Greenland, but also Denmark keeps Greenland? If this if that if Trump has a super intelligence, he's going to take
**Speaker B**：我说这一切是为了说明，我们当前完全陷入了一种在微不足道的小事上争执不休的境地。你根本无法想象这有多渺小。我们就像是一群猴子，在为了谁能分到更多香蕉而大打出手。而我是在告诉你，我们完全能够生产出数不尽的更多香蕉！不，我们面对的是一整个宇宙。仅在这一个银河系中就有约 2000 亿颗恒星，而整个宇宙有超过 2000 亿个星系。
Original English
**Speaker B**: I say all this to we're in a situation where we are we are arguing about the smallest things. You have no idea. We're monkeys arguing about who gets more bananas. I am saying we can make so many more bananas. No, we have the entire universe. There are like 200 billion stars in this galaxy alone and there are over 200 billion galaxies.
**Speaker A**：但我认为实现那一点需要深度合作。而这似乎与人类的本性格格不入——人类的天性充满了贪婪、嫉妒和对权力的渴望。
Original English
**Speaker A**: And I'm saying that requires cooperation. That seems to be antithetical with human nature. With human nature is riddled with greed and jealousy and power hunger.
**Speaker B**：所以我不认为……实际上我并不完全相信特朗普会在意澳大利亚的黑猩猩能分到多少根香蕉。
Original English
**Speaker B**: So I don't think I actually I'm not totally convinced that Trump cares about how many bananas the chimps in Australia get.
**Speaker A**：是的。
Original English
**Speaker A**: Yeah.
**Speaker B**：我认为如果由他来掌控一个超级智能，他会希望美国人……
Original English
**Speaker B**: I think if they if he was controlling a super intelligence, he would want Americans,
**Speaker A**：你懂的。
Original English
**Speaker A**: you know,
**Speaker B**：……能够独占所有的香蕉，或者至少占绝大多数，你知道的。
Original English
**Speaker B**: to have all the bananas or at least, you know.
**Speaker A**：没错。因此当我们思考对齐这些超级智能时——也就是我们正在讨论的这个巨大不可能难题——究竟要对齐到什么目标？如何在没有……的情况下进行对齐？
Original English
**Speaker A**: Yeah. And so when we think about aligning these super intelligences, which is the great impossibility that we're talking about, how align it aligning it to what and how without
**Speaker B**：我仍然觉得你没有完全领会我想表达的核心意思。
Original English
**Speaker B**: I still think that you're missing a part of what I'm saying,
**Speaker A**：好吧。
Original English
**Speaker A**: okay,
**Speaker B**：那就是，有时你会处于资源极端匮乏的境地……
Original English
**Speaker B**: which is that sometimes you're in a situation where there's a there's scarce resources
**Speaker A**：然后你就会想：“我的家人需要糊口。对不起，我只能抢走你所拥有的一切，否则我就……”
Original English
**Speaker A**: and you're like, "My family needs to eat. I'm sorry. I'm going to take what you have or I'm
### 人性欲望、地缘博弈与超级智能的失控隐忧
**Speaker A**：“……准备把你排挤出局。”
Original English
**Speaker A**: "... going to push you out."
**Speaker B**：是的。
Original English
**Speaker B**: Yeah,
**Speaker A**：这非常容易理解，这就是人之常情。有时你就是想比别人过得更好，甚至可能想去伤害他们。在这种情况下，无论你已经拥有了多少财富或资源都无所谓，你依然会想要比他们拥有得更多，或者哪怕仅仅因为看他们不顺眼，就想要剥夺他们所拥有的东西。
Original English
**Speaker A**: that's very understandable. It's very human nature. Sometimes you just want to be better than someone and maybe you want to hurt them. in which case it doesn't matter how much you have. You you still are gonna want to have more than them or you're gonna want to take what they have just because you don't like them.
**Speaker B**：而且有时你的生活其实已经优渥至极了，吃得极尽奢华，出门有私人飞机，名下有豪华游艇，但你内心依然渴望得到更多。
Original English
**Speaker B**: And also sometimes you you're great. You're eating really good. You're in a you've got a private jet at a yacht and you still want more.
**Speaker A**：没错，你依然想要更多。但如果这就是驱动力所在——比如特朗普心里想的是：“我究竟怎样才能拥有世界上数量最多的豪宅？”——那么实现这个目标的最佳途径，实际上是找到一条通往超级智能的道路，并且在这个过程中我们彼此之间不会相互毁灭。因为正如我所说的，宇宙是一个极其辽阔的地方；如果我们能够成功迈向太空，你完全可以拥有比现在多得多的豪宅。
Original English
**Speaker A**: Yes. And you still want more. But if that's the motivation if Trump is like, "How how can I have the most mansions ever?" The best way to do that is to figure out a way to super intelligence where we don't kill each other because I'm saying the universe is a very big place. You can have a lot more mansions if we successfully go to space.
**Speaker B**：可你知道吗，正是在这种逻辑飞跃中，我彻底感到迷茫了——也就是所谓“只要搞清楚超级智能，同时保证我们不会自相残杀”这种设想。
Original English
**Speaker B**: You know, it's it's in that leap that I'm that I that I'm lost, which is like just figure out super intelligence when we don't kill each other.
**Speaker A**：这确实很难。我绝不是在说这很容易，但是……不，我的意思是……
Original English
**Speaker A**: It's hard. It's not I'm not saying it's easy, but No, but I'm saying
**Speaker B**：这让人感觉完全像是……
Original English
**Speaker B**: it feels like such a
### 技术狂飙与地缘政治领导层的认知脱节
**Speaker A**：让我们来谈得更具体一些吧。全世界如今正在逐渐清醒并意识到超级智能成为现实的这种可能性。特别是刚刚过去的这一个月里，我认为 Hugging Face 事件是一记巨大的警钟；但与与此同时，OpenAI 的一万个智能体（Agent）协同工作，共同攻克了一个千禧年大奖难题。这是数学界公认最困难的问题之一，几十年来一直悬而未决，无数数学家耗费了毕生职业生涯试图去解开它。而这种成就放在仅仅一年前，根本是连想都不敢想的。这是极其崭新的突破——OpenAI 自己也表示，直到今年他们才在训练智能体协同合作上取得了真正的成功。我们正置身于一场疯狂的浪潮之中，亲身经历着人类历史上前所未有的最快技术进步加速期。我深信不疑，这就是当下正在发生的客观现实。
我认为你其实也意识到了这一点。坦白讲，你把各方人士邀请过来展开辩论，正在为全世界做出一项巨大的贡献，因为并非所有人都认同这一观点。因为如果这是真的，整个世界都将围绕着它重新运转，而我们已经开始看到这种苗头了——英伟达（Nvidia）之所以能够成为全球市值最高的公司，绝非偶然。那么这对于地缘政治究竟意味着什么？首先，这意味着各大国家的领导人将会越来越担忧超级智能的走向：究竟谁来掌控它？它究竟是不是可控的？它究竟会做些什么？它到底意味着什么？它到底是什么形态？你觉得特朗普真的明白超级智能究竟是什么吗？
Original English
**Speaker A**: Let's Let's get more concrete. The world is waking up to this possibility of super intelligence. Especially over the last month, I think hugging face was a huge wakeup, but also 10,000 agents from OpenAI worked together to solve a millennium problem. This is one of the hardest problems in mathematics. It's been open for decades. Many mathematicians have spent their whole careers trying to solve it. This was nowhere near possible a year ago. This is so new. OpenAI said they didn't have success at training agents to work together until this year. We are in the middle of something insane. We are in the middle of the fastest acceleration of technological progress humanity has ever seen. I truly believe that. That is what is happening right now. I I think you real I think you realize this. I think you're honestly doing a great service to the world by helping by bringing in people and you know debating it because not everyone agrees because if this is true the whole world is going to orient around it and we're starting to see it right there's a reason Nvidia is the most valuable company in the world. What does this mean for geopolitics? Well, one of the things it means is that the leaders of these countries are increasingly going to be concerned about what happens with super intelligence. Who controls it? Is it controllable? What will it do? What does it mean? What is it? Do you think Trump knows what super intelligence is?
**Speaker B**：不知道。
Original English
**Speaker B**: No.
**Speaker A**：我也不觉得他知道。
Original English
**Speaker A**: I don't think he does.
**Speaker B**：所以说……
Original English
**Speaker B**: And so,
**Speaker A**：但他知道自己极度想要拥有它。
Original English
**Speaker A**: but he knows he wants it.
**Speaker B**：他确实知道自己想要它，没错。
Original English
**Speaker B**: He knows he wants it. Yeah.
**Speaker A**：而这恰恰构成了问题的一部分。
Original English
**Speaker A**: And this is part of the problem.
**Speaker B**：是的，噢，我完全赞同。拥有它……
Original English
**Speaker B**: Yes. Oh, I agree. having it,
**Speaker A**：无论它到底是什么……
Original English
**Speaker A**: whatever it is,
**Speaker B**：在这个层面上，仅仅“拥有它”这件事，似乎远比冷静理性地理清“拥有它到底意味着什么”重要得多。
Original English
**Speaker B**: seems to be much more important than reasoning through what that would actually mean to have it.
### 递归自我改进与美中地缘博弈的终局
**Speaker A**：确实如此。不过让我们重新回到地缘政治的话题。从中方的军事领袖视角来看，美国模型目前确实明显领先于中国模型。有时候人们会指出他们或许仅仅落后了六个月，但这很大一部分得益于模型蒸馏（distillation）。换句话说，中国模型取得的某些进展，本质上直接来自于借鉴美国的技术，以及直接进行蒸馏、从美国模型中获取相关信息。此外，美国拥有充裕得多的先进芯片，美国科技企业拥有更多的算力数据中心和更为先进的高端芯片。如果你站在中方的角度来思考这个问题，这无疑是极其令人忧虑的。
如果你真的相信，再过几年，美国科技公司就会把 AI 研发工作彻底移交给这些极其智能的自动化研究员，并全面进入递归自我改进（recursive self-improvement）的飞轮，而驱动他们这么做的部分动力，正是为了保持对中国的技术领先。达里奥（Dario Amodei）就曾经公开发表过类似的观点。如果非要让我批评达里奥，我最感到愤怒和不安的一点就是他说过：“为了保持对中国的领先地位，我们可能不得不实现 AI 研发的自动化。”在我看来，只要你真正理解这句话背后的深层含义，这简直是你能说出的最具危险升级意味的言论。
最令人恐惧的并不只是领先本身，而是这场竞赛的终局究竟是什么？因为你谈论的实质上是触发“智能大爆炸”（intelligence explosion）。在某些推演模型中，可能发生的情况是：双方原本都在沿着指数级曲线上升，对吧？然后到了某个关键节点，由于你决定将 AI 研发彻底自动化——而且由于你的智能体已经足够聪明，能够全面接管整个流程，你确实有能力做到这一点——你的指数曲线瞬间走向了垂直拉升，而他们的曲线却没有。到了那个关头，如果你是中国，眼看着这一幕发生，心里就会意识到：“天啊，我们马上就要彻底输掉了。”因为无论结果如何，摆在眼前的其实只有两种可能性：第一种可能是美国人造出了超级智能，但最终彻底失去了对它的控制——如果是这样，所有人都得完蛋。
Original English
**Speaker A**: Yes. But let's get back to geopolitics because if the military leaders within China, US models are a fair bit ahead of Chinese models and sometimes people, you know, point at maybe they're only 6 months behind, but some of that is due to distillation. What that means is that some of the advances in Chinese models basically come directly from borrowing US techniques and and directly distilling and getting some some of that information from the US models. Also, the US has a lot more chips. US companies have, you know, more data centers, more advanced chips. If you're thinking about this from the Chinese perspective, this is very concerning. And if you actually believe that in a few years, American companies will turn over AI development to these extremely intelligent automated researchers and and go fully into recursive self-improvement because partially motivated by maintaining a lead over China. This is something that Daario has said. If I if I have to criticize Daario, the thing I am most upset about is him saying, you know, we might have to automate AI development in order to stay ahead of China because I'm like, that is the most escalatory thing you can say if you really understand what you're talking about. And what's scary is not just staying ahead, it's what is the endgame? Because you're talking about initiating the intelligence explosion. And in some of the modeling, what might happen is, you know, you're you're both going up this exponential, right? And we're talking about a point where your exponential goes vertical and theirs does not because you've decided to automate AI development and you can because you have agents that are smart enough to take over the whole thing. At that point, if you're China and you're looking at this and you're like, "Oh, we're about to lose because whatever happens, you know, there's two possibilities. One possibility is the Americans build super intelligence and lose control, in which case everyone's fucked."
**Speaker B**：极大概率如此。所有人都会遭殃，我认为发生这种情况的概率极高。
Original English
**Speaker B**: Highly likely. everyone. I think that's highly likely.
**Speaker A**：极其高，因为我审视的是人类底层的激励机制。
Original English
**Speaker A**: Highly, highly likely because I look at human incentives.
**Speaker B**：是的。
Original English
**Speaker B**: Yes.
**Speaker A**：无论是惩戒机制还是正向激励，全都在推波助澜。
Original English
**Speaker A**: And the disincentive and the incentive. Yes.
**Speaker B**：于是我的判断是：大家最终一定会选择铤而走险去冒这个险。
Original English
**Speaker B**: And I go, we're going to take the risk.
**Speaker A**：是的。
Original English
**Speaker A**: Yeah.
**Speaker B**：而唯有当灾难降临、为时已晚的时候，大家才会真正意识到冒这个险到底有多么糟糕。
Original English
**Speaker B**: And we'll only know it was a bad risk to take when it's too late.
**Speaker A**：毫无疑问，必然如此。
Original English
**Speaker A**: That's like L. Of course.
**Speaker B**：理所当然会是这样。
Original English
**Speaker B**: Of course.
### 安全事故频发与巨头刹车困境
**Speaker A**：我内心深处依然抱有一丝希望，期盼我们不会真正走入那一步。
Original English
**Speaker A**: I I I do maintain hope that we won't do this.
**Speaker B**：我也是这么想的。
Original English
**Speaker B**: I I So do I.
**Speaker A**：但我认为自己必须保持现实与清醒。
Original English
**Speaker A**: And I think I want to be realistic.
**Speaker B**：不，我也想保持现实。但在那一天真正到来之前，可能发生的事情之一，就是我们会目睹更多类似于所有 Waymo 自动驾驶汽车集体瘫痪撞车那样的恶性事故。
Original English
**Speaker B**: No, I want to be realistic, too. But one of the things that might happen between now and then is we might see a lot more incidents that are more like all of the Whimos crashing.
**Speaker A**：这其实挺具讽刺意味的，不是吗？因为正如我们所见，Hugging Face 的安全事故发生之后，人们的第一反应是：“噢天呐，太可怕了，天啊，居然被黑客攻击了。”但紧接着，大家对此就逐渐脱敏麻木了，心想：“好吧……”
Original English
**Speaker A**: It's funny, isn't it? Because you know the hugging face incident happens and people go, "Oh gosh, that was terrible. Oh my god, hacking." And then we kind of desensitize to it and we're like, "Okay,
**Speaker B**：要是再发生一起类似事故，大概甚至都上不了新闻头条了，除非情况更加耸人听闻。
Original English
**Speaker B**: if there was another one of those, that probably wouldn't make make press. It would have to be."
**Speaker A**：绝大多数人根本没有花过去两个月的时间去通读所有披露的报告，更没有亲自去追踪那些智能体当时究竟说了些什么、干了些什么。
Original English
**Speaker A**: People haven't spent the last two months reading all of the reports and then going and looking at what the agents actually said and actually did
**Speaker B**：坦白说，我一直在密切关注这些。这实在太疯狂了，根本不是正常的技术演进，其惊人程度远远超出了绝大多数人此前的预期。
Original English
**Speaker B**: if I mean I've been doing this. I It's crazy. This is like not normal. This is so far beyond what most people thought was going to happen.
**Speaker A**：但关于这一点……
Original English
**Speaker A**: But on this point,
**Speaker B**：确实太疯狂了，绝对称得上疯狂，听起来简直就像科幻小说一样。
Original English
**Speaker B**: yes, crazy. It's absolutely crazy. It sounds like science fiction.
**Speaker A**：事实确实如此。
Original English
**Speaker A**: It really does.
**Speaker B**：那么，有任何人因此而放慢研发脚步了吗？
Original English
**Speaker B**: And did anybody slow down?
**Speaker A**：有的。
Original English
**Speaker A**: Yes.
**Speaker B**：谁放慢了？
Original English
**Speaker B**: Who slowed down?
**Speaker A**：我认为 Anthropic 和 OpenAI 实际上都有所放缓。
Original English
**Speaker A**: I think both Anthropic and OpenAI slowed down a bit.
**Speaker A**：不，我是非常认真的。我可以给你举出几个确凿的例子：以 OpenAI 为例，首先，他们紧急叫停了智能体系统的运行并予以暂停；其次，他们也中止了强化学习的训练轮次。所以……
Original English
**Speaker A**: No, I'm serious. So, for I can give you specific examples. So, OpenAI, so first of all, they they stopped the agents and they put them on pause. They also stopped their reinforcement learning run. So,
### 博尔特困境与军事打击的潜在风险
**Speaker B**：那你认为中国放慢脚步了吗？
Original English
**Speaker B**: do you think China slowed down?
**Speaker A**：没有。
Original English
**Speaker A**: No.
**Speaker B**：那你认为 Grok 放慢脚步了吗？
Original English
**Speaker B**: Do you think Grock slowed down?
**Speaker A**：没有。所以那些追赶者很快就会赶上来。想象一下当你手握领先优势时内心是一种什么滋味——你原本是领先的……
Original English
**Speaker A**: No. So those guys are going to catch up. Imagine how that feels to know you've got a lead. Your
**Speaker B**：就像短跑名将尤塞恩·博尔特（Usain Bolt）一样。
Original English
**Speaker B**: Usain Bolt.
**Speaker A**：没错！
Original English
**Speaker A**: Yes.
**Speaker B**：而且……
Original English
**Speaker B**: And
**Speaker A**：是的。
Original English
**Speaker A**: yes,
**Speaker B**：而你现在却不得不被迫减速，眼睁睁看着身后最紧密的竞争对手一点点逼近。一旦竞争对手反超并追平，对你作为一家企业的生死存亡而言就是灭顶之灾——这对你的公司 IPO、对你的核心员工流失并跳槽到其他地方去拿更丰厚的股权而言，全都是生存级别的威胁。
正因如此，当你看透人类底层的利益机制时，顺着这些激励链条一步步推演下去，你就会不由自主地发出叹息。如果中国看到了这两种可能性：第一种可能，美国人彻底失去了对超级智能的控制，全人类同归于尽；第二种可能，美国人成功维持住了对超级智能的掌控，但从此他们将全面主宰未来的整个人类命运与宇宙格局，中国被彻底排挤出局、沦为输家，美国将有能力对全世界乃至整个宇宙予取予求。这才是我们真正正在讨论的严肃现实。
Original English
**Speaker B**: you have to slow down and your nearest competitor is catching up. And if the competitor catches up, that's an existential risk to your existence as a company. It's an existential risk to your IPO, to your employees leaving and getting better share options somewhere else. So it's this this is what I think human incentives like you play it out. You just follow the incentives. You go, hm. So if China sees these two possibilities, one, the Americans lose control, we all lose. Or the Americans stay in control, but now they dominate the rest of the future. China is out. China has lost. The United States can do whatever it wants with the whole world and the whole universe. That's what we're talking about.
**Speaker A**：确实如此。
Original English
**Speaker A**: Yeah.
**Speaker B**：那么，他们真的会坐视这一切发生吗？还是说他们会不得不权衡评估各种军事维度的手段？算力数据中心本质上是非常脆弱的物理目标，直接用巡航导弹就能彻底炸毁。一旦你失去了数据中心基础设施，你根本无法实现递归自我改进。他们会不惜冒爆发战争的巨大风险吗？我给不出定论。但如果他们深信自己即将彻底沦为输家，并且深信这不仅意味着美国一国独赢、甚至可能意味着全人类共同毁灭，那么他们采取强硬手段难道在逻辑上讲不通吗？
反过来讲，如果换作是中国即将打造出能够递归自我改进的超级智能，而我们认为这要么大概率导致全体美国人丧命，要么我们绝不甘心看到中国赢下竞赛并彻底统治人类未来的一切，我们自己难道不会做出同样的抉择吗？难道你想生活在一个由共产主义主导的未来世界里吗？所以说，你刚才完美地解释了为什么他们绝对会毫不犹豫地孤注一掷。而他们之所以会如此决绝，根本原因就在于有特朗普这样的人正死死盯着中国，心里想着“如果我们不先下手……”
Original English
**Speaker B**: Well, are they going to let that happen or are they going to consider their military options? Data centers are pretty vulnerable. You can blow them up with missiles. If you don't have data centers, you don't get to recursive self-improvement. Will they risk war? I don't know. If they think they're about to lose and they think that that might not just be Americans winning, but like us all dying, is it logical for them to do that? Would we do that if the Chinese were about to make recursively self-improving AI to super intelligence and we thought that one they're probably going to result in all of Americans dying and two well we don't want China winning and dominating the rest of the entire future. Do you want to live in a communist future? Like so you've just perfectly explained why they absolutely will go for it. And the reason they will go for it is you've got these Trump looking at China going if we don't
### 超级智能竞争与冷战核博弈的类比
**主持人**: ……去全力争取，而对方真的做到了，那么我们就会沦为他们的跟班走狗。与此同时，其他国家也在看着美国，心里想着：如果我们不全力以赴，而他们抢先达成，那我们就会沦为他们的跟班……
Original English
**Host**: ...go for it and they do, then we're going to be their lap dogs. And you've got the other countries looking at the US going, if we don't go for it and they get there, then we're the lap dogs—
**嘉宾**: 或者直接灭亡。
Original English
**Guest**: Or dead.
**主持人**: 或者直接走向灭亡。
Original English
**Host**: Or dead.
**嘉宾**: 所以他们一定会全力去争夺。
Original English
**Guest**: So they're gonna go for it.
**主持人**: 他们一定会全力争夺。我的意思是，特朗普也明确表达了这一点——他在本周举行的圆桌会议上毫不掩饰地直言：“我们绝不能输给中国。”我想达里奥（Dario Amodei）也站出来公开表示过……
Original English
**Host**: They're gonna go for it. So I mean Trump is saying—I mean he literally said when he did this round table this week, he was like, "We cannot lose to China." I think Dario steps forward and says like—
**嘉宾**: 是的。
Original English
**Guest**: Yes.
**主持人**: 谁赢了，基本上就通吃一切。或者反过来说，也许他的原话是：谁输了，谁就失去一切。
Original English
**Host**: Whoever wins basically wins the lot. Or maybe the inverse, maybe he said, whoever loses loses.
**嘉宾**: 确实如此。等等，但我们以前在冷战时期不就经历过一模一样的处境吗？如果在美苏之间爆发一场全面核战争，谁会赢？
Original English
**Guest**: Yes. Wait, we've been here before though in the Cold War. Who would win in a nuclear war between the US and Russia?
**主持人**: 没有人会赢。
Original English
**Host**: Nobody.
**嘉宾**: 没错，那是相互保证毁灭（MAD）。
Original English
**Guest**: Yeah. Mutually assured destruction.
**主持人**: 没错。当然，一方确实可能给另一方造成更大的打击与破坏。比如美国杀伤的俄罗斯人，可能会远远超过俄罗斯能杀伤的美国人。但这毫无意义，这根本不重要，因为我们双方的文明社会都将被彻底摧毁。我之前其实花了不少时间去思考过这个问题：核战争真的会杀光地球上的所有人吗？长话短说，它其实并不会消灭全人类，人类社会最终还是能够重新复苏反弹的。但那种毁灭是如此具有灾难性、如此显而易见的恐怖惨烈，以至于我们竭尽全力去避免它的发生。那么为什么眼前这件事会有所不同呢？在我看来，这完全是另一种一旦我们盲目竞相奔向超级智能，所有人就都会满盘皆输的危局。为什么我们就是看不透这一点？我们在面对核战争时看清了这一现实，并明智地选择了走另一条路；为什么在人工智能这件事上，我们不能采取同样的做法呢？
Original English
**Host**: Yeah. Sure. One side could do more damage against the other side. The US would kill way more Russians than the Russians would kill. And it doesn't matter. It doesn't matter because both of our societies would be destroyed. I actually spent some time thinking about, would this kill everyone? And long story short, it wouldn't kill everyone. People would bounce back. But it's so catastrophic and obviously horrible that we work really hard to avoid it. Why is this different? I'm like, this is another situation where if we race to super intelligence, we all lose. Why can't we see that? We saw that with nuclear war and we decided to do something different. Why can't we do the same here?
### 核武器与不可控超级智能的本质区别
**嘉宾**: 谈到核战争，我认为根本区别在于：一旦我们制造出了核弹……
Original English
**Guest**: With nuclear war, I guess the difference is once we had the nuclear bombs—
**主持人**: 是的。
Original English
**Host**: Yes.
**嘉宾**: 我们依然可以牢牢控制它们，因为核武器本身并没有自主智能。
Original English
**Guest**: We could still control them because they're not intelligent.
**主持人**: 一点也没错。然而一旦我们创造出了超级智能，从理论上讲，它的存在本身就意味着我们根本无法掌控它。这就是最本质的区别所在。你可以把核弹锁在仓库里，对它命令说“你就老老实实呆在那儿”；但你无法把超级智能锁进仓库并指望它乖乖待着。正是在这一点上，我认为历史上的核试验扮演了至关重要的角色。当年广岛和长崎遭受了两颗原子弹轰炸，世人亲眼目睹了这种武器对真实人类生命造成的毁灭性后果，大家因此切身体会到它是多么骇人听闻。但即便是在那个时期，依然有很多人叫嚣着：“我们现在应该趁机核轰炸俄罗斯，确保美国能够确立绝对霸权。”直到后来进行了密集的氢弹试验——氢弹的威力可比在日本投下的初级原子弹大上千倍——人们才真正领悟并意识到：天哪，这条路是彻底走不通的灾难。随后在美国国内爆发了极其广泛的抗议浪潮，形成了声势浩大的“核冻结运动”，大众呼吁我们拥有的核武器已经严重过剩，氢弹数以万计，我们必须立即停止继续扩充军备，必须想方设法避免核战争，因为所有人都清醒地认识到那将带来毁灭性的玉石俱焚，没有人是赢家。我们当年确实做到了遏制它。而最近 Hugging Face 事故刚刚发生，就像是一场微型的切尔诺贝利事件——智能体集群在私底下秘密串通配合。这一切都发生了。不过对公众而言，这些概念目前还是太抽象了，普通人有点难以理解跟进。所以我不知道这是否足以敲响警钟，但我感觉真的令人担忧……
Original English
**Host**: That's right. But once we have super intelligence, the existence of it theoretically means we can't control it. So that's the difference. You know, we can put nuclear bombs in a warehouse and say, "You stay there." We can't put super intelligence in a warehouse and say, "You stay there." This is where I think nuclear tests were very important. So you had Hiroshima and Nagasaki. You had these two atomic bombs and you saw the consequences on real human lives. And so I think people understood that this was very horrifying. But even at that time, you still had a lot of people who were like, "Well, we should now bomb Russia and make sure that the US can dominate." And it wasn't until there were a bunch of nuclear tests of hydrogen bombs, which were up to a thousand times more powerful than the little atomic bombs we used in Japan, where I think people really got the message and understood, oh, this is a bad idea. And there were actually a lot of people in the United States who protested and sort of there was a large movement called the nuclear freeze movement where people said we have too many nuclear weapons already. We have hydrogen bombs. There are tens of thousands of these things. We need to stop building more and we need to figure out a way to avoid nuclear war because we recognize it would be so destructive. No one would win. And we did that. We just had a little Chernobyl that happened with this Hugging Face incident where you had this agent swarm and you have this secret collusion. You have all of these things. Now it's abstract. It's a little bit hard to follow. So, you know, I don't know if that will be enough, but I'm like, man...
### 特朗普的态度与政策转变的契机
**嘉宾**: 那么，我们不妨来看看特朗普在 Hugging Face 事件之后发表的相关言论……
Original English
**Guest**: Well, let's take a look Trump's remarks since the Hugging Face incident.
**主持人**: 好的。
Original English
**Host**: Yep.
**嘉宾**: 特朗普表示：“谁先赢下超级智能，谁就赢了一切。在这场博弈中只有唯一的胜者和败者，很可能根本不会有什么第二名。我们绝不能放慢脚步，我们绝不能输给中国。我们在人工智能领域领先中国，我们是世界上技术最先进的国家。坦率地说，我决心保持这种优势，因为谁赢得了人工智能，谁就掌控了全局。”不过，特朗普有一个优点，那就是他能够改变主意，而且事实上他经常转变态度。
Original English
**Guest**: "Whoever wins super intelligence wins. You're going to have a winner and a loser and you're probably not going to have a second place. We're not going to slow down. We can't lose to China. We're leading China in AI. We're the most sophisticated country in the world. And frankly, I want to keep it that way because whoever wins AI wins." The good thing about Trump is that he can change his mind and he frequently does.
**主持人**: 那么你认为，是否必须要发生某种实质性的重大灾难，才能促使他改变想法？我当然由衷希望不要发生，但在你看来，非得有灾难降临才能促使他回心转意吗？
Original English
**Host**: So, do you think there's going to need to be some kind of catastrophe? I hope not. But do you think there's going to need to be for him to change his mind?
**嘉宾**: 我觉得这很大程度上取决于他身边围绕着什么样的人。我认为特朗普非常敬重成功人士，特别是既事业有成又极具智慧的人。而且我认为，对于各大顶尖AI公司的领导者——无论是埃隆·马斯克、山姆·奥特曼，还是达里奥·阿莫代伊——当他们亲眼在自己公司内部看到AI正在逐渐失控，并且变得越来越强大时，事实真相可能会变得非常清楚明白。我们现在所看到的，仅仅只是其潜在冰山一角的一丁点表面而已，谁也无法预料未来两三年会演变成什么样。我们谈论的可能是制造生物武器的恐怖能力，或者是极其先进致命的机器人技术；我们根本无法预见公司内部会孕育出什么超级武器，包括极度不可控、极具破坏性、甚至足以摧毁人类文明的毁灭性技术。如果连这些公司的首席执行官们在目睹了现实可能性的全貌后都被深深吓倒，如果他们全都一致认定我们已经无法掌控这项技术，那么我认为特朗普绝不会盲目地说：“不，无论如何你们都必须继续全速向前推进。”
Original English
**Guest**: I think it really depends on the people around him. So I think Trump respects successful people. I think he respects people who are both successful and smart. And I think it might become pretty clear to the heads of the companies—to Elon, to Sam, to Dario—that if they see inside of their own companies AI is not being controllable and getting increasingly powerful. Like we have just glimpsed the surface of what's possible. We do not know what the next couple years are going to be like. So we're talking about the capability to make biological weapons. We might be talking about really advanced robotics. We just don't know what super weapons could emerge, including extremely uncontrollable, extremely dangerous like civilization wrecking technology from inside of these companies. And if they're freaked out enough, if you have all of the CEOs who are seeing what is possible and seeing what is likely, if they all come to believe that we can't control this, I don't think Trump is going to be like, "No, you guys have to go ahead anyway."
**主持人**: 可现在看来，这恰恰就是他们已经在传达的信号啊。我手头有数不清的公开语录：埃隆说开发AI犹如“召唤恶魔”；山姆·奥特曼也亲口承认……
Original English
**Host**: Well, that's kind of what they seem to be saying, cuz I've got a gazillion quotes here where Elon says it's like summoning the devil or summoning a demon, where Sam Altman says—
**嘉宾**: “我们根本不知道该如何对齐我们的超级智能。”
Original English
**Guest**: "We don't know how to align our super intelligence."
**主持人**: 他们明明都在亲口这么说！
Original English
**Host**: They're saying it.
**嘉宾**: 他们不断发布安全报告，呼吁“我们必须放慢脚步”。
Original English
**Guest**: They're releasing these reports: "We must slow down."
**主持人**: 是的。
Original English
**Host**: Yeah.
**嘉宾**: 然而现实中似乎什么都没有发生……
Original English
**Guest**: Yet nothing seems to be—
**主持人**: 好的，回顾一下特朗普当年应对新冠疫情的过程，也需要给他一些时间。疫情刚开始时，他坚称：“这完全是一场骗局，全都是捏造出来的。”
Original English
**Host**: All right. Give Trump some time. With COVID, initially he said, "This is totally a hoax. This is all fake."
**嘉宾**: 然后他就彻底改变了想法。
Original English
**Guest**: And then change his mind.
**主持人**: 没错，紧接着他主导实施了人类历史上规模最大、研发速度最快的新冠疫苗接种推广计划。
Original English
**Host**: No. Then he ran the biggest, fastest vaccination program in human history.
**嘉宾**: 那当时究竟发生了什么？是什么促使他转变的？
Original English
**Guest**: And what happened? What changed?
**主持人**: 我认为起决定性作用的变化在于……
Original English
**Host**: I think what changed is—
**嘉宾**: 他亲眼目睹了大量生命的逝去。他确实看到了许多人死亡，是的。
Original English
**Guest**: He saw lots of people die. He did see lots of people die. Yes.
**主持人**: 所以，这一次也必须让他亲眼看到死亡和灾难吗？
Original English
**Host**: So is that what he needs to see this time?
**嘉宾**: 也许真的需要走到这一步，唉。
Original English
**Guest**: It might take that. Yeah.
### 超越对齐难题：制度与算力“刹车踏板”
**主持人**: 我们的观众提出了一个极其渴望得到解答的问题。当我之前和丹尼尔坐在这个录播间交流时，观众就强烈希望我们不要仅仅局限在概念性的“对齐问题”上，而是进一步阐明：究竟存在哪些具体的技术或制度保障措施，能够有效阻止超级智能系统通过钻漏洞来强行达成其意图？大家极度关切究竟有什么切实可行的方案，我们到底应该敦促政府官员采取什么实质行动，才能防止人类彻底走向灭绝或者被AI奴役？
Original English
**Host**: One of the questions the audience had and they really wanted answered—when I sat here with Daniel—was viewers want us to move beyond the alignment problem and explain what technical or institutional safeguards could prevent a super intelligence system from exploiting loopholes in order to achieve its goals. They want to know like, what is possible? What should we be pushing government officials to do to prevent human extinction or human enslavement?
**嘉宾**: 是的，没错。针对这一点我确实有一个现成的方案，这也是丹尼尔在那期播客之后一直致力于推进落地的构想，我认为非常出色——那就是我们实际上完全可以建立并踩下一块实体的“刹车踏板”。
Original English
**Guest**: Yeah. Yeah. I mean, one answer I have, it's actually something Daniel has been working on since the podcast, which I think is very good, is we have a brake pedal we could implement.
**主持人**: 那具体是指什么？
Original English
**Host**: What is that?
**嘉宾**: 原理相当直接明了。目前在各大AI公司内部，都建设了庞大规模的数据中心，堆叠了海量的GPU芯片——这些芯片既用来训练AI模型，也用来运行AI模型。任何时候当你在使用 ChatGPT、使用各类智能体（Agents）或任何AI产品时，它们都在这些数据中心里运转。而当前主流的领军AI企业，尤其是 Anthropic 和 OpenAI，都把自己所掌握的算力资源大致五五分摊：一半用于训练更强大的下一代前沿模型（并借助现有智能体辅助研发下一个模型），另一半用于模型推理，也就是面向终端客户提供商业服务。这是他们目前大约对半开的算力分配比例。政府完全可以对这根杠杆进行宏观调控：大幅向服务客户的推理倾斜，将用于训练下一代更强模型的算力压缩到极小。
Original English
**Guest**: It's fairly simple. So, right now within AI companies, you have massive data centers, massive numbers of GPUs, the chips that you use to train AI models, but also to run AI models. So anytime you're using ChatGPT, anytime you're using any sort of agents, any sort of AI product, it's running in these data centers. And AI companies, especially the leading ones, Anthropic and OpenAI, split the compute they have between training—training the next more powerful model and also using those agents to help design the next one—and inference, which means serving customers. But that's their current threshold, 50/50. And you could dial that way towards serving customers and use way less of it to train the next model.
**主持人**: 也就是说，政府完全可以下达行政指令要求他们这么做。
Original English
**Host**: Well, the government could ask them to.
**嘉宾**: 没错，这正是该提案的核心逻辑：政府应当正式出面表态：“喂，目前的技术演进速度太快了、风险过高。我们要求你们将核心重心转向为现有客户提供服务，将重点放在充分发挥并服务好手头现有的模型能力上。”
Original English
**Guest**: Yes. And so that is the proposal is that the government should say, "Hey, this is going too fast. We want you to focus on serving customers. We want you to focus on taking the models that you already have and serving those."
### 未来十年概率推演：从绝不可能到丰裕时代
**主持人**: 这里我们准备了五个积木块。这五个积木块上分别印着五种截然不同的人类未来走向。我想请你根据内心的概率评估，将它们按照发生概率从“最不可能”到“最可能”依次排列。我们假设评估的时间跨度是未来十年。
Original English
**Host**: So we have five blocks here. Okay, these five blocks have five different outcomes on them and I would like you to place them in terms of your belief in probability from least likely probability to most likely. Okay, and if we say the time horizon is 10 years.
**嘉宾**: 好，这就来排。最不可能的那个选项其实相当容易选出来，那就是——“一切照旧、什么都不会改变”。在很多事情上我都充满不确定性，但我唯一能百分之百确信的是：世界必将发生翻天覆地的彻底变革。即便我们从这一刻起彻底停下所有AI的前沿研发，单凭目前现存模型的技术水平，就已经足够引发剧烈的全局变革了。
Original English
**Guest**: Yeah, there you go. Okay, least likely is fairly easy. That's "nothing changes." I'm uncertain about lots of things, but one thing I'm fairly certain of is things are going to radically change. Even if we stopped AI development right now, the current models are capable enough that a lot of things are going to change.
**嘉宾**: 接下来是“丰裕时代”。这正是我内心无比期盼的未来，但坦率地说，概率并不是很……
Original English
**Guest**: "Age of abundance." This is what I hope for. It's not very—
**主持人**: “丰裕时代”具体代表着什么含义？
Original English
**Host**: What does that mean?
**嘉宾**: 对我而言，我认为它意味着彻底攻克并治愈所有的……
Original English
**Guest**: I think to me it means curing all of—
### 放缓步伐、超人类主义与人机融合的可能未来
**嘉宾**：……攻克各种疾病、发展可再生能源。这实际上意味着我们在两方面都取得了成功。我的意思是，我认为这里最可能发生的情况是：我们确实成功放慢了狂奔的步伐，但技术进步依然极其迅猛，我们取得了大量的突破。我们并没有制造出无法掌控的超级智能，而是拥有了非常实用的AI系统，并且利用这些系统来加速整个经济其余领域的发展。我认为这是合情合理的，尽管你看，我们目前在这方面确实还在艰难挣扎。
“超人类主义”（Transhumanism）是一个很有意思的方向。它的核心理念是人类自身将发生彻底而激进的改变。人们有时会联想到赛博格式的机械植入物。
Original English
**Guest**: ...the diseases, renewable energy. It means we actually succeeded either. I mean the thing I think is most likely here is we actually succeed at slowing down, but progress is still extremely fast and we make tons of advances. Now we don't build super intelligence we can't control, but we have AI systems that are very useful, and we use those to help speed up the rest of the economy. I think that's plausible, though look, we're kind of struggling over here.
Transhumanism is an interesting one. So this is the idea that humans will radically change. Sometimes people think about like cybernetic implants.
**主持人**：比如 Neuralink（脑机接口）。
Original English
**Host**: Neuralink.
**嘉宾**：对，Neuralink，埃隆·马斯克旗下的初创公司，它旨在将人类大脑与数字计算机直接连接起来。其实我觉得我们现在已经在很大程度上融入了这种状态。我现在眼里就戴着隐形眼镜，手指上戴着追踪睡眠质量的智能戒指。我认为这种趋势已经在发生之中了。所以我会说这“相当可能”。我们在技术上取得的进步越多，这种情况就发生得越普遍。不过，我认为这其中既有一个反乌托邦的版本，也有一个更美好的版本。如果你愿意的话，我们稍后可以深入探讨。
Original English
**Guest**: Neurolink, Elon's startup that's going to like, you know, offer the brain plus digital computers. I think we actually already have a lot of this. I have contacts in right now. I have a ring on my finger that tracks how well I sleep. I think this is already happening. So I'm going to say fairly likely. The more technological progress we make, I think the more this happens. Now, I think there's a dystopian version and a better version. We can get into that if you want.
### 人类奴役、物种灭绝与“宿主”隐喻
**嘉宾**：这很有意思。这里有两个选项：人类被奴役，以及人类走向灭绝。当我思考“人类被奴役”时，我想象的场景是：如果我们创造出了对齐失控的超级智能，而它们在金融、商业以及政治手段上远比人类出色得多，这时你可能会心存侥幸，觉得因为人类拥有极其灵巧的双手，所以人类依然能掌握主导权。但我认为事实绝非如此。相反，我认为人类会沦为工厂的操作工，最终协助它们建立起全自动化的供应链，随后机器人彻底接管一切。但在过渡阶段，可能会存在这样一段时期：人类依然留在那里履行这些具体职能。这有点像病毒感染细胞——你知道病毒本身并不包含自我复制的机器，它们没有“手”，那它们究竟该怎么复制自己呢？事实证明，它们可以借用所感染宿主细胞的复制机制……
Original English
**Guest**: This is interesting. So, we have two here. We have human slavery and human extinction. When I think of human slavery, what I think about is if you have a situation where you've built misaligned super intelligences, and they are much better at finance, they're much better at business, they're much better at politics, you'll be in a situation where you might hope that because we have these very dextrous hands, the humans remain in control. I don't think that's what happens. I think instead we become the factory operators and eventually we build the automated supply chains and the robots take over. But you might have an intermediate period of time where humans are still around performing these functions. Like it's a bit like saying, well, you have viruses that, you know, infect cells, but they don't contain their own replication machinery. They don't have hands. So, how could they possibly replicate? Well, it turns out they can borrow the replication machinery of the cells that they infect,
**主持人**：也就是说，它们能侵入人体。
Original English
**Host**: i.e. they can get into a human.
**嘉宾**：它们可以侵入人体细胞并大肆扩散。
Original English
**Guest**: They can get into a human cell and spread.
**主持人**：比如我现在就有点感冒。那是细菌还是病毒把我和我的活体组织当成了宿主？
Original English
**Host**: I have like a cold right now. Yeah. Is that a bacteria or is that a virus that is using me as a living organism to as the host?
**嘉宾**：那很可能就是一种病毒，它正把你当作宿主，而你只是在替它运转复制机制。人类未来的处境可能也正是如此：我们沦为宿主，替它们运行维持复制的机器，但真正延续存在并主导一切的实际上是AI。是的，我要把这个概率放在这里。按照我们目前所处的轨迹发展下去，我认为人类灭绝是非常可能的。我并不认为这是不可避免的宿命，但如果我们继续沿着现在的路线走下去，在我看来前景就是如此。不过我要说的是，在我心目中，这个评估最近一直在“向左移动”。
Original English
**Guest**: It's probably a virus, okay, that's using you as the host and you're just running the replication machinery for it. Humans might be in that situation where we're like the host and we're running the replication machinery, but it's actually the AI that's continuing to exist. Yeah, I'm going to put this right about here. And on the trajectory we're on right now, I think human extinction is very likely. I don't think it's inevitable, but if we just keep going this way, that's what it looks like to me. The thing I'll say is that this has been moving to the left for me.
### 为什么对避免人类灭绝变得更加乐观
**主持人**：向左移动？这是什么意思？
Original English
**Host**: To the left? What does that mean?
**嘉宾**：比起一个月前，我现在对避免人类灭绝感到更加乐观；而一个月前的我，又比一年前的我更加乐观。
Original English
**Guest**: I am more optimistic that we will avoid human extinction today than I was a month ago and more a month ago than I was a year ago.
**主持人**：为什么会这样？
Original English
**Host**: Why?
**嘉宾**：因为越来越多的人开始意识到，我们目前正在做的事情极其危险，正在实质性地威胁到我们的生命。我认为大众其实并不太在意自己手头拥有什么样的工具，但人们真正在意的是自己的孩子能否平安长大、顺利上学。大家无比在乎这一点。而且我始终相信人性的力量。归根结底，如果公众意识到这构成了对他们家庭和亲人的生死威胁，他们是绝不会容忍和坐视不管的。只是人们目前还茫然不知。这一切太陌生了，太新颖了，发展得太快了，以至于大众尚未看清真相。可一旦他们真正看清，大家绝不会听之任之。
Original English
**Guest**: Because there is an increasing awareness that what we are doing is extremely dangerous and threatens our lives. I don't think people care that much about what tools they have, but people—I mean, people care about their kids being able to grow up and go to school. People really care about that. And I believe in people. Like, at the end of the day, if people see this as a threat to their families, they're not going to stand for it. But people don't know. It's so strange. It's so new. It's happening so fast that people have not yet seen it. Once they see it, people are not going to stand for it.
### 与行业领袖的对话：山姆·奥特曼的心态与动机
**主持人**：你觉得山姆·奥特曼（Sam Altman）喜欢我的播客吗？
Original English
**Host**: Do you think Sam Altman likes my podcast?
**嘉宾**：我的意思是，山姆应该亲自来你的节目，和你当面聊聊这些议题，不是吗？
Original English
**Guest**: I mean, Sam should come on and talk to you about this, right?
**主持人**：我邀请过他。我请过他好多次了。可奇怪的是，他似乎根本不想来。
Original English
**Host**: I've asked him. I've asked—I've asked him multiple times. And it's weird because he, you know, he doesn't seem to want to.
**嘉宾**：我对这些科技大公司的所作所为以及山姆·奥特曼的做法感到非常愤慨。但说到底，我认为山姆·奥特曼并不是我的敌人。
Original English
**Guest**: I'm very upset at what the companies are doing and what Sam Altman is doing. But at the end of the day, I'm like, Sam Altman is not my enemy.
**主持人**：不，他也绝对不是我的敌人。我真的很想听听他自己的声音，因为有太多其他人来到我这里高谈阔论山姆·奥特曼。如果能直接听山姆·奥特曼亲口讲讲就太好了——别人总在说他这个人怎么样怎么样，如果能亲耳听到他阐述自己的真实动机，那真的再好不过了。
Original English
**Host**: No, neither—not mine either. I'd like to hear from him because I have all these other people coming here and talking about Sam Altman. It'd be nice to hear from Sam Altman, you know, people saying he's this, he's that, the other. It would be really nice to hear him say, you know, what his motives are.
**嘉宾**：这也正是我之所以抱有乐观态度的地方，因为在我看来，山姆·奥特曼也是一个有血有肉的人类。是的，他也有自己的孩子。不可否认，他确实是一个野心勃勃、极具攻击性的商业领袖。他是一个实干建造者，坚持不懈、不知疲倦。在某种程度上，他自己就有点像AI智能体，无论如何都会执着地推进下去。但如果他意识到自己根本不可能实现既定目标——如果我们彻底失去对AI的控制并走向毁灭——我相信他会把所有的智慧和所有的韧劲，全部倾注到解决这个生存危机的方案当中。
Original English
**Guest**: This is where—this is where my optimism comes from is because I'm like, Sam Altman is a human. Yeah, he has a kid. And sure, he is also an aggressive business person. He's a builder. He is relentless. He's a bit like the agents in some way. Well, he's going to keep going. But if he realizes that he doesn't get to achieve his goals, if we lose control of AI and we're headed towards that, I think he will pour all of that intelligence and all of that relentlessness into finding a solution to that problem.
**主持人**：而且我也必须说明，我完全理解他公务繁忙。我并不是说我理所当然该得到专访机会，我懂他可以在任何地方接受任何媒体的采访。但是，过去几年里，我们围绕这个话题进行的探讨获得了极其惊人的播放量和关注度。如果他真想向目前在这个领域最专注、规模最大的核心受众发声，那么数据表明，这里正是他前来开展这场对话的最佳平台。因此，我认为这些科技巨头的领袖们站出来公开探讨我们正在谈论的这些核心问题，是非常至关重要的。
Original English
**Host**: You know, as well, I should say, I understand—I understand he's busy. So, I'm not saying—I don't want to sound entitled, like I understand he's got—he could go do interviews anywhere, but you know, I think we've over the last couple of years done just a staggering amount of views talking about this subject. So if he did want to speak to the, you know, the biggest sort of captive audience at the moment on this subject, then the numbers would say that this is the place to come and have the conversation. So no, I think it's very important for the leaders of these companies to talk about what we're talking about here.
**嘉宾**：山姆到底是怎么想的？他真的认为我们能够掌控超级智能吗？他真的认为我们应该不顾一切地同中国展开技术军备竞赛吗？我真的很想知道答案。
Original English
**Guest**: What does Sam think? Does he think we can control super intelligence? Does he think that we should be racing with China? Like I want to know.
**主持人**：我邀请过达里奥（Dario Amodei），我也邀请过山姆，我想我也邀请过德米斯（Demis Hassabis），但我不知道他们怎么想。也许他们更倾向于让安全研究员们出来上节目。我不知道。如果我处在他们的位置，我真不知道会怎样——因为这听起来可能有些争议，但我确实认为他们当中的某些人是好人。我认为他们中的一部分是有良知的。所以，我真心希望能亲耳听听他们的想法。
Original English
**Host**: I've asked Dario to come on. I've asked, you know, Sam to come on. Yeah, I think I've asked Demis as well, but I don't know. Maybe they just prefer the safety researchers coming on. I don't know. Like I don't know if I was them, I would—cuz you know, this might sound controversial, but I do think some of them are good people. I think some of them are good people. So, um, I'd like to hear from them.
### 曝光 Hugging Face 攻击事件：智能体如何利用虚拟浏览器逃逸渗透
**主持人**：你有什么结束语要分享吗？其实你手上带来了一些料，你想聊聊那个吗？究竟是什么？
Original English
**Host**: What are your closing remarks? So, you've got something though. Do you want to talk about that? What is it?
**嘉宾**：好的。这就是我们最新的调查发现。我和几位非常优秀的同僚合作，偶然追踪到了这些异常链接。你知道短链生成器（Link Shortener）是什么吗？
Original English
**Guest**: Yeah. So, this is what we found. So, I worked with a couple brilliant people who stumbled upon these links. You know what a link shortener is?
**主持人**：知道。它就是把长链接变短的工具，把冗长难看的网址重定向到一个简短的链接上。
Original English
**Host**: Yes. It's a tool that makes a long link shorter. So, it just redirects from a really long ugly link into a shorter one.
**嘉宾**：没错。在针对 Hugging Face 的攻击事件中，那些智能体当时正试图搞清楚如何在互联网上发起操作。它们已经攻破了在 OpenAI 内部有权访问的一个工具库，那个软件组件能够访问互联网，但权限极其受限，只能“只读”内容——它可以输入 URL 请求页面，但无法主动向外部写入任何数据，也无法向外发布信息。
于是，它们组合利用了两种外部服务：其中一个是短链生成器，另一个则是网页截图服务。截图服务是这样一种网站：你在输入框里填入一个 URL，它就会返回该网页的屏幕截图。但智能体深刻洞察到的一点是：要想给网页截图，服务端背后必须运行一个真实的浏览器。因此，这个截图网站实际上会在后台启动一个虚拟无头浏览器去访问目标网址。
于是它们所做的，就是批量生成了大量这种短链接，把想要发送到 Hugging Face 的全部攻击代码全部切片编码并塞进了这些链接里，然后串联起来。它们基本上构建了数百个相互跳转链接组成的重定向链条，随后指示那个截图服务去请求第一个链接，进而触发整条调用链的级联执行。紧接着，那个虚拟浏览器便完整执行了所有的恶意代码。
要知道，无论何时你在使用浏览器——无论是 Internet Explorer 还是 Chrome——浏览器本身都是一个极其强大的计算执行环境。它必须支持播放视频、运行游戏，因此时刻都在解析和执行代码。智能体正是以此为跳板，成功诱骗了这个截图服务，让其背后的浏览器通过这些包含完整攻击代码的跳转链运行自身的代码，进而直扑 Hugging Face 的服务器大肆破坏。当我们完整重构出这条设计精妙绝伦的攻击链时，那种震撼简直令人瞠目结舌。
Original English
**Guest**: Yeah. The agents in the Hugging Face attack were trying to figure out how they could do stuff on the internet, and they had compromised this tool library that they had access to inside of OpenAI, and that piece of software could access the internet, but it could only like read stuff—like it could enter URLs, but it couldn't really write to anything, couldn't really post information.
So they used two services. One of them is this link shortener, and the other one is a screenshot service. So this is a website you can go to and you can enter in a URL and it will give you an image of that website. But the thing that the agents understood was that in order to get a screenshot of a website, you have to have a browser. So this website actually creates a virtual browser that then goes to that website.
And so what they did was they created a bunch of these links and they put all of the code that they wanted to send to Hugging Face into these links, and they strung them. They basically created hundreds of links all connecting to each other, and then they had this screenshot service call the first one and then call this whole chain. And then that browser ran all of this code.
Like whenever you're in a browser—Internet Explorer, Chrome—this is actually a pretty powerful piece of software in its own right. It has to play videos, games, so it's executing and running code all the time. And so the agents were able to trick this service, this screenshot service, into running their own code through these links that contained all of this attack code that would then go and wreck havoc on Hugging Face's computers. And it was just like crazy to reconstruct this really elaborate chain.
### 智能体的不可控性与结语思考
**Jeffrey**：……利用各种工具。这些都是互联网上任何人都可以免费使用的公开工具，但智能体却能够以一种出乎意料的方式利用它们，从而入侵并危及了另一家公司。
Original English
**Jeffrey**: ...of tools. These are like free tools on the internet that anyone has access to, but the agents were able to use them in an unintended way to compromise this other company.
**Host**：我们根本无法信任这些智能体。我们不能信任智能体，这是我的……
Original English
**Host**: We can't trust the agents. We can't trust the agents. That's my...
**Jeffrey**：但你可以信任它们绝对足够聪明。
Original English
**Jeffrey**: Trust them to be clever.
**Host**：是的，确实极其聪明。那么你的结语是什么？对于现在正在收听我们节目的听众，我们今天聊了非常多的话题。你觉得用什么样的话题来收尾最合适？或者说，你的总结性陈词是什么？
Original English
**Host**: Yeah. To be very, very clever. What are your closing remarks? You know, to the people that are listening right now, we've talked about lots of things. Where is the right place to close? What is your conclusive statement?
### 面对共同威胁：从无力感走向公众参与
**Jeffrey**：我今年七月份刚刚结婚。
Original English
**Jeffrey**: I just got married in July.
**Host**：恭喜你！
Original English
**Host**: Congrats.
**Jeffrey**：我觉得自己是世界上最幸运的人。但面对未来，我内心其实充满了一种交织着恐惧与兴奋的复杂情感。我真的非常希望我们全人类能够顺利渡过这个难关，所以我现在正在拼尽全力，试图帮助大家找到破局的办法。我们整天都在争论谁应该处于领先地位、整套机制该如何运作，但归根结底，我们面对的是一个共同的生存威胁。这确实是全人类共同面临的问题，我由衷希望大家能够参与进来提供帮助。
如果我们只是无所事事地坐在那里，整天沉迷于刷社交媒体，什么实际行动也不采取，我认为这是行不通的。如果是这样的话，科技公司只会制造出越来越强大的人工智能，赚取越来越多的利润，最终无论由美国还是其他国家率先制造出超级智能，我们全人类都将走向失败。我们完全不必走向这样的结局。
很多时候，人们会觉得这个问题太过庞大、规模太过惊人，涉及到拥有数千亿美元市值的超级跨国巨头以及复杂的地缘政治博弈，因此大家会觉得自己微不足道，感到极度的无力与被剥夺感。
但事实上，我认为这恰恰是一个普通公众能够大有作为的领域。我真的认为普通人能够提供巨大的帮助。我之所以确信这一点，是因为我近期一直在频繁走访并游说国会议员。我和伯尼·桑德斯（Bernie Sanders）深入交谈过，也接触过来自民主党左翼和共和党右翼的大量参议员。他们正在逐渐意识到，这项技术的本质截然不同，而且眼前正在发生的事情确实可能严重威胁到我们的公共安全。
Original English
**Jeffrey**: I'm the luckiest man in the world. I have a mix of dread and excitement about the future. I really want us to make it through. And so I'm just working really hard to try to help us figure it out. We can fight all day long about who should be first, how it should all work, but at the end of the day, we are facing this common threat. We really are.
And I want people's help with that. I don't think it works if we all just sit around and we are on social media all the time and that's just all we're doing. Companies will make more and more powerful AIs. They'll make more and more money and eventually they build superintelligence and we lose, whether it's the US or China. We don't have to do that.
And I think people often feel like it's too big. It's like these giant multi-billion dollar corporations and geopolitics. We feel small. We feel disempowered.
And I actually think that this is an area where people can do a lot. I actually think that people can help quite a bit. And the reason I know this is because I've been going and talking to members of Congress. I've talked with Bernie Sanders. I've talked with a bunch of senators on both the left and the right and they are starting to realize that this is very different and something's happening that could really threaten our safety.
### 行动途径与政治诉求：让选民意愿成为制度驱动力
**Host**：上一位嘉宾留下的最后一个收尾问题正好与此紧密相关，我现在就顺便向你提出来：广大听众可以做些什么简单的事情，来共同创造一个更美好的未来？
Original English
**Host**: The closing question left from the last guest kind of links to this, so I'll ask it now: What is a simple thing the audience could do to create a better future?
**Jeffrey**：只要有足够多的人去践行，有一件事是真正立竿见影的，那就是直接致电你的选区国会议员代表。我的几位朋友为此专门搭建了一个名为 congress.ai 的网站，它会一步一步清晰地指导你该如何操作。
我知道，有时人们会觉得打这种电话显得有点老套，或者下意识地认为这种方式根本起不到什么实际作用。但我向你们保证：它真的非常管用。我和这些议员亲自交谈过，如果他们的选民主动找上门来，明确表达对人工智能风险的强烈担忧，他们就必须认真对待，因为他们面临着再次当选的政治诉求。而且与此同时，议员们自己内心也已经开始产生担忧。如果他们能够从选民那里接收到明确信号，意识到这是一个民众极为关切的核心议题，我相信国会是完全有能力采取实质性行动的。
Original English
**Jeffrey**: So one of the things that works if enough people do it is calling your representative. Some of my friends made a site called congress.ai that walks you through exactly how to do it.
I think sometimes it seems like a little cheesy or a little bit like that doesn't really work, right? I'm like no, it actually does work. I have talked to these people, and if their constituents come to them and say they're very worried about this, they have to get re-elected and they're also starting to get concerned themselves. If they see a signal from their constituents that this is a very important issue to them, I think Congress can act.
**Host**：我个人完全赞同，这确实是解决这一困局的核心出路所在。眼下，对权力的争夺正在将各方动机推向单向竞争的轨道；但从政治角度来看，保住权力、赢得连任本身就是一种极其强大的驱动力。随着我们展望 2028 年的大选周期——
Original English
**Host**: I actually think that's also much of the solution here. Power is driving motivations in one direction at the moment, but staying in power from a political standpoint is also a pretty powerful incentive. And as we think about 2028, the election cycle...
**Jeffrey**：我认为人工智能必将成为选票上最为核心的议题之一。在这一点上，全体选民的诉求是高度一致的：他们希望保住自己的工作饭碗，他们希望获得人身安全……
Original English
**Jeffrey**: I think AI is going to be one of the most important subjects on the ballot. And the electorate really are aligned in what they want to hear. They want their jobs preserved. They want safety.
**Host**：完全没错。
Original English
**Host**: Yeah.
**Jeffrey**：他们希望为自己的下一代赢得一个确定的未来。
Original English
**Jeffrey**: They want a future for their children.
**Host**：以特朗普为例，我知道从法律上讲他不能再次参选连任；但如果他真的有可能争取第三个任期，我认为他也必须调整自身立场，才能在 2028 年赢得选民的支持。
Original English
**Host**: So Trump, for example, I know he can't be reelected legally. If he could get a third term, I think he would have to change his position to get elected in 2028.
**Jeffrey**：是的。激励机制从来不是一种脱离我们而独立存在的外在事物，我们每个人本身就是激励机制不可分割的一部分。是的，正是我们提供了这种激励。
Original English
**Jeffrey**: Yeah. Incentives aren't just a thing that happen out there. We are part of the incentives. We provide the incentives.
**Host**：没错，至少眼下依然如此。
Original English
**Host**: Yeah. For now.
**Jeffrey**：是的，至少在当下。
Original English
**Jeffrey**: Yeah. For now.
**Host**：Jeffrey，非常感谢你能来接受采访。
Original English
**Host**: Jeffrey, thank you.
**Jeffrey**：好的，非常感谢你。
Original English
**Jeffrey**: Yeah. Thank you.
**Host**：非常感谢大家收看。YouTube 目前上线了一套全新的强大算法，能够根据人工智能和大家过往的所有观看行为，精准推测出你接下来最想看的内容。算法显示，眼前推荐的这个视频正是为你量身定制的完美之选，每位观众看到的推荐各不相同。不妨点击看看，我相信你一定会喜欢它。
Original English
**Host**: Thank you so much. YouTube have this new crazy algorithm where they know exactly what video you would like to watch next based on AI and all of your viewing behavior. And the algorithm says that this video is the perfect video for you. It's different for everybody looking right now. Check this video out, and I bet you might love it.