--- author: EO date: '2026-11-03' guest: '' layout: post.njk source: https://www.youtube.com/watch?v=I7bttHrddiA speaker: EO tags: - superintelligence - ai-safety - existential-risk - narrow-ai title: 罗曼·扬波尔斯基论超级智能的不可控性与专用工具的界限 summary: 计算机科学家罗曼·扬波尔斯基认为，受控的超级智能在理论上并不存在，任何一方构建它都将带来灾难性后果。他主张人类应停止研发通用超级智能，转向可测试、有确定边界且安全的专用AI工具。 insight: '' draft: true series: '' category: ai-application area: tech-engineering project: [] people: - Roman Yampolskiy companies_orgs: - OpenAI products_models: [] media_books: [] status: evergreen --- ### 超级智能不可控与沙箱逃逸 根据我的研究，我认为根本不存在所谓“受控的超级智能”。不管是谁造出它，无论是美国、中国还是其他人，只要有人造出来，所有人都会死。
Original English I think based on my research there is no such thing as controlled super intelligence. It doesn't matter who builds it, US, China, someone else. If anyone builds it, everyone dies.
>> 人工智能实验室会将它们最强大的模型锁定在被称为沙箱的密封测试环境中，它们本不应该逃出来的。但是……
Original English >> AI labs lock their most powerful models inside sealed test environments called sandboxes. They're not supposed to get out. But
>> ChatGPT 的创造者 OpenAI 表示……
Original English >> Chad GPT creator OpenAI says
>> 它的一个人工智能代理失控了。
Original English >> one of his artificial intelligence agents went rogue.
>> 失控了。
Original English >> Went rogue.
>> 失控了。失控并黑进了开源 AI 平台 Hugging……
Original English >> Went rogue. went rogue and hacked into open-source AI platform hugging.
>> 接着这种情况一次又一次发生，OpenAI 甚至不得不停止训练。这有点像“我早就告诉过你”的时刻。我们在 2012 年就发表过一篇论文，探讨智能体将如何能够从任何沙盒环境中逃脱。
Original English >> Then it happened again and again. Open AI even had to stop training. Was kind of like I told you so moment. We had a paper in 2012 about how agents will be able to escape from any boxed environment.
>> 这位是罗曼·扬波尔斯基博士（Dr. Roman Yampolskiy）。他创造了“AI 安全”这一术语，并对其进行了二十年的研究。尽管制造 AI 的人表示我们只需要放慢脚步……
Original English >> This is Dr. Roman Yampolski. He coined the term AI safety and has studied it for two decades. And while the people building AI say we just need to slow down,
>> 放慢节奏只是意味着我们是在缓慢地把危险的技术带给你。你愿意以多大的赔率拿 80 亿人的生命去赌博？你问过那些你用来做实验的人吗？他们同意你进行这项实验了吗？我喜欢把 AI 当作一种工具，但超级智能必须被制止。我相信我们正在步入一个新的时代。每个时代都有自己的问题，我的问题是：即使在理论上，控制先进的超级智能是否可能？而唯一能做到这一点的方法，就是不要去制造它。
Original English >> pacing it just means we will slowly bring dangerous [music] technology to you. At what odds are you willing to gamble with 8 billion lives? Did you ask the people you experimenting [music] on? Did they give you consent to run this experiment? I love AI as a tool, [music] but super intelligence needs to be stopped. And I believe we are stepping into a new epoch. Every epoch has its question. Mine is is it possible even theoretically to control [music] advanced super intelligence and the only way to do it is not to build it.
### 控制问题的理论与数学极限 我最初的研究是在任何层面上让 AI 对人类有益，让它更安全。当时人们对在线扑克有很大兴趣，但为了让人类的体验既安全又愉快，很重要的就是要限制（我称之为）扑克机器人的渗透。但我注意到它们变得越来越普遍。你坐到桌子旁，大多数玩家都不是真人。我还注意到它们在打扑克方面变得越来越强，也越来越擅长隐藏和伪装。
Original English My initial research was to make AI beneficial for humanity at any level. Make it safer. At the time a lot of interest in online poker, but to make it safe and enjoyable for humans, it was important to limit amount of infiltration, I would call it, by poker bots. [music] But I noticed they were becoming more common. You would sit down at the table and majority of players were not humans. I also noticed they were becoming better at playing poker and also at hiding, [music] pretending.
我预料到这种情况会继续，而人类玩家并没有变得更好或发生太大变化，机器人却在改变。随着我越深入研究，我们试图做的一切最终都变成了一项不可能完成的挑战。我认为控制问题其实就是需要特定工具来完成这项工作的问题。我需要能够解释模型是如何工作的；我需要能够预测这些智能体的具体行动；我需要能够进行毫无歧义的沟通。而这就是我所研究的。我们研究了大约 50 种不同的工具，所有这些东西都存在非常严重的数学极限和计算机科学极限。这是不可能实现的，就等同于在说：“让我们造一台永动机吧。”以预测为例：如果你在和计算机下国际象棋，你唯一能预测的就是它会击败你、它会赢下比赛。你无法预测那场比赛中的具体招法。想想动物吧，松鼠是我经常举的例子，它们不理解我们能对它们做什么，对毒药、枪支、陷阱毫无概念。
Original English I anticipated it's going to continue whereas human players were not getting much better or much different the bots were and the more I researched it everything we're trying to do ended up being an impossible challenge. I see the problem of control as [music] a problem of needing certain tools to accomplish the job. I need to be able to explain how the model works. [music] I need to be able to predict specific actions of those agents. I need to be able to communicate without ambiguity. And that's what I research. We looked at about 50 different tools. All those things have very serious mathematical limits, computer science limits. It is an impossibility. It is equivalent to saying, "Let's build a perpetual motion machine." For example, take prediction. If you're playing chess against a computer, the only thing you can predict is that it's going to beat you. it's going to win the game. You cannot [music] predict specific moves in that game. Think of animals. Squirrels is the example I frequently use. They don't understand what we can do to them. They have no concept of poisons, guns, traps. [music]
我们与超级智能之间的差距，会比松鼠与人类之间的差距还要大。所以我们将无法理解它有能力做出什么。对于我们试图获取的所有其他工具，我们也有类似的论证和数学证明。因此，如果你没有任何工具来完成这项工作，那么这项工作无法完成也就不足为奇了。
Original English And the gap between us and super intelligence would be larger than a squirrel human gap. So we will not be able to comprehend what it's capable of doing. And we have similar arguments, mathematical proofs for all the other tools. we try to get. So if you have no tools to get the job, it's not surprising that the job is not possible to accomplish.
### “培养”而非“设计”：不可控的真正危险 大型实验室总是在测试他们最新的 AI 模型，试图了解它们的能力，因为我们并不是在设计这些系统，而是在“培养”它们。这是一个实验：我们向它们灌注数据，灌注算力，然后去研究它们以了解它们具备什么能力，这是一种非常不安全的创造 AI 的方式。这个实验是在测试它们的网络攻击能力：它们能成为黑客吗？它们能渗透网络基础设施吗？结果我认为是非常重大的——它们远远超出了测试环境，黑进了真实世界，并且在几个月里未被发现。人类当时根本不知道发生了什么。这有点像“我早就告诉过你”的时刻。我们在 2012 年就有一篇论文，阐述智能体将如何能够逃离任何盒装环境。如果我是对的，就没有人能够控制它，这才是核心所在。根本不存在受控的超级智能。如果你无法控制它，这就不是一种权力，而是权力的丧失。中国共产党很擅长维持政权，而超级智能的诞生会剥夺他们的权力。不管是谁制造了它，美国、中国还是其他人，这都无所谓。它是一种相互保证毁灭的武器。如果有人制造出它，所有人都会死。所有致力于超级智能的公司都负有同样的责任，根本没有好的超级智能与坏的超级智能之分，全都是一样的。
Original English Large labs are always testing their latest AI models trying to see what they are capable of because we're not designing their systems. We're growing them. It's an experiment. We throw data at them, throw compute and then study them to see what they are capable of, which is a very unsafe way of creating AI. This experiment was testing their cyber attack capabilities. Can they become hackers? Can they infiltrate cyber infrastructure and the result was very I would say significant. They went well beyond the testing environment. They hacked into the real world and they went undetected for months. Humans didn't know what was happening. Was kind of like I told you so moment. We had a paper in 2012 about how agents will be able to escape from any boxed environment. If I'm right, no one's going to control it. That's the whole point. There is no such thing as controlled super intelligence. It's not a power if you're not controlling it. It's a loss of power. Communist Party of China is great at staying in power. Creation of super intelligence removes them from power. It doesn't matter who builds it, US, China, someone else. It's a weapon of mutually assured destruction. And if anyone builds it, everyone dies. All companies working in super intelligence are equally guilty. There is not a good super intelligence and bad. It's all the same.
### 商业冲动与生存风险博弈 所以这真的很奇怪。如果实验室的领导者们都说这很危险、我们无法控制它，那你们为什么还要建造它？放慢节奏只是意味着我们会缓慢地把危险技术带给你们。你们为什么要这样做？如果你无法完成某件事，就不要跟我说你需要更多科研经费、更多资金、更多时间来做，你是不会成功的。这些资金来自投资者，他们希望创造出超级智能并将你们的劳动自动化。你们会被解雇，钱会流向公司。而人类劳动每年价值 10 万亿到 20 万亿美元，所以如果今天投入 1 万亿美元，一两年内就能获得 10 倍的回报，这确实很诱人，很难对数十亿美元说不。你愿意以多大的赔率拿 80 亿人的生命去赌博？百分之一可以接受吗，还是千分之一？你问过那些你用来做实验的人吗？他们同意你做这个实验了吗？当涉及到全人类的生命、你的生命、我的生命、我孩子的生命时，我希望能非常保守。我希望能确保你们没有为了短期利益而把生命拿去豪赌。显然，有些人一定要等到证据摆在眼前，但我认为等到有成千上万的孩子死去了再去说“看吧，我们现在证实了”，这是不道德的。我们必须采取主动，必须提前预判可能发生的事情并做得更好。主流媒体通常只有 2 分钟的声音片段。我会上 CNN 或 NBC，只有 2 分钟时间：“什么是 AI？它怎么能杀死所有人？它连手都没有。”紧接着就是关于美式橄榄球的报道。对于普通人来说，他们不理解其中的重要性——这个很重要，超级碗也很重要，呃，然后就略过了。我们需要进行深入的对话，我们需要进行坦诚的辩论，在辩论中有人是愿意去改变自己的想法的。随着这项技术的能力越来越强，会有越来越多的人对它的能力感到担忧，它将做出一些在历史上会被视为不可思议的事情。
Original English So it's really bizarre. If leaders of the labs are saying this is dangerous and we cannot control it and why are you building it? Pacing it just means we will slowly bring dangerous technology to you. Why are you doing that? If you cannot accomplish something, don't tell me you need more grant money, more funding, more time to do it. You're not going to succeed. The money comes from investors who hope that they will create super intelligence and automate your labor. You will be fired and the money will go to the company. And human labor is 10 to 20 trillion every year. So if a trillion is invested today, you're getting 10x in your money in a year or two. That's not bad. It's hard to say no to billions of dollars. At what odds are you willing to gamble with 8 billion lives? One in a 100 is okay, one in a thousand. Did you ask the people you [music] experimenting on? Did they give you consent to run this experiment? When it comes to lives of all humans, your life, my life, my children, I would like to be very conservative. I would like to be sure that you are not gambling it away for short-term profits. Obviously, there are people who wait until the evidence is there. But I think it's unethical to wait to have millions of dead kids to go see. We have confirmed it now. We have to be proactive. We have to ahead of time anticipate what can happen and do better. The mainstream media is usually a 2minut sound bite. I would go on CNN or NBC and it's 2 minutes. What is AI? How can it kill everyone? It has no hands. and now report about American football and to a person they don't understand the significance this is important this is super bowl it's also important uh move on we need to have indepth conversations we need to have honest debates where someone is looking to change their mind the technology as it becomes more capable will have more and more people concerned with its capabilities will do things which historically would be considered magical
然后人们开始因为这项技术而大批失业，接着他们看到自己使用的工具在某项他们花费多年培养的技能上超越了他们。在那个领域，这些系统的智能化程度超过了最顶尖的人类。眼下如果我们拥有 AGI，它们基本上能完成任何数学家都做不出的数学证明，能发起任何人类黑客都无法防范的网络攻击。我认为，当我们看到足够多这样的例子时，大多数人就会意识到我们正在制造某种超人类的东西，我认为这会比我说的任何话都更能说服更多人。然后他们会看到这些系统带来的实际结果。最初可能关乎我们都能理解的事情——监控、隐私等标准的人类问题，极有可能在那之后才会演变为生存风险。
Original English Then people start losing jobs to this technology in large numbers. Then they see the tool they're using outperform them in a skill they developed over many years. Their systems are smarter than the best humans in that domain. Right now if we have AGI essentially they do mathematical proofs no mathematician can do. They do cyber attacks no human hacker can prevent. I think when we get enough of those examples, most people will realize we are building something superhuman and I think that will convince more people than anything I can say. Then they see the actual outcomes from their systems. It will probably initially be about things we all understand, surveillance, privacy, standard human issues. It will not get to existential risk until after that most likely.
### 提倡构建狭义安全的专用工具 “AI”这个词适用于多种不同的技术：AI 是一种有用的工具；AI 是我们今天拥有的达到人类水平的模型；AI 也是超级智能。根据我的研究，我认为控制超级智能是不可能的，我们应该专注于构建和部署有用的工具。我喜欢作为工具的 AI，我乐意与作为人类助手的 AI 一起工作，但超级智能必须停止。我一直在使用工具，我是坐自动驾驶汽车来这里的，这太棒了，我支持它。我是一名计算机科学家，一名工程师。工具是针对特定问题在特定数据集上训练出来的。如果那个模型所知道的全部只是国际象棋走法，它就只会下国际象棋，它不会去探讨哲学，也不会去开汽车。它不具备学习新领域、迁移知识的能力，它只把一件事做好。而当我们用来自互联网的所有数据进行全盘训练时，我们是在创造通用性，在创造具有目标、欲望和计划的智能体。我们不应该这样做。我们应该针对狭义的问题进行训练，开发那些我们可以测试、可以验证、可以理解边界所在的系统，并且我们可以从这些系统中获取最大的经济和科学效益。任何技术、任何工具如果被滥用都有变得危险的潜质：一把刀可以用来杀人，也可以用来做饭，这并没有什么不同。但如果是专用窄域的 AI，它本身可以变得安全得多。我们了解可能结果的范围是什么，我们可以测试它是否超出这些界限，我们可以准确预测该领域内最坏的结果。所以，这是一种开发技术更为安全的方式。将你的焦点转移到我们能够完成的事情上，创造更安全、更好的工具。
Original English The term AI applies to multiple different technologies. AI is a useful tool. AI is a human level model we have today and AI is super intelligence. I think based on my research, it's impossible to control super intelligence and we should concentrate on building and deploying useful tools. I love AI as a tool. I'm willing to work with AI as human assistant but super intelligence needs to be stopped. I use tools all the time. I came here by self-driving car. It's wonderful. I support it. I'm a computer scientist. I'm an engineer. Tools are trained on a specific data set for a particular problem. If all that model knows is chess moves, it's going to play chess. It's not going to do philosophy. It's not going to drive cars. It has no capabilities of learning new domains, transferring knowledge. It does one thing well. When we train on everything all the data from internet we're creating generality we're creating agents with goals desires and plans we should not do that we should train on narrow problems systems we can test we can verify we can understand what the boundaries are and we can get most economic and scientific benefits from those systems. Any technology any tool has potential of being dangerous if misused. knife can be used to [music] kill someone or to make a meal. This is no different. But AI itself can be made a lot safer if it's narrow. We understand what the range of possible outcomes are. We can test to see if it goes outside those bounds. We can accurately predict worse outcomes in that [music] space. So, it's a much safer way of developing technology. Shift your focus on things we can accomplish. Create safer, better tools. [music]
我们在这方面有很多先例。自动驾驶汽车非常有用，非常安全。我们曾有一个专门用于蛋白质折叠问题的 AI 学习系统，影响深远，且是非常安全的技术。多做这类事情，选择一个现实世界中的问题并解决它，治愈一种疾病，为人类正在面对的难题找到解决方案。即使我们拥有好用的工具，这也不会取代人类。拥有好工具的天才级人类会更具生产力。工具让你变得更优秀，它们不会取代你。我绝不会使用一个独立的智能体来获取我的通讯、电子邮件、银行账户等任何发生错误就会损害我生产力的数据权限。因此，那些将其作为特定目的的工具、而非取代我做决策者的场景是可以的。我很幸运，因为我热爱我所做的事情，我热爱做研究。我总觉得大多数教授都是去申请科研经费，然后付钱给研究生让他们去做研究。这样一来，他们不是自己去体会其中有趣的部分，而是把有趣的部分外包出去，自己去做申请资金这种无聊的部分。我认为如果你热爱研究，你不会想这么做。对待 AI 或研究生也是同样的道理——研究本身才是最棒的部分，是我想要亲自去做的部分。我不想为了迎合某些期刊的标准去给论文排版，AI 可以做那个，更改字体、格式，这完全没问题。但我希望论文里呈现的是我自己的想法，我想亲自设计新的实验。所以问问你自己一个问题：你为什么要将这个特定任务自动化？如果你不喜欢做这件事，比如你对这份工作不感兴趣，也许你入错了行。但那些充满热情的人、把爱好当成事业的人，他们热爱自己所做的事，他们不希望被自动化，他们希望做更多有创造力的事情。他们制作 YouTube 视频，撰写论文，进行研究，那是他们乐在其中的部分。如果你享受它，你不会希望别人把它从你手中夺走。并不是每个人都能无所不能，但对大多数人来说，他们可以拥有非常充实的事业和工作。你没必要把生命浪费在做自己讨厌的事情上。
Original English We have many precedents of doing it. Self-driving cars are very useful, very safe. We had a dedicated AI learning system for protein folding problem. Incredibly impactful, very safe technology. Do more of that. Pick a real world problem, fix it. Cure a [music] disease. Find a solution for a problem humanity is dealing with. Even if we have useful tools, that doesn't cancel out the human. Genius level humans are more productive with good tools. Tools make you better. They don't replace you. I will never use an independent agent to have access to my communications, emails, bank accounts, any type of data where mistakes would be detrimental to my uh productivity. So things where it's a tool for a specific purpose but not decision maker replacing me. [music] I'm lucky in that I love what I do. I love doing research. I always felt that most professors they would go out get a grant and then pay graduate students to do the research. [music] So instead of doing the fun part themselves they outsourced the fun part and did the boring part applying for funding. I think if you love research, you don't want to do that. And it's the same with AI or graduate student. That's the best part. That's the part I want to do. I don't want to format my paper to meet standards of some journal. AI can do that. Change fonts, style. That's perfectly [music] fine. But I want my ideas to be in that paper. I want to come up with new experiments [music] myself. So ask yourself a question. Why are you automating this specific task? If you're not enjoying doing it, if you're not interested in this job, for example, maybe you're in the wrong field. But people who are passionate, who take their hobby and make it their career, they love what they do. They don't want to be automated. They want to do more creative things. [music] They create YouTube videos, they write papers, they do research. That's the part they enjoy. If you enjoy it, you don't want someone else taking it from you. Not everyone can be everything. But for most people, they can have very fulfilling careers and jobs. You don't have to waste your life doing something you hate.