---
author: AI Engineer
date: '2026-09-24'
guest: ''
layout: post.njk
source: https://www.youtube.com/watch?v=g0vqT_wZtXA
speaker: AI Engineer
tags:
- generative-media
- model-customization
- inference-pipeline
- low-latency-inference
- data-storage
title: 2026 巴黎 AI 工程师大会：从图像生成到机器人基座的深度定制与推理优化
summary: 本次大会聚焦于生成式媒体、软件工厂和图检索增强生成等前沿议题。核心内容深入探讨了基础模型（如 Flux）如何通过深度定制，从单纯的图像/视频生成拓展到直接控制机器人和电子游戏。同时，文章还剖析了推理管线，重点讨论了低延迟推理的优化策略，包括对工作负载和硬件的理解，以及如何通过系统性优化实现超大规模模型服务的性能提升。
insight: ''
draft: true
series: ''
category: ai-ml
area: tech-engineering
project: []
people: []
companies_orgs:
- Mistral
- Nvidia
- Black Forest Labs
- Backblaze
products_models:
- Flux
media_books: []
status: evergreen
---
### 开场致辞与大会第二日日程
**大会司仪（Announcer）**：女士们、先生们，请大家和我一起欢迎本次 2026 巴黎 AI 工程师大会（AI Engineer Paris 2026）的主持人——来自 Replit 的开发者关系工程师，Rou Chevrey 登台！
Original English
**Announcer**: Ladies and gentlemen, please join me in welcoming to the stage your MC for the AI engineer Paris 2026, developer relations engineer at Replit, Rou Chevrey.
**Rou Chevrey**：大家早上好！麦克风试音。各位早上好，大家今天感觉怎么样？嘿，这听起来完全像是第二天的精神状态啊，我们得把气氛燥起来！好的，大家今天状态如何？
Original English
**Rou Chevrey**: Good morning. Mic check. Good morning everybody. How's it going? Hey, this feels like day two energy, so we need to bring it up. Okay. How's it going everybody?
**现场观众（Audience）**：好——！
Original English
**Audience**: Yeah.
**Rou Chevrey**：太棒了，很好，很好！欢迎大家来到巴黎 AI 工程师大会的第二天。好的，昨天有谁在场？请举一下手。好的，我看绝大多数人昨天都在，太赞了！昨天我们的核心议题是探讨技术的演进逻辑，思考当今 AI 的现状，随后我们探讨了 AI 的未来——没错，包括 AI 软件工厂（software factories）的未来，以及前沿模型（frontier models）的发展趋势。但我对今天的日程尤其感到兴奋：今天我们将重点探讨生成式媒体（generative media），再次深入讨论软件工厂，我们还将探讨图检索增强生成（Graph RAGs），此外还有极其丰富的其他精彩内容。
Original English
**Rou Chevrey**: Nice, nice, nice. Well, welcome to AI Engineer Paris day two. All righty. So, who was with us yesterday? Show hands. Okay, most of you guys. That's awesome. So yesterday was all about how to think about the evolution of technology to think about AI today and then we talked about the future of AI right the future of AI um software factories um and also uh the the future of frontier models but I'm particularly excited about today today we're going to talk about generative media about software factories again we're going to talk about uh graph rags and we're and we're going to cover so much more so
**Rou Chevrey**：因此，我希望大家认真思考今天最想了解的内容。请务必全神贯注地倾听台上分享的所有干货。我们的演讲嘉宾们全天都会在会场交流，所以请大家务必抽出时间去展区逛逛，多和大家交流沟通，尽可能多地吸收知识。好的！如果你们昨天在场，想必已经清楚接下来的流程了，对吧？如果没有 Mistral，这场盛会根本无法成真，所以我想请大家为他们献上热烈的掌声！请大家为 Mistral 欢呼喝彩，感谢他们精心组织了这次大会！同时，让我们把掌声献给赞助商们，尤其是我们的白金赞助商——英伟达（Nvidia）！是的，掌声不要停！接着，让我们再次用热烈的掌声感谢我们所有的金牌赞助商、银牌赞助商、铜牌赞助商以及所有支持型赞助商！非常感谢大家！好的。
Original English
**Rou Chevrey**: So with that, I want you to uh think about what you want to see. Please pay attention to everything that is going to be said here. Our speakers are going to be hanging out. So also take the time and go to the expo and talk to everybody to learn as much as you can. All right. So you know the drill by now if you were here, right? This event wouldn't be possible without Mistral. So I would like to give give it up for them. So please let's give it up for Mistral. Thanks for organizing and let's give it up for our sponsors, our platinum sponsor, Nvidia. Yes, please. Yeah, keep it going. So, to for all our uh gold sponsors, silver sponsors, bronze sponsors, and also our supporting sponsors. Let's give it up one more time. Yeah, thank you. Okay.
**Rou Chevrey**：那么话不多说，我们的下一位演讲嘉宾来自 Black Forest Labs。他有个独特的习惯——特别喜欢去看牙医，我也不知道为什么，也许大家待会儿可以亲自问问他。不过他今天将为我们分享他们是如何从图像生成一步步拓展到机器人领域的。让我们掌声欢迎 Jakob Pörschmann 登台！
Original English
**Rou Chevrey**: All right. So, without further ado, our next speaker is from Black Forest Labs, and he's got some tendency. He loves going to the dentist. I don't know why. Maybe we can ask him later. But he's going to talk to us about um about how they went from image generation to robotics. So, please join me in welcoming to the stage Jakob Pörschmann.
### 从图像生成到机器人基座：Flux 的能力拓展
**Jakob Pörschmann**：非常感谢大家！非常高兴在这里与各位相聚。可能需要先帮我把幻灯片切出来。太棒了！大家好，很高兴见到各位。我的名字叫 Jakob，我在 Black Forest Labs 负责领导 AI 解决方案团队——也就是我们公司的前向部署工程团队（forward-deployed engineering team）与解决方案工程团队。在开始之前，我想先抛出一个应该算非常简单的小问题：请大家简短举手表决一下，在场有谁以前使用 AI 生成过图像或视频？请大家快速举一下手。这应该是个送分题吧，来吧，希望在场的每一个人几乎都举手了。
Original English
**Jakob Pörschmann**: Thank you very much. Super nice to meet you everyone. Just need my slides maybe in here. Awesome. Nice to meet you everyone. My name is Jakob. I lead the AI solutions team. Um that's the forward deployed engineering team and solutions engineering team at Black Forest Labs. And I would like to start in with a hopefully trivial question. So give me a brief show of hands of who has generated an image or a video before using AI. Just give me a brief show of hands. This should be trivial. Come on. This is almost everyone hopefully.
**Jakob Pörschmann**：那么，既然大家以前都生成过图像或视频，我希望你们在此之前多多少少都接触过 Flux 或者我们 Black Forest Labs。因为在 Black Forest Labs，我们是一家总部位于德国的研究型实验室，致力于为图像与视频生成构建基础前沿模型。大家在我身后的屏幕上可以看到一些样例视频和图像，这些都是通过我们最新的模型 Flux 3 生成出来的。
Original English
**Jakob Pörschmann**: Um and um then hopefully if you generate an image or video before um you hopefully were in contact with Flux uh or Black Forest Labs before because Black Forest Labs at Black Forest Labs we are a research lab based in Germany and we build foundational image and video models for image generation. Um and you can see some of the samples um videos and images that were generated with our recent model flux 3 in the back uh back here.
**Jakob Pörschmann**：现在我要问大家的第二个问题是：如果我告诉你们，大家可以使用同一个视频生成模型来控制机器人以及打电子游戏，在座的有谁会相信我？好的，举手的只有寥寥几个人，显然并没有很多人相信。嗯，这显然是一个修辞性的设问，否则我今天也不会站在这里专门讲这个话题了。因为这正是我今天演讲最后要收尾的核心亮点：我们究竟如何对 Flux 进行定制，使其不仅能够生成媒介内容、生成图像、生成视频并对二者进行编辑，而且能够用来直接操控视频游戏和操纵实体机器人。
Original English
**Jakob Pörschmann**: Now the second question that I have for you is who of you would believe me um if I would tell you that you can use that same video model to also control robots and play video games. Okay, that's a few people, not really everyone. Um well, it's kind of a it's kind of a rhetorical question obviously. um otherwise I wouldn't talk about it because that's exactly the final thing that I'm going to close with today is how we're going to customize Flux um to not just generate media, generate images, generate videos and and edit um edit both um but we're going to talk about how to steer video games and steer robots with it.
**Jakob Pörschmann**：大家在这里可以看到两个不同的样例演示，左侧的内容也是由 Flux 生成的。在左边，Flux 实际上正在实时玩一款电子游戏——这不仅仅是生成了一段游戏画面的录像，而是 Flux 真正接管了游戏控制权，并实时生成当前发生的每一帧画面。在右边，则是一个 LeRobot 正在执行抓取与放置（pick and place）任务，它正是将我们的 Flux 视频模型（Flux 3）作为其智能决策骨干网络（intelligence backbone）。
Original English
**Jakob Pörschmann**: So you can see two different samples here that were also on the left generated with Flux. On the left you have Flux actually playing a video game. Um so this is not just a video of a generated but it's actually flux taking the control and and generating the frames that are happening. And on the right you have a LeRobot that is completing pick and place tasks um with Flux the video our video model flux 3 as a intelligence backbone
**Jakob Pörschmann**：那么，在纯粹的视频生成与将 Flux 作为机器人的智能基座之间，究竟存在着什么桥梁？这恰恰就是我们今天所要探讨的核心：连接并弥合这两者之间鸿沟的，正是对 Flux 模型的深度定制（customization）。我们拥有一个极其强大的基础模型，我们在海量数据上对它进行了预训练（pre-trained）和中程训练（mid-trained），使其从本质上理解视觉世界，深刻理解视觉世界中各类物体如何运动和交互。而我们所做的，就是获取这个强大的模型，并针对特定的具体业务场景对其进行定制。这正是我们 AI 解决方案团队日常的核心工作。
Original English
**Jakob Pörschmann**: and what is in between video generation and using flux as a backbone for a robot that's exactly what we're going to talk about today because the part that is in between that's basically bridging that gap is customization of this flux model right there is a super powerful model that we pre-trained trained mid-trained on essentially understanding the visual world having having an understanding for um how objects are behaving in the visual world and we are taking this model and customizing it for certain use cases and that's exactly what we do in our AI solutions team as well
### 推理管线解剖与六大定制维度
**Jakob Pörschmann**：接下来我想分几个不同阶段向大家剖析这一过程。在深入探讨机器人之前——那将是我们今天演讲的最终压轴环节——我想先带大家剖析一下推理管线（inference pipeline）的内部结构。因为通过梳理推理管线，能够让大家建立一个全局视角，全面了解当我们与 Flux 打交道、以及与希望在特定垂直场景下落地 Flux 的企业级客户沟通时，我们所拥有的各种定制化切入维度（customization vectors）。总体而言，整条管线大致可以拆解为六个步骤。
Original English
**Jakob Pörschmann**: and this is what I want to talk to you about um in a couple different steps to break this down a little bit before we go into robots that's going to be the final stage of what we're going to talk about um I want to talk a little bit about the anatomy of the inference pipeline because this is also going to give you an overview of sort of the customization vectors that we have when we walk work with flux and we talk to enterprise customers that want to use flux for very specific use cases. Um this breaks down in approximately six steps.
**Jakob Pörschmann**：第一步，自然是可以优化的用户输入（user input）。这通常包括文本提示词（text prompt）、图像提示词（image prompt）或视频提示词（video prompt），逻辑非常直接明了。第二步是提示词升采样（prompt upsampling）。提示词升采样本质上是指我们在用户发送给我们的输入提示词与底层的 Flux 模型之间，接入一个大语言模型（LLM）或视觉大模型（VLM）。我们通过它来深度理解用户的真实意图，洞察用户到底想要达成什么效果，并尝试对提示词进行扩展和润色，从而使模型生成的最终结果能够无限逼近用户心中的预期。
Original English
**Jakob Pörschmann**: First of course there is a user input uh that can be optimized. Um this is usually a text prompt, an image prompt, a video prompt, fairly straightforward, right? Um second step is the prompt upsampling. Prompt upsampling essentially means that we have an LLM or a VLM that sits between the user prompt that is sent to us and the flux model. And we essentially try to understand the intent. We try to understand what the user actually wants to do and we try to improve the prompt that the model generates something that's closer to what the user is actually intending to do.
**Jakob Pörschmann**：第三步是内容审核与安全治理（moderation）。当谈到模型的定制化时，大家可能会惊讶为什么会把审核机制单列出来。事实上，内容审核是至关重要的核心产品层面（product surface）之一。因为审核机制直接决定了用户最终能看到什么、不能看到什么。在多数情况下，审核并不直接内嵌在模型内部，但它构成了用户整体体验不可或缺的关键环节。我们稍后会专门讨论如何定制审核机制，甚至如何将安全审核原则深度嵌入到模型自身的权重之中。
Original English
**Jakob Pörschmann**: Then there's moderation. And you might be surprised in seeing moderation when it when it comes to when it comes to customization of a model. Well, actually moderation is one of the most important product surfaces. Um because moderation essentially decides what you're going to see or what you're not going to see. It's not part of mostly not part of the model. Um but it is part of the user experience and we're going to talk a little bit about how we customize it and also how we actually embed moderation into the model weights.
**Jakob Pörschmann**：第四步当然就是模型本身了。也就是模型的权重层面，我们可以进行微调（fine-tune），比如训练低秩适配器（LoRA）。我们可以针对特定的艺术风格、特定的人物角色、特定的商业产品进行微调定制；我们也可以针对特定的行为表现进行微调；甚至通过微调，我们还能为模型注入全新的模态能力——这正是我们在 Flux 3 Action 上所做到的突破，我会在演讲结尾详细展开介绍。第五步是后处理阶段（post-processing）。这一步包含的内容相对灵活多样，例如超分辨率放大（upscaling）、图像修复与画质还原（restoration），或者是二次深度审核等等，这里蕴藏着无穷无尽的定制空间。
Original English
**Jakob Pörschmann**: Fourth, there's of course the model. There are the model weights which we can fine-tune. We can run LoRAs on it. We can fine-tune for style, for characters, for products. And we can fine-tune also for behavior and using fine-tunes, we can also add whole new modalities. And that's exactly what we did with flux flux 3 action, which I'm going to talk about in the end of my presentation. Then there's a post-processing step. Of course, this is kind of arbitrary. There can be upscaling, there can be restoration, there can be further moderation, right? Um, endless customization opportunities here.
**Jakob Pörschmann**：最后，我们在模型实际的服务化部署层面也投入了大量精力——包括如何高并发地托管模型、如何优化推理引擎、如何进行知识蒸馏（distillation）等等。在接下来的二十分钟左右时间里，我想系统性地带大家快速翻阅一下前向部署工程师的工具箱，至少先初步揭开我们在日常工作中是如何对 Flux 进行全方位深度定制的面纱。
Original English
**Jakob Pörschmann**: And then finally there is um we also work a lot about how we actually serve the model, how we deploy it, how we optimize it, how we distill it um etc etc. And what I want to do in the next 20-ish minutes is I want to go through essentially the forward deployed engineers toolbox at least scratch the surface a little bit um of how we customize flux um in our day-to-day.
### 深入提示词升采样机制
**Jakob Pörschmann**：首先，从第一个重头定制维度开始，我想把焦点放在提示词升采样（prompt upsampling）这一步骤上。正如我刚才所提到的，提示词升采样意味着在终端用户和图像模型之间架设了一个视觉语言模型（VLM）或大语言模型（LLM）。在实际工程实践中，它的运作形态大致是这样的：在幻灯片左侧，大家可以看到用户输入的初始提示词。这个输入可能极其朴素、直接，比如仅仅只有“森林中的小木屋”（a cabin in the forest）。如果你直接把这样一句极简的提示词径直发送给 Flux 模型并生成图像，你就会得到类似左图所示的效果：画面中就是一栋朴实平淡的林中小木屋，任务确实完成了。
Original English
**Jakob Pörschmann**: So starting uh with the first major customization I want to focus on this prompt upsampling step. As I already mentioned prompt upsampling means that there is a VLM or an LLM that sits between the user and the model. Um that looks in practice something like this. Um so on the left here you have a user prompt. So this might be super super straightforward. a cabin in the forest. And if you send that straight to the flux model, generate an image, you'll see something like the image on the left. It's a plain um cabin in the forest, right? The mission completed.
**Jakob Pörschmann**：然而，这样的生成结果可能并没有针对用户的真实意图进行充分优化，也可能未曾契合用户的审美偏好。当然，这在很大程度上取决于用户在这里到底想要得到什么。在这个具体案例中，虽然原始提示词非常简单，但它却非常出色地展现了提示词升采样的价值所在。因此在本质上，我们接收了“森林中的小木屋”这一简短输入，并对该提示词进行了实质性的扩展丰富。在这种情况下，默认的升采样器会专门针对用户的主流偏好进行优化，它将用户原始的“森林中的小木屋”提示词大幅扩展为类似于“超写实、直角结构的石质……”这样详尽的描述。
Original English
**Jakob Pörschmann**: However, it might not be optimized for user intent. It might not be optimized for user preferences. Um it highly depends, of course, what the user wants here. In this case, the prompt is fairly simple. Um but it demonstrates the purpose quite well. So essentially we take the cabin in the forest and we essentially extend the prompt. In this case the default upsampler just optimizes for user preference. Extends the prompt the user the cabin in the forest to something like a hyperrealistic right angle stone
### Prompt Upsampling：弥合用户意图与设计规范
**讲者**: ……烟囱、柔和的阳光等细节。然后大家可以在右侧看到最终的结果。至于大家是更喜欢左边还是右边的效果，当然见仁见智、可以讨论。但这里的核心要点在于，Prompt Upsampling（提示词上采样）步骤为我们提供了一个极其强大的工具，能够真正地将用户的意图和生成方向引导至任何可能的方向。
Original English
**Speaker**: ...chimney, soft, warm golden light, etc., etc. And then you see the final result on the right. Whether you like the left or the right better, that's up for discussion, of course. But the point being here is that the prompt upsampling step gives us an incredibly powerful tool to actually steer the user intent and the generation that we're taking into any direction possible.
### 客户实战案例：结构化插图生成
**讲者**: 接下来让我们来看一个具体的案例。这是一个真实的客户案例。这位客户找到我们说：“我们实际上希望在应用程序中，用 Flux 生成的图像来为用户正在进行的特定对话配上插图。”因此，他们希望为各种主题创建插图。这些主题完全是随机的，而且全是由机器生成的。换句话说，它们并非经过人工策划，也不是现成的图像提示词。比如，主题可能是“适合家庭亲子游的旅行目的地”，或者“排空洗碗机里的积水”。完全是随机的主题，不是图像提示词的形式，完全由机器生成，而且涵盖的种类范围极其广泛。
Original English
**Speaker**: So let's take a look at a concrete case. This specific case was a real customer case. The customer essentially came to us and told us, "Well, actually we want to illustrate certain conversations that users are having in our application with Flux images." So they want to create illustrations for topics. These topics could be completely random and they're machine generated. So they're not hand-curated and they're also not image prompts. So for example, this could be "family-friendly travel destinations" or "draining water for my dishwasher." Right? Completely random topics, not in the form of an image prompt, machine generated. This could be a vast variety of topics.
**讲者**: 此时我们的任务就是要生成插图，以某种方式把这些主题漂亮地表现出来。但这其中的难点在于，我们当然必须满足某些特定的设计要求。例如，在这个特定案例中，主体必须位于画面右侧；整张图片必须非常干净利落；画面的左侧必须留出大量空白空间以供文字叠加；此外还适用一些具体的设计规则，比如不能出现人物、不能出现地标建筑，以及其他一些限制。
Original English
**Speaker**: Now our task is to generate an illustration, somehow illustrate this nicely. But the trick to this is that, of course, there are certain design requirements that we need to meet. So for example, in this specific case, the subject is supposed to be on the right. It's supposed to be a very clean image. There's supposed to be a lot of space on the left for a text overlay. And there are certain design rules that also apply. For example, no people involved, no landmarks, and a couple others.
**讲者**: 因此，我们实际上所做的，或者说归结到底的关键在于：图像模型本身实际上已经不再是瓶颈，因为在技术层面上，模型生成所有这些不同的内容都毫无问题。真正构成瓶颈的，实际上是这些设计规范。而 Prompt Upsampling 所发挥的作用，就是将用户的原始提示与设计规范相互匹配并搭建桥梁，同时将这些设计规范精准匹配到 Flux 的训练数据分布中。换句话说，就是从训练分布中找到我们需要发送给模型的那个最精准的锚点，从而生成真正遵循设计规范的精准图像。
Original English
**Speaker**: So what we essentially do, or what this actually comes down to, is that the image model is actually not the bottleneck anymore, because the model technically can generate all of these different things, right? No problem. But really the design requirements end up being the bottleneck. And where prompt upsampling comes in is matching and bridging the user prompt with the design requirements, and matching these design requirements to the training distribution of Flux. So finding the exact right spot that we need to send to the model from the training distribution to generate the exact image that actually follows the design requirements.
### 对比验证：Raw Prompt、默认 Upsampler 与定制化 Upsampler
**讲者**: 接下来我用大家刚才看到的两个主题举例说明。首先是“适合家庭亲子游的旅行目的地”。如果你直接把这个原始提示词发送给 Flux，完全不做上采样，你看到的将是完全不知所云、纯粹垃圾级别的图像。比如出现类似明信片风格的东西。而且如果在完全不进行上采样的情况下仅仅切换不同的随机种子，不同图像之间的方差差异会极其巨大。如果我们使用默认的 Upsampler——它纯粹是针对普通用户偏好优化的——那么你看到的效果就会像中间的图一样，可能稍微更接近插图一点，但完全不符合设计要求。然而，如果我们对上采样步骤进行深度定制，直接将设计规范嵌入到 Prompt Upsampling 中，你就会得到右侧的效果：主体位于右侧，画面干净且富有光泽感，左侧留有大片空间供文字排版叠加，完美契合了设计规范。
Original English
**Speaker**: So to give you two examples from the topics that you already saw. First, "family-friendly travel destinations." If you send that as a raw prompt to Flux, completely un-upsampled, you're going to see complete gibberish, complete trash images essentially. For example, there's this postcard style. The variety between different seeds also is extremely high when you just send this completely un-upsampled. If we use our default upsampler, which is just optimized for general user preference, then you'll see something like in the middle, probably a little bit closer to somewhat of an illustration, but not at all matching the design requirements. But if we actually customize the upsampling step, and we embed the design requirements directly in the prompt upsampling, you'll see something on the right. So the subject is on the right. It's a very clean image, very sort of glossy. There's a lot of space on the left for text overlay. So meeting exactly the design requirements.
**讲者**: 第二个例子也是完全同理。对于“排空洗碗机里的积水”，在没有 Prompt Upsampling 的情况下，生成出来的完全是随机混乱的图片；稍微好一点的情况下可能会生成类似说明书视频截图一样的画面，但这在第二种情况下依然根本不是我们想要的。但如果我们定制这个 Upsampler——它本质上是一个负责理解用户意图、熟知设计规范并努力将其映射到训练分布中的系统提示词——我们就能获得一张非常漂亮、极其干净的图像：主体明确置于右侧，左侧留足大面积空间，完全支持文字覆盖。
Original English
**Speaker**: Same goes for the second example. So "draining water in a dishwasher" without prompt upsampling is a completely random image. Getting a little bit better, some sort of instruction video, but still not at all what we want in the second case. But if we actually customize this upsampler—which is essentially the system prompt that looks at the intent and tries to match that into the training distribution knowing the design requirements—we'll get a very nice, very clean image: subject on the right, a lot of space on the left, text overlay possible.
### 系统提示词迭代的灵活性与工程优势
**讲者**: 这本质上是一个非常简单但极为巧妙的技巧。我们从中所学到的最大收获，以及这种做法带来的巨大优势，就在于它的迭代速度极快。你本质上只需要对系统提示词（System Prompt）进行微调迭代，完全不需要进行模型训练，同时还能获得最大程度的业务场景灵活性。引入这一步骤也是非常必要的，因为我们在与企业级客户沟通时深刻体会到：用户本质上是“懒惰”的。用户没有必要也不想去深入了解我们的训练数据分布究竟是什么样的，或者究竟该如何写出完美适配 Flux 的提示词。因此在很多场景下，都是由我们介入，通过定制 Prompt Upsampling，精确匹配用户意图、设计规范与模型训练分布，从而在这里找到最完美的黄金平衡点。是的，Prompt Upsampling 表面看起来极其简单，但它却是一个不可思议的强大工具，正因为它迭代起来如此敏捷，且不需要任何模型训练成本。
Original English
**Speaker**: And this is essentially a pretty simple trick. The massive advantage and what we essentially learned from this is that it's super quick to iterate, right? You essentially just iterate over a system prompt. There's no training needed, and you have maximum use case flexibility. And this is sort of necessary as well to include this step because what we learned, especially when talking with enterprise customers, is that users are just lazy. Users don't necessarily want to know exactly how our training distribution looks like, right, or how to perfectly prompt Flux. That's in a lot of cases where we come in and we customize prompt upsampling to match exactly user intent, design requirements, and training distribution to find exactly the right sweet spot here. Yeah, prompt upsampling seems super simple, but it's an incredibly powerful tool because it is so simple to iterate over and it doesn't need any training.
### 审核与安全：核心定制化产品维度
**讲者**: 接下来我们进入下一个主题：内容审核与安全（Moderation and Safety）。大家看到审核出现在这里的定制化功能层级中可能会再次感到有些意外，但实际上，内容审核是我们拥有的最关键的产品交互界面之一，尤其是在通过 API 对外提供模型服务的时候。因为本质上，审核机制决定了用户最终能看到什么、不能看到什么，以及我们允许哪些提示词输入、拦截哪些提示词。因此，这是我们所提供的整个用户体验中极其重要的一环。
Original English
**Speaker**: Moving on to the next stop is moderation and safety. And you might be again a little bit surprised to see this in the customization surface here, but moderation is actually one of the most important product surfaces that we have, especially when offering a model through the API. Because essentially moderation decides on what the user sees and what not, which prompts we allow and which not. So this is an incredibly important piece of the user experience that we offer.
**讲者**: 如果大家过去尝试过业界某些顶尖的视频生成模型，可能对这类错误提示非常熟悉：你上传了一张自己的照片并希望让它动起来，但系统却弹出了“图像或视频中可能检测到真实人物肖像”之类的警告信息。这显然不是模型的生成能力受限，而纯粹是审核服务在起作用，是它阻止了你的生成。当我们面对那些有非常特定业务需求的大型客户时，这就会成为一个巨大的痛点，我们必须对其进行定制并解决。
Original English
**Speaker**: If you have tried some state-of-the-art video models in the past, you might be very aware of error messages like this: you put in an image of yourself and you want to animate yourself, but you're getting something like "the images or videos might find likenesses of real people," etc. This is obviously not a model limitation. This is purely the moderation service, right, that doesn't allow you to generate something. And when talking to larger customers that want to do very specific things, this is a massive issue and we need to customize and solve for this.
### 基于配置文件的细粒度审核策略
**讲者**: 关于内容审核，我想探讨两种实现路径。第一种是最直接、最简单的方式，本质上是在配置文件（Configs）中进行管理。我们构建了一套模块化的架构来定制配置文件。这里有一个相当直接的安全参数，允许客户直接通过 API 进行设置。在为特定客户定制审核规则时，我们拥有非常细粒度的配置架构，不仅能让我们设置安全容忍度，还能针对特定类别分别进行放行或拦截，例如：暴力、色情提示、自残、知识产权与肖像权侵权等。与此同时，我们还将这一机制与针对每个客户专门设定的黑名单（Blocklists）和白名单（Allowlists）相结合。
Original English
**Speaker**: So two ways that I would like to talk about moderation. First is the most simple way, and it is essentially in the configs. We have a modular setup to essentially customize configs. There is a fairly straightforward safety parameter here that allows you to set that via API. And when customizing moderation for specific customers, we have a very fine-grained config setup that essentially allows us to set safety tolerance, but also allow or block certain categories specifically: so violence, sexual prompts, self-harm, IP and likeness. And we also combine that with certain blocklists and allowlists that we can set specifically for every customer.
**讲者**: 举例来说，儿童平台的要求显然会严苛得多，必须严格拦截暴力、色情、自残、IP 与肖像侵权内容，甚至会配置特定的敏感词黑名单；而时尚零售商则可能会更开放一些，或许对带有一些性感倾向的提示词更具包容度，但他们会明确封杀竞争对手的品牌，同时将自身品牌组合列入白名单，自然只希望用户生成他们自己的商品；此外，媒体或电影制片厂的诉求又会灵活得多，他们甚至允许适度的暴力内容，并且特别希望能生成已获授权的特定影视角色（即使这些角色受版权保护），同时封杀所有未获授权的角色。这种方式非常直接：我们只需通过配置文件进行配置，架构高度模块化，简单明了。
Original English
**Speaker**: So think about, for example, a kids' platform being much more strict, blocking violent, sexual, self-harm, IP likeness, obviously, and then having even a specific blocklist. A fashion retailer being a little bit more open, maybe allowing a little bit more of sexually leaning prompts, but blocking specifically competitor brands and allowing their own portfolio, essentially only trying to generate their own stuff, of course, right? And then there might be a media or a movie studio that is much more flexible, allows even some violence, and specifically wants to allow certain licensed characters even if they're IP-protected, and wants to block every other character that they don't have a license for. So this is fairly straightforward. We just set the config: super modular, super simple.
### 本地私有化部署与 Slider LoRA 权重级安全防护
**讲者**: 然而，这种配置方式并不能解决所有场景的问题，因为我们也会直接把模型部署在客户的私有基础设施上。在这些本地私有化部署的场景中，我们无法将整套外部审核技术栈一同交付过去。我们必须找到某种方法，真正将审核与安全机制直接内嵌到模型本身的权重之中。而这正是 Slider LoRA 大显身手的地方，它是我们现场技术工程师（FTEs）工具箱中的又一利器。Slider LoRA 致力于学习某种抽象概念——或者说精准学习我们试图拦截的特定抽象概念，然后通过反转其学到的权重方向，来实现对该概念的精准抑制与拦截。
Original English
**Speaker**: However, this doesn't work for every case because we also deploy our models directly on customers' infrastructure. So in these cases, we don't ship the entire moderation stack. We somehow need to find a way to actually embed the moderation and the safety into the model weights. And this is where, for example, a Slider LoRA comes in, which is the next sort of tool in the toolbox of our FTEs. A Slider LoRA tries to learn an abstract concept, or tries to learn the specific abstract concept that we're trying to block, and tries to then turn around the weights that it learned to block that specifically.
**讲者**: 让我们以“去色情化”（Desexualization）Slider LoRA 为例，幻灯片上展示了其简化原理。本质上，你需要微调模型，也就是微调 LoRA 适应矩阵，尝试将权重调整到朝向纯粹“色情化”概念的方向。我们实现这一目标的方法是使用一个通用数据集……
Original English
**Speaker**: So let's take the example of a desexualization Slider LoRA, which you have essentially simplified here on the slide. Essentially you try to tweak the model, tweak the LoRA adaptation matrix—you try to tweak that, the weights into the direction of the pure concept of sexualization. How we do this is that you have a generic dataset...
### 通过权重调优与滑块 LoRA 实现模型内置审核
**Speaker**: 比如针对某些图像，例如以某张身穿黑裙的女性照片为例，你把它输入到训练循环中。每个训练步骤都会尝试两次：一次搭配端庄朴素的提示词，另一次搭配性感化的提示词。这样一来，你本质上就是在训练模型去识别：面对同一张图像，分别用朴素和性感两种描述去拟合，最后在权重中沉淀下来的，实际上就是两次学习之间的增量差值（Delta）。而这两个学习结果之间的差值在于，两边的提示词其实都包含了“一名女性、黑发、黑裙”等基础信息，但它们之间的差值，本质上正是“性感化”这一概念，因为性感描述包含了这层含义，而朴素描述则没有，后者要纯粹写实得多。因此，你最终会得到一个 LoRA 矩阵，也就是一个刻画了“性感化”概念的适配矩阵。
Original English
**Speaker**: ...of images, for example of people. In this case, one image of a woman in a black dress, and you feed that to every training loop: tries once with a modest caption and once with a sexualized caption. So then you essentially train the model. This is the image, and you see it tries once modest and once sexualized, and what is left in the weights is essentially the delta between both learnings. And the delta between both learnings is—both captions essentially contain "there's a woman, black hair, black dress," whatever. But the delta between those is essentially the concept of sexualization, because the sexualized caption contains that, but the modest caption doesn't; it's much more descriptive. So you end up with a LoRA matrix, with an adaptation matrix that essentially describes the concept of sexualization, right?
**Speaker**: 在推理阶段，我们就可以直接将这个矩阵以负的 alpha 权重应用到模型权重中。这样我们实际上把这组权重完全反转，让它朝完全相反的方向运作。在这个具体的测试案例中，这种处理带来了非常显著的效果：裸露检出率下降了 20 次，并且有 68% 的评价认为图像画面的性感程度明显降低；与此同时，整体图像质量评测完全持平，因为我们字面上仅仅反转了“性感化”这一个特定概念。所以这是一个非常巧妙的微调手法，能够真正把内容审核与安全偏好直接烘焙（bake in）进模型权重当中。这也是可以直接交付给客户的方案，他们可以将其部署在自己的私有基础设施上，即使外层没有任何确定性的过滤审核系统，也能获得一个安全得多的模型。
Original English
**Speaker**: And in inference, we can now go ahead and apply that to our model weights with a negative alpha. So we turn the weights exactly around to go in exactly the opposite direction, which then turned out in this specific case for minus 20 nudity detections and 68% judged less sexualized image quality, while at the same time the image quality generally is a complete tie, because we literally only turned around the concept of sexualization. So this is a nice tweak to really bake in moderation into the weights, and this is something that you can also ship directly to customers. They can deploy it on their own infrastructure to have a much safer model without any deterministic moderation around it.
**Speaker**: 从内容审核这部分我们可以总结出几点经验：首先，安全合规是一个极其重要的产品界面。从某种意义上说，它是用户最先交互到的层面，它决定了用户能看到什么、看不到什么，这本身就可以成为产品的核心卖点（USP），在某些特定场景下甚至能构成独立的商业模式。其次，模块化往往更胜一筹。将审核逻辑直接烘焙进模型权重确实很酷、很有意思，作为制作调节滑块（slider）的思路也很好，但它的训练成本很高，而且权重是静态的，后续如果想要调整就必须重新更新。因此在绝大多数情况下，我们仍然倾向于采用基于配置文件的简单定制，因为这种方式足够模块化、轻便、可扩展且构建成本低得多。正如我前面提到的，在权重中实现审核完全可行，只是成本高且相对静态。
Original English
**Speaker**: So what we learned from the moderation piece is that, first of all, safety is an extremely important product surface. It's essentially the first thing that users interact with in some sense. It decides what users see, what users don't see, and it can be a USP. It can even be a business model in certain cases. Modularity wins. Baking in moderation into model weights is cool, it's fun, it's a nice concept to build some sliders. However, it's expensive to train, it's static, you need to update it somehow. So in many cases, or in most cases, we just go with simple config customizations because it's just modular, it's easy, it's scalable, right? It's much cheaper to actually set up. And yeah, as I already mentioned, moderation in weights is possible; however, it is expensive and static.
### 模型微调范式拓展与跨模态具身：Flux Action
**Speaker**: 好的，我们继续往下讲。前面我们探讨了如何在审核方向上通过微调改变模型行为，但之前我向大家承诺过要讲讲机器人，承诺过要展示 Flux 是如何用来控制实体设备的。最后，我们将深入探讨模型权重的定制，以及如何专门针对动作预测（action prediction）来对 Flux 进行微调。先为大家梳理一下全貌：通常我们在微调一个模型时，都是带着明确目标来的。如果是在视频模型或图像模型的语境下，我们要么是针对某种特定的视觉风格进行微调，要么是为了精确呈现某个特定物体或一组物体，要么是为了极度精准地还原某个特定角色，再或者就是为了塑造模型的某种特定行为。我们也有一些专门做行为微调的模型，比如图像外绘（outpainting）、局部精准编辑、去性感化等，这些都属于行为层面的微调范式。
Original English
**Speaker**: All right, let's move on. So, we already looked at the fine-tune that changes behavior in the moderation direction, but I promised you some robots and I promised you some Flux controlling some things. So, finally, we're going to look at customization of model weights and how to fine-tune Flux specifically for action prediction. Just to give you an overview, generally if you fine-tune a model, usually we come with a certain goal in mind. If we're talking about video models or image models, we either fine-tune for a certain style, we fine-tune for a certain object that we want to present or a set of objects, we fine-tune for a certain character that needs to be extremely accurately displayed, or we fine-tune for behavior. We also have some models that we fine-tuned for, for example, outpainting, specific local editing, desexualization, right? All of these are behavioral fine-tunes.
**Speaker**: 如果在此基础上再往前推进一步，甚至可以考虑通过微调来赋予模型全新的模态，这就涉及到了中途训练（mid-training）和后训练（post-training）的范畴。而这正是我们在 Flux Action 上所做的工作。Flux Action 是基于 Flux 主干网络打造的一个全新版本，我们在中途训练和后训练阶段持续推进，专门针对机器人动作预测进行了微调。这个模型实际上昨晚才刚刚作为开放权重正式发布，非常新鲜，已经上线了 Hugging Face，大家完全可以亲自去体验一下，下载模型权重并用它来控制属于你自己的 LeRobot 机械臂。
Original English
**Speaker**: And then if you go even further, you can even think about fine-tuning to add certain modalities if we even go a little bit into mid-training and post-training. Again, that's exactly what we did with Flux Action. Flux Action is a new version of Flux, a Flux backbone that we continued mid-training and post-trained and fine-tuned specifically for action prediction. This was actually released into open weights yesterday night. So check it out, it's brand new. It's on Hugging Face. You can try it out, pull the weights, and control your own LeRobot with it.
**Speaker**: 接下来我简单拆解一下它的工作机制。我们本身拥有一个图像模型，它主要基于文本输入来预测帧画面，本质上是在预测视频帧序列。那么 Flux Action 的运作方式是：它同时接收文本条件约束和视频条件约束。在这个具体案例中，我们使用的是一台执行“抓取与放置”（pick-and-place）任务的 LeRobot 机器人。该机器人配备了两套摄像头：一套安装在机械爪手部，另一套安装在上方作为俯视主相机。与此同时，我们还专门针对 SO-100（SL101）这款 LeRobot 机械臂的具身物理结构（embodiment）对 Flux Action 进行了微调。也就是说，我们教会了 Flux：这台特定的机器人具有六个运动轴，而这些轴的运动状态在每一帧中都可以由一个向量来表示，该向量定义了每个轴需要移动的方向。简而言之，我们在这里试图预测的，就是一个六维的动作向量。
Original English
**Speaker**: But just to give you a brief rundown on how it works: so we have an image model that essentially predicts frames, predicts video frames based on a text input mostly, right? So how Flux Action works is that it has a text conditioning and it has a video conditioning. In this specific case, we work with a LeRobot that completes pick-and-place tasks. The robot in this case has two camera sets: one is at the grabber hand, one is above, the top camera essentially. At the same time, we also fine-tuned Flux Action for the embodiment of the SO-100, the LeRobot robot arm specifically. So we actually teach Flux that this specific robot has six axes, and the movement of these axes is essentially represented by a vector that in every frame defines the direction that every axis needs to move into. So essentially, simplified, a six-dimensional vector that we're trying to predict here.
**Speaker**: 现在的流程是：机器人的物理具身先验知识已经被固化在模型权重之中；我们向模型输入当前的视频帧序列以及定义操作任务的文本条件；模型的输出则包含两部分：一部分是后续视频帧连续画面的预测，另一部分则是真正定义机器人为完成任务所需移动方向的动作向量。大家可以在右侧看到 LeRobot 实际上是如何被 Flux 驱动控制的。虽然具体机制你们可能得先相信我的演示，但你们完全可以亲自去上手检验。机器人确实精确完成了预定任务，大家也能在界面上看到对应的文本提示词。这整个过程非常有趣。
Original English
**Speaker**: Now the embodiment knowledge is already embedded in the model weights. We feed in the video frames and the text conditioning which defines the task, and the output essentially is a video of the continuation of these video frames and also the actual vector that defines the direction that the robot needs to move in to actually complete the task. And then you can on the right see how the LeRobot is actually controlled by Flux. I mean, you'll have to trust me that it's actually controlled by Flux, but you can check that out yourself. And it actually completes exactly the task that it's supposed to. You actually can see the text prompts there. But so that is pretty fun.
### 世界动作模型架构与核心总结
**Speaker**: 那么它在底层到底是如何构建的呢？我们来看看具体的架构设计。传统上，我们有一个视频模型，它接收特定的视频帧，并通过变分自编码器（VAE）进行处理。自编码器将像素空间的连续画面投影到潜空间（latent space），生成这些特定帧的抽象特征表征；随后进入去噪扩散过程，逐步消除噪声并尝试预测接下来的视频帧序列。通常在单纯预测图像或视频时，系统还会配备一个解码器，负责把潜空间中的抽象表征重新还原投影回像素空间。
Original English
**Speaker**: How that works in the background, let's take a little look at the architecture. Is that we have a video model. Traditionally, it takes certain frames and it feeds them through a variational autoencoder. The autoencoder projects my frames, my pixel space into a latent space, which is an abstract representation of these exact frames, and then we go through this denoising process that gradually reduces noise and tries to predict these next video frames. Usually, when we just predict an image or a video, we then have a decoder as well that then takes these latents, these abstract representations, and brings them back into the pixel space.
**Speaker**: 而现在的 Flux Action 是一个“世界动作模型”（World Action Model）。这意味着我们甚至不需要再把这些动作向量通过变分自编码器压缩进潜空间，而是直接将动作向量归一化到与潜空间相同的维度，并在同一个 Transformer 骨干网络中直接对动作向量进行去噪生成。这是一个极大的架构优势，因为视频特征与动作向量是在同一个 Transformer 模型内部联合去噪的。最终在执行具身动作时，我们甚至不需要为视频帧部分运行解码器，只需将去噪完成的动作向量进行反归一化处理，直接发送给实体机器人执行动作指令，从而顺利完成作业任务。本质上，视频模态需要编码器和解码器，而物理动作本身天然就是向量形式，因此我们可以顺理成章地将其输入到经过中途训练与微调的模型中，专门用来高效求解这类决策控制问题。
Original English
**Speaker**: Now, Flux Action is a world action model. That means that we don't even send these action vectors through the variational autoencoder into the latent space anymore, but we just normalize them into the dimensionality of the latent space and we directly denoise essentially the action vector. That is a huge advantage because we essentially denoise both in the same transformer. In the end, for the action case, we actually don't even use the decoder for the frames and video part. We only denormalize the action part, we have the robot actually apply it, and that's how we complete the task. Essentially, the video needs encoder and decoder; actions are already a vector, so we can actually naturally feed them through the model that has been mid-trained and fine-tuned on exactly solving these tasks.
**Speaker**: 大家可以去查阅关于 Flux Action 的最新研究成果，下载模型权重。它昨晚才刚刚释出，目前在动作预测领域直接登顶各大排行榜榜首，代表了该领域最前沿的 SOTA 水平。强烈推荐大家去了解一下，这是一项非常令人振奋的研究成果，而对于能够将探索边界拓展到机器人具身这一全新模态，我们整个团队也感到极其兴奋。
Original English
**Speaker**: So Flux Action, you can check out the research, you can download the weights here. It has dropped literally last night. It's right now topping the leaderboards and being state-of-the-art in exactly this action prediction space. So check it out, it's pretty exciting research, and we're pretty excited about venturing into this domain as a new modality.
**Speaker**: 这也正是我想做总结收尾的地方。在本节分享中，我们系统探讨了针对特定业务场景定制 Flux 的核心路径：首先探讨了提示词升采样（Prompt Upsampling），解析了它如何作为一种超高效手段，将用户的原始意图与具体设计规格完美对齐，映射进 Flux 的预训练分布中，进而在规模化生产中稳定输出符合设计要求的图像；随后我们探讨了安全审核策略，对比了以模块化方式嵌入系统配置、具有确定性或准确定性特征的工程化审核方案，以及利用滑块 LoRA 直接将审核偏好烘焙注入模型权重的进阶方案；最后，我们共同探讨了如何为 Flux 注入全新模态，重点剖析了 Flux Action 的内部机理，展示了如何改造 Flux 来实现机器人实物控制与下一阶段动作预测。如果大家要从我们探讨的所有这些维模方案中提炼出一条最核心的认知……
Original English
**Speaker**: And this is also already where I'd like to wrap up. What we basically looked at is customizing Flux for certain use cases. We looked at prompt upsampling and how prompt upsampling can be an incredibly efficient method to actually match user intent with the design requirements into the training distribution of Flux to actually get out an image that at scale conforms to certain design requirements. We also looked at moderation: we looked at sort of static deterministic or more or less deterministic moderation that modularly is embedded in config; we also looked at how to embed moderation preferences into model weights directly using a slider LoRA; and then finally, we looked at how to add a new modality to Flux, specifically looking at Flux Action and how to tweak Flux to actually control a robot and predict the next action. If you want to take away one single thing across all these...
### 模型定制化的核心原则与演讲结语
**Jakub**: 这些方法……这显然还只是冰山一角。如果大家想从整体上的 AI 模型优质定制化、以及针对 FLUX 的具体定制化中汲取最核心的一个认知，那就是：在所有这些场景下，优秀的定制化都应当做到“按需极致具体，同时尽可能保持通用”。也就是说，它们精准解决某一个特定问题，其解决程度刚好契合该问题的实际需求；但与此同时，它又具备通用性，甚至能够推广去解决整个领域的问题，成为一套可以进一步向外扩展部署的解决方案。
Original English
**Jakub**: methods and this is obviously just scratching the surface. If if you want to take away one single thing um about good customizations of AI models in general and also flux specifically it is that great custom customizations in all of these cases are as specific as needed and as general as possible. So they solve one specific problem to exactly the extent that it needs to solve them. But it also generalizes to actually maybe solve an entire domain, maybe be a solution that you can also roll out further.
**Jakub**: 讲到这里，我今天的分享就差不多告一段落了。最后我再插播一条招聘广告：我们目前正在招聘产品工程师（Product Engineers）和解决方案工程师（Solution Engineers）。如果有意向，欢迎了解一下。大家在大会现场也可以随时找我交流，或者在网上联系我。非常感谢大家的参与！
Original English
**Jakub**: So with that, I'll wrap it up. I have another advertising block here. We are hiring for the product engineers, solution engineers. Um if you want, check that out. You can also find me um find me at the conference, find me online. And yeah, thank you very much for attending.
### 主持人串场与嘉宾介绍
**Host**: 感谢 Jakub，讲得太棒了！好的，各位，让我们再次为 Jakub 鼓掌致谢！
Original English
**Host**: Thank you, Yak. Good job. All right. Okay. Yeah. Let's give it up once more for Yaku, please.
**Host**: 谢谢大家。我对 Black Forest Labs 的发展感到非常激动。他们很棒的一点在于他们立足于欧洲，总部位于德国，在生成式多媒体领域的各方面都处于最前沿，而现在他们又进军了机器人领域，这实在太酷了。
Original English
**Host**: Thank you guys. I'm so excited for Black Forest Labs. And uh the nice thing about them as well is that they're based in Europe. They're based in Germany. And they're the at the front of everything generative media related. and now they do robotics which is so cool.
**Host**: 不过大家也知道，另一家同样扎根欧洲且走在 AI 最前沿的企业就是 ElevenLabs。我们接下来的演讲嘉宾要和大家探讨的主题是：如何借助 Orbs（发光球体）实现大规模的生成式视觉身份塑造。在座的各位有谁用过 ChatGPT 的语音模式（Voice Mode）或者 ElevenLabs 吗？如果用过，大家一定见过那个不断转动的小圆圈，也就是那个 Orb。只要你用过它，那很大概率你就使用过我们下一位演讲嘉宾所亲手打造的产品。掌声欢迎 ElevenLabs 的创意体验工程师——Dorian Lods 登台！
Original English
**Host**: But you know another company that is European based and also at the the frontier of AI is 11 Labs. So our next speaker is actually is going to talk to you about generative identity at scale with orbs and basically uh who who here has used the chat GPT voice mode or 11 labs. So you've seen that little circle, that orb. If you use that, you probably have used something that our next speaker has built. So please join me in welcoming to the stage creative experience engineer at 11 Labs, Dorian Lods.
### 从传统创意工程到 OpenAI 标志性语音 Orb
**Dorian Lods**: 看来现场的网络断了，真是个绝妙的开局。好的，没问题。
Original English
**Dorian Lods**: Seems that internet's not working. That's a good start. All right.
**Dorian Lods**: 大家好，我是 Dorian。我是 ElevenLabs 的创意工程师（Creative Engineer），目前常驻巴黎。如果大家想进一步了解我们在 ElevenLabs 的业务，或者稍后有任何问题，欢迎随时关注我的社交账号。
Original English
**Dorian Lods**: Hi everyone. I'm Dorian. I'm a creative engineer at 11 Labs but currently based in Paris. Um if you want to know more about what we do at 11 Labs or if you have any questions later uh please feel free to note the my socials.
**Dorian Lods**: 我今天来到这里，是想从创意视角的维度和大家聊聊过去几年间行业发生的变化，以及我们如何驾驭这一整套全新的、由 AI 赋能的工具体系。不妨先让我往回退一步，讲讲我过去是如何亲手制作单件作品的，而现在我又是如何去构建“能够自动产出作品的系统”的——这无疑是一个巨大的转变。
Original English
**Dorian Lods**: So I'm here today to talk to you about our how in a creative per perspective things have changed a little bit in the past few years and how how we deal with all these new tools set of tools AI tools enabled and so on. So uh just let's take uh let's let me just take a take a step back and tell you how I used to make the thing and now how I make the thing that makes the thing which is a bit of a change.
**Dorian Lods**: 我最初出身的行业，主要职责是打造沉浸式创意体验网站（多数基于 WebGL 构建）、沉浸式互动装置等等。这些项目全都是围绕着某种极致的感知体验而设计的，经历过成千上万个小时的反复精雕细琢。每一个项目都有其独一无二的工艺考究与存在目的。在那个阶段的探索中，我深入接触了如何通过系统化的手段来理顺流程、如何提升产能，或者仅仅是通过全新工具来拓展自身的能力边界——例如像 ComfyUI 这样不可思议的工具，它极大助力了创意工作者去搭建全新、更深层次的沉浸式 AI 系统，但当时它依然只是为了交付某一个单一的结果而服务的，稍后大家就会发现这一点有多么关键。
Original English
**Dorian Lods**: So I came from an industry where used to make immersive creative experience websites are mainly based in WebGL or immersive installation or whatnot. They were all designed around a feeling uh refined over thousands of hours. Each one has a specific craft and and purpose. Um during that journey um I interacted a lot with uh how you can streamline stuff with a system, how you can produce more or just expand your capabilities um with new tools for example comfy UI which is an amazing tool that helped creative new immersive like new deeper uh AI systems but still made for a single outcome and you will see that's important for
**Dorian Lods**: 所有这些背景积累，最终把我引向了我曾经参与过的一个具体项目——就是画面上的这个家伙。这是 OpenAI 的语音形象，也就是他们在发布语音模型时所展现的“面孔”。在此做几点说明：这是我当时必须攻克并创造的第一个标志性 Orb。当时所有的细节全部都是手工精心雕琢的，必须完全契合品牌调性，其核心目标是找到一种视觉化的方式，在代表品牌形象的同时，将你正在与之对话的声音具象化表达出来。它所采用的技术栈与之前基本一致：纯手工精调、完全确定性，大家在屏幕上看到的一切全部由 Shader（着色器）代码驱动。换言之，本质上是用数学公式去渲染视觉图形，所有的输入参数、所有的颜色配比、以及这个物体的一切动态交互行为，都是经过人工严密校准的。但说到底，它只是为了一个单一的目的而构建的。我花了整整几个星期，就为了设计这单独的一颗球体。
Original English
**Dorian Lods**: And all of this background led me to a specific project I work on which is this guy. Um it's the OpenAI voice the face of OpenAI when they released the voice model. Um few notes but it's was the first iconic orbs that I had to do. uh everything was handcrafted, had to fit on brand and was the goal was to to find uh a way to visually represent the voice that you talk to uh while representing the brand. So pretty much the same stack than before. It was hand tuned, deterministic, everything you see on screen is made with a shader. Um so math actually makes the visuals, all the inputs, all the colors, all the way the thing behaves were carefully tuned. But it was just made for a single purpose. And I spent few weeks designing one sphere.
### 面向 4000 万声音：ElevenLabs 的业务规模与社会价值
**Dorian Lods**: 而在那之后，摆在我面前的任务却是要为 4000 万种不同的声音进行设计——毫不夸张地说，这是一个规模维度的剧烈跃迁。
Original English
**Dorian Lods**: Then I had to design for 40 million voices which is a big change of scale if you ask me.
**Dorian Lods**: 于是我加入了 ElevenLabs。对于还不了解我们的朋友，ElevenLabs 是一家以欧洲为根基优先发展的企业。我们处于行业的引领地位，拥有专属的前沿研究团队，深耕于任何语音界面、语音交互、音乐创作——老实说涵盖了几乎所有声音领域，但我们的业务范畴远不止于此。我们拥有自主的创意平台，用户可以在上面进行配音译制、创造全新声音、克隆自己的声音、通过提示词生成声音等等。已经有海量的创作者在深度使用我们的平台，我们拥有网页端，有移动端 App，如今还推出了 Eleven Agents。我们用它承载客户支持通话、构建自身的业务编排调度，涵盖了非常多的应用场景。
Original English
**Dorian Lods**: So I joined 11 Labs. Uh for the one who don't know us, 11 Labs is a European firstbased company. Uh we are leading the way on we have our own frontier in the research team dedicated to any voice surface any voice interaction music any sounds to be honest but we do much more than that. We have our own creative platform where you can um dub make new clone your voice prompt another voice um and so on. So many of our or many many creators uh already use the platform. We have the web one, we have the app, we also have uh 11 agents now. So we handle our customer support call, we do our own orchestration, many many many things.
**Dorian Lods**: 接下来列举一些核心业务数据，毕竟大家都喜欢看真实数字，对吧？我们目前支持超过 90 种语言；平台上已汇聚了数以百万计的企业开发者与创作者；截至 6 月底，我们的年度经常性收入（ARR）已突破 6 亿美元；截至今年 2 月，公司估值达到 110 亿美元；目前我们的团队规模已扩张至 700 多人，遍布全球 50 个国家；平台活跃的智能体（Agent）数量已超过 1000 万个。我知道这些数字非常庞大。
Original English
**Dorian Lods**: Um here are some key numbers because everyone loves numbers, right? Uh we support over 90 languages. Uh we already have millions of business developers, creators on the platform. Uh we crossed 600 millions of RR by end of June. uh current valuation is at 11 billion as of February and now we actually expanded the team to over 700 people across 50 countries. We have more than 10 million agent on the platform. It's a lot of numbers I know.
**Dorian Lods**: 但这里还有几项更加有趣的数据：每一天，我们的语音转文字（STT）模型所生成的音频时长累计超过 34 年——这个规模体量可见一斑；我们在音乐模型以及各类音乐触点上已经累计生成了 2500 万首音轨；而其中我个人最深爱、也认为最有意义的数据是：截至目前，我们已经帮助超过 11,000 名因疾病或意外事故失声的人群重新找回并复原了他们的声音。我们对这一事业怀有极其坚定的承诺，我们的社会影响力团队（Impact Team）做出了非凡的贡献，免费将发声能力重新带给那些真正迫切需要它的人们。
Original English
**Dorian Lods**: Uh but some here are some fun ones. So every day we generate more than 34 years of audio with our speech to text model. Um so the scale factor is definitely here and uh one that uh we we made 25 million tracks with our music our music models and different music touch points and one that I really really u that I really love I think it's my favorite number here is that we so far helped to restore more than 11,000 voices for people who have lost them due to illness or injury. We're super committed towards that goal. Our impact team is doing an exterior job uh to bring voice capabilities back to the people who actually need it for free.
### 视觉身份重塑：声音即面孔的生成系统设计
**Dorian Lods**: 讲得更具体一些，我是加入了 ElevenLabs 的设计团队。我们整个设计团队基本都在这里了。我们致力于在品牌呈现、产品体验、人机交互方式等各个前沿阵地进行突破，结合平台拥有的丰富能力，探索如何实现规模化提效。
Original English
**Dorian Lods**: Um but more specifically I joined 11 Labs design. So the design team is pretty much all us here. Um we try to push the frontier of both the brand, the product, the way you interact with AI with all our stuff that we have on the platform and how we can streamline that.
**Dorian Lods**: 我入职后迎来的第一个真正挑战，就是去探索“声音”（作为我们产品中最核心的交互媒介）应当如何拥有属于自己的视觉身份，以及这一套身份系统如何在整个产品矩阵中融会贯通。换言之，就是如何为我们平台上的声音注入更多的“视觉灵魂”，同时这些视觉灵魂又必须准确映射出 ElevenLabs 的品牌特质。下面这段视频展示了我们在过去几个月里秘密研发的成果……千万别卡住……好的。嘿，视频放出来可能已经提前剧透了一些细节。
Original English
**Dorian Lods**: Um I join with my first uh challenge actually uh was to explore how voices which is our main interaction uh could have a visual identity and how that identity could work across the product. So how you bring more soul visual souls to our voices and how those souls should reflect the brand. Uh here's what we've been cooking in the past few months. Do not first down. Yeah. Hey. Hey. I might have spoiled a few things with that video.
**Dorian Lods**: 我们的核心立意其实非常纯粹，但实现起来却绝非易事：我们坚信“每一种声音都是一张面孔”（Every voice is a face）。可当时平台上已经存在着 4000 万种声音。用户可以通过提示词生成声音，可以克隆自己的声音，我们承载着种类极其繁杂的声音形态，如今全部汇聚成了超过 4000 万种声音。你该如何设计出一套足够具有辨识度、能够准确转译声音内在特征，同时又完整收敛于统一视觉体系之内的设计语言？每一种声音既要传递出 ElevenLabs 的品牌形象，其观感又必须始终让人觉得“这就是 ElevenLabs”。这一切完全关乎感知体验，以及如何在宏大的规模下将其工程化落地。
Original English
**Dorian Lods**: Uh so um the pitch was simple but yet not so much. We believe that every voice is every face. Um and then we had 40 million voices on the platform. So uh users they can prom voice, they can clone their voice, you have we serve many type of different voices and so they all uh they all represent now more than 40 million voices. And how do you design something that is iconic enough that still translate the voice characteristic but keep it in one visual system that each voice needs to bring the identity of 11 labs and it still need to feel like 11 Labs. So it's all about feeling and developing it at scale.
**Dorian Lods**: 首先第一点，思维框架必须彻底重构。工作模式再也不能由我个人去手工产出单个视觉结果，而必须转变为由我去打造一个“能够源源不断产出视觉结果的系统”。这非常贴近我们昨天以及今天上午所听到的讨论——比如代码工场（Code Factories）之类的概念，只不过我们将其应用到了设计领域。Orb 的核心技术架构本质上是两层叠加系统：底层是图像层（Image Layer），承载着品牌的视觉特征并与声音的本质属性深度锚定；顶层则是实时渲染层（Real-time Layer），赋予其鲜活的生命力，为底层的静态视觉增添动态交互性与呼吸感。相比我之前在 OpenAI 做的项目，这在设计思想上是一个相当重大的转变。这种视觉触点随后会全面扩散贯穿至整个产品的各个角落。
Original English
**Dorian Lods**: Uh first thing first it needed a reframe. So it could not just be me making the outputs anymore. It was me making the system that makes an output. very close to what we've been hearing in the past yesterday basically and today this morning uh with code factories and so on but here more on the design side uh the core concept of the or is it's actually a stack of two layers you have the image layer that carries the brand characteristic the voice that it's actually linked to the voice and then you have the realtime layer on the top which brings that to life at the interactivity had the living piece to to the one in the back. So, a bit of a change compared to the first one I worked on at OpenAI. Uh, and that touch point then is diffuse everywhere across the board.
**Dorian Lods**: 我们先重点解析图像层。它的底层逻辑与声音直接挂钩：一种声音对应一张面孔。只要你生成了一个新的声音，就会同步生成一个新的 Orb。因此我们追溯至源头，对每一种声音的音频输出进行深度分析，提取出核心指标与关键数值，随后将其归类至音高（Pitch）、能量（Energy）、表现力（Expressiveness）等不同维度中，以此驱动整个生成管线的后续运转，使其核心与声音本身的声学特征建立紧密连接。其实在一开始，我们原本打算直接采用声音嵌入向量（Embeddings），但是……
Original English
**Dorian Lods**: Let's focus first on the image layer. Um, it's as it core it's linked to the voice. So, one voice equals one face. If you make a new voice, you make a new a new or So we need to we went down to the source and we we analyze the voice output for air like the audio output for every voice. We extract key metrics, key numbers um that that then fall under different category of pitch, energy, expressiveness that that drive the rest of the pipeline where at this core it's very much linked to the voice. Uh in the first place we wanted to use embeddings directly but
### 声音分布分析与品牌视觉模型训练
**ElevenLabs 设计师**：金丝雀发布（canary）后来变得有些复杂。为了开展工作、设计整体流程，并切实拓展或探索我们究竟需要覆盖多大的范围，我们从平台上提取了 20 万个最受欢迎的声音样本。随后我们发现，例如这些声音的分布其实非常不均匀：低沉的声音数量要多得多，这主要是用于旁白叙述的目的。因此，我们必须重点关注这类主流类别，而其他类型可能属于小众领域——比如那些音调极高、极富活力，甚至类似于卡通风格的声音。不过，这确实在整个过程中给了我们很大的帮助，让我们得以在大规模层面去探索这件事。
Original English
**ElevenLabs Designer**: canary became a bit complicated later. Um to work to design the process and to really uh expand or to explore how much range do we need to work with. We extracted a set of 200,000 voices the most popular we have on the platform. And we realized for example that they are not distributed evenly. you have much more deep voices mainly for narration purposes. So we really had to focus in those kind of categories where the other ones might fall under like niche where it's super high super energetic like maybe cartoonish style of voices. But uh that really helped us a lot during the process to to explore the thing at scale.
**ElevenLabs 设计师**：首先，我们希望这些声音能够传达出品牌的感觉。所以我们需要教会模型去“说”ElevenLabs 的语言。每一个声音都是独一无二的，目标是为每一个新声音生成专属的全新光球视觉形象（orb visual）。因此，我们必须找到一种方法，让模型真正学会我们的设计语言以及如何进行这种映射转换。开箱即用的现成模型（off-the-shelf models）根本不了解你的品牌。于是我们构建了一个合成数据集，在基础模型之上专门训练了一个特定的 LoRA 模型。实际上我们还探索了一件非常有趣的事情：我们通过提示词文本（caption）来引导推理过程。
Original English
**ElevenLabs Designer**: Uh first thing first we wanted to um the voices to feel like the brand. So we needed to teach a model to speak 11 laps. Each voice is unique. The goal is to create a generate new voice per new visual new or per voice. So we needed to find a way so a model can actually learn our design language and how that translate. Uh so of the chef they don't really know your brand. Um so we built a synthetic data set to train a specific lura on the top and actually there's pretty fun thing that we explored is that we steer the inference uh with the caption.
**ElevenLabs 设计师**：因此，除了动作词之外，我们还人工挑选了多种色彩以及各种想要赋予不同声音类型的特征，比如高音调、中高音调、中音调等。接着我们将这些内容放入训练样本的描述文本（caption）中。这样在训练期间，模型就能受到一定的定向引导，展现出更丰富的多样性。所有这些关键词随后都被重新放回了特征库中。所以在推理阶段重建提示词时，模型就可以精准选择在训练数据中出现过的完全相同的关键词和子关键词。这真的非常有趣。
Original English
**ElevenLabs Designer**: So on top of the of the action words we hand selected multiple type of colors and things that we wanted to attribute to different kind of voices. Let's say high pitch, mid pitch, medium pitch. Um and then we we put that in the caption. So during the training um the model could actually steer a bit and like develop a bit more uh the varity and all those keywords were actually were put back into inventory. So at inference level when you rebuild the prompt the model can choose exactly the same keywords like sub keyword that has been in the training data. That was really fun.
### 开源模型权衡与后推理图像处理
**ElevenLabs 设计师**：我们探索了不同类型的模型，其中一个来自我们的朋友，这正好是一个很好的过渡。我们尝试并探索了各种开源权重模型（open-weight models）。这里的目标是在推理成本和大规模开发之间找到一个最佳平衡点，同时依然保持极高的画面质量。在生成基础图像之后的“后推理”（post-inference）阶段，核心思路基本上是对其进行模糊处理，并在基础图像内部进行局部裁剪。这样一来，每一个变体、每一个光球（orb）都能蕴含非常出色的差异与变化。这就是我们所谓的后推理阶段，它为后续着色器（shader）的处理做好了准备。
Original English
**ElevenLabs Designer**: Um we explore different type of model uh one from our friends. So it's a good segue actually. Um so we we try to yeah we explored different kind of open weight model. The goal was to f to find a good sweet spot between uh between like inferencing cost and development at large scale while still maintaining the quality. um post inference after you generate the base image the idea is that basically you blur it and you crop it inside the inside the base image. So each variation each or has a really really good variation into it. Um that's what we call post inference. Um and that prepares for the rest of the shader.
### 光球着色器架构与多平台渲染
**ElevenLabs 设计师**：现在让我们聚焦在这些光球的实时渲染层上。在设计阶段，我们实际上将代码作为核心的设计媒介，因此它最终以着色器（shader）的形式存在。它以背景图像作为输入，并在其上叠加多个图层。所有这一切都是通过一个交互界面完成的，整个设计团队都能在这个界面中微调参数，直观感受视觉效果的变化。其中一些参数会根据声音本身以及音频反馈进行动态调整。WebGL 负责平台和营销网站上的交互式体验，但我们将其转换并支持了 OpenGL 服务端渲染（SSR），用于生成预览图或者支持那些本身无法直接运行着色器的其他触点。
Original English
**ElevenLabs Designer**: So let's now focus on the real time layer of those orbs. Um during the design phase we we actually used code as a core design medium. So it lives as a shader. Uh it takes the image in the back and has multiple layers on top of it. Everything was uh made through an interface where all the design team was able to tweak and like see how we can feel things. Uh some parameters are tweaked with the sound itself with the audio feedback and so on. Um, WebJ is the interactive experience on the platform on the on the marketing website but uh uh but we converted this to OpenGL support for server server side rendering for many previews or other touch points that cannot actually run the shader themsel.
**ElevenLabs 设计师**：这个着色器本身基本上是由六个堆叠层组合而成的。首先是一个实时流体模拟，具有与音频联动的涟漪效果（ripple effect），这赋予了它极其逼真的有机生命感。这里我们的目标——我刚才可能漏提了这一点——是将光球锚定在自然属性中。“自然”是光球最主要的核心特征，我们希望它们让人感觉是有机的、充满生命的。接着在顶层叠加了多阶段的噪声来注入生机，例如分形布朗运动（FBM）等。此外还有进一步的声音波形层，与我们之前的声波视觉相衔接，并加入了我们在许多产品触点上都会应用的品牌专属颗粒感胶片质感（brand grain）。
Original English
**ElevenLabs Designer**: The shader itself is composed of basically six six stacks together. You have a realtime fleet simulation that has um the reaper effect that's linked to the audio. It's really bring the organicness into it. The goal I might have skipped that but the goal was to anchor the orbs into nature. Nature is the main characteristics of the orbs. You wanted to fill them to feel organic. Then you have multiple phase of of noise on the top to bring it to life. FBM and so on. And we have a a further sound wave that is actually a link with the one we had before and the the the brand grain that we put on many of our of our touch points.
### 着色器演练场与大规模评审工具
**ElevenLabs 设计师**：所有这些催生了着色器演练场（shader playground）。正如我之前提到的，在早期阶段，大部分的打磨和工艺探索都是在这里完成的。但在其背后，我们实际上开发了一套评审工具，以便在大规模生产环境中对一切进行调试排查。因为每一个声音对应一个输出结果，这也意味着用户无法手动挑选输出画面，每一个生成的结果都必须准确无误。我们很快意识到，为了能够在全平台范围内维持一致的品质，我们必须具备规模化重新验证的能力。这是一个至关重要的关键因素，因为单独调试时效果可能看起来非常惊艳，但一旦放到大规模数据中运行，系统可能就会崩溃。因此，构建用于大规模评审的内部工具确实带来了根本性的改变。
Original English
**ElevenLabs Designer**: Um so all of that led to the shader playground like I said before it was the place where most of the craft was happening early on. Um but behind that we actually developed a reviewing tool to debug everything at scale. So one voice equal one output. So you also mean that you cannot choose the output and every output had to be right. So we quickly realized that in order to be able to maintain quality across the board, we needed to redo it at scale. Uh and that was really a key factor because you might end up with your debug ones that are super nice and then as soon as you run it at scale, it's actually fall down. So building internal tools to review things at scale really really make the make a change.
### 成本与延迟优化：图集生成架构
**ElevenLabs 设计师**：接下来是不那么有趣的部分。我们原本拥有一套非常出色的操作流程和极其顺畅的工作流，然而当我向财务部门汇报时，他们告诉我：好吧，这个方案确实非常棒，但如果是数百万次调用乘以每次 10 美分，那成本实在太高了。所以你必须想办法同时降低延迟和成本。于是我们重新审视了之前展示的平台上声音分布图表，并将其作为核心基准数据源。我们不再每次都生成一张全新的图像，而是一次性生成一张庞大无比的纹理图集（atlas）。
Original English
**ElevenLabs Designer**: Um that's the less fun part. Uh so I had an we had an amazing an amazing op process and amazing workflows. Uh and then I run through uh finance. Then they told me that okay D is super nice but for millions times 10 cents it's way too much. uh so you need to find a way to reduce both latency and the cost. So we went back to the uh to the chart that I shared before where we have all the distribution of voices in the platform and we actually um took this as the core source. So instead of generating a new image every time, we generated a gigantic atlas at once.
**ElevenLabs 设计师**：随后，我们从这张图集中在各图像更深处进行采样裁剪。这种方式让我们无需在每次创建新声音时都重新生成整张图像，但依然保持了视觉的独特性。每一个声音依然独一无二，因为我们对其进行了旋转、差异化裁剪、色相微调等处理，而且它们仍然高度绑定于底层的音频声学特征。这一优化帮助我们节省了超过 360 万美元，效果非常显著！同时运行速度提升了 10 到 15 倍，每个声音生成视觉形象的时间缩短到了 3 秒以内。当用户创建一个新声音时，这一切都在后台静默发生：音频通过整套处理流程，在基础架构上迅速启动任务并生成视觉图，整个过程不到 3 秒，因此对用户来说完全是无感知的。
Original English
**ElevenLabs Designer**: And then from this atlas, we actually crop a bit deeper in each of the image. And so uh that help us to not generate every time we we make a new voices but still maintaining a uniqueness. So every voices is unique because we rotate it, we crop it differently, we change a bit to U and so on. uh and they are still really much um locked into the audio characteristics. That helped to save more than 3.6 million. So pretty good. Uh run it 10 to 15 times faster and have less than three seconds per voice to generate the visual. So it have it's happening on the back as soon as you as you make a new voice and then it run through all the the process audio process. It spin up it spin up the task on the infra and then generate the one. So less than 3 seconds actually make it not visible for the user.
### 全链路管线与多端视觉一致性
**ElevenLabs 设计师**：这里简要回顾一下针对声音形象的整体流水线。首先是声音分析阶段，将核心音频指标映射到视觉特征上，例如中频声相、能量级别、纹理等。接着进入我刚才展示的完整生成式流水线，负责准备各类素材资产、准备着色器基底、纹理图层、预览图像、缩略图、动态 MP4 视频，并将所有这些资源分发至 CDN 等基础设施。最终生成了超过 4000 万个视觉成果，广泛服务于各个业务场景，大家稍后就会看到。
Original English
**ElevenLabs Designer**: Um and so here's a recap of the pipeline for the voices. You have the voice analysis in the first place that maps that map key metrics to uh visual mapping like a medium pen and energy level texture and so on. Then we have the entire generative pipeline I just showed you that prepared the assets, prepared the shader base, um the texture, the preview, the thumbnail, animated MP4, all of these diffused CDN and so on. And at the end you have 40 million plus outputs uh that serve pretty much everywhere as you can as you're going to see in a few.
**ElevenLabs 设计师**：高质量的工程开发至关重要。核心设计理念固然很酷，但我们希望它在任何地方都能栩栩如生，与我们在平台上构建的光球全流程管线紧密贴合。在左边可以看到声音克隆界面，光球会根据你的声音特征实时产生变化。我们希望在 iOS 和 Android 端保持一致的质量——也就是中间所展示的效果，每个着色器呈现的视觉质感必须高度统一，这对品牌至关重要。同时我们必须在大规模环境下进行审查，右边展示的就是后台管理界面，供我们审查系统中的每一个声音。
Original English
**ElevenLabs Designer**: Quality development was really really important. Uh the concept the core design concept was was cool but we wanted it to feel alive everywhere to have to to to follow all the pipelines of the orbs that we have on the platform. Uh here you see on the left you have the the voice cloning um the voice cloning interface. So the orbs change depending on your voice. Uh we wanted to maintain quality across iOS, Android, uh it's the one you see in the middle. Every shader had to look the same. Super important for the brand. Uh and we needed to review it at scale. So on the right side, you have like the admin interface where you can review every voice that we have.
### Picasso 服务与 4000 万存量回填
**ElevenLabs 设计师**：所有这一切规模都在不断扩大。我们构建了一套专用服务——一个专门用于生成光球视觉的专用云服务。现在它被称为“毕加索”（Picasso），因为他是位伟大的画家，而我本身是法国人。这是一套跨所有产品触点共享的统一 API，我们将这些光球推向了各个地方：核心应用程序、营销官网、iOS 与 Android 客户端、我们的音乐品牌 11 Music，并进一步扩展到品牌核心体系本身。我们打造了品牌工具套件、动态视觉设计，如今所有的东西都在使用光球，它们是品牌的承载体、声音的承载体，也是产品视觉面貌的承载体。
Original English
**ElevenLabs Designer**: All of that started to grow. Uh we made a dedicated uh service. So we have a dedicated um cloud service for making our orb visuals. Now it's called Picasso because it's painter and and I'm French. Uh it's a shared API across all our touch points. We we push those orbs everywhere. So in the core app the marketing website iOS and Android app uh 11 music which are which is our music brand branding and expanded into the brand core itself. So we made brand tools motion designs everything now used orbs the carrier brand the carrier of voice and the carrier of face.
**ElevenLabs 设计师**：它必须保持向后兼容，因为平台之前就已经存在旧版光球。因此我们对平台上的 4000 万个存量声音执行了历史数据回填（backfill）。这项回填任务连续运行了好几天，那确实是一段令人压力倍增的时刻，但最终圆满成功。现在光球无处不在，已经势不可挡了。它们成为了我们官方主页的重要组成部分，你打开主页第一眼看到的就是光球；它们也深度融入了 App 本身。比如大家在这里看到的，更低沉的声音往往颜色偏深；如果声音拥有更高的能量，就会泛出蓝色的光晕色调……
Original English
**ElevenLabs Designer**: Uh it was backward compatible because we had already previous type of orbs in the platform. So we were able to run a back field of 40 million voices. Uh that back that back field run for a few days. Um really stressful moment but it worked at the end. Uh and now they are everywhere. They we cannot hold them anymore. They they are part of our um homepage. Uh the home page the first thing you see on the homepage is the orbs. Um they are part of the app itself. So you see for example here you have deeper voices has a tendency to be a bit darker. if they have more energy, they will have like blue hints
### 品牌视觉生成实践与多地户外广告活动
**Dorian**：……等等诸如此类的特性。因此，我非常真诚地邀请大家亲自去体验这个平台，动手尝试筛选不同的声音特征，看看声音的每一项独特属性究竟是如何具体转化为视觉画面的。
Original English
**Dorian**: ...on and so on. So, I really invite you to play with the platform and to try to filter voices to see how each characteristic of voice translates into visuals.
**Dorian**：这里展示了更多的渲染效果，它们实际上已经跳出了屏幕，走进了现实世界，这真的非常酷，也是我个人感到超级自豪的一项成果。我们把这些视觉实体作为核心品牌元素，贯穿应用到了我们所做的所有事情当中。
Original English
**Dorian**: Uh more renders and they actually came out of the screen, which is really, really cool and something I'm super proud of. We use them as a core brand elements toward everything we do.
**Dorian**：比如在左边这里，展示的是我们在法国举办的一场户外广告（OOH）宣传活动。在活动中，你可以根据我们部分客户自身的品牌视觉定制生成专属的光球。例如这里展示的就是客户 Alan 的案例。而在右边展示的是伦敦的活动，我们直接承包了伦敦的一个核心主广场，用客户 Trainline 的品牌色彩 DNA 等视觉元素进行了全面呈现。
Original English
**Dorian**: So on the left here it's an out-of-home campaign we made in France where you could have spot orbs based on the branding of some of our clients. Here is Alan for example, or on the right side you have London where we took over one of the main squares of London with Trainline color DNA and so on.
**Dorian**：所以这又是我们的另一个客户案例。光球这一载体本身其实并不承载某种固定单一的身份，它本质上是一个形态变换器（shape-shifter），因此我们完全可以用它来生动表达和呈现不同客户在我们所构建的业务场景下的个性化诉求。
Original English
**Dorian**: So once again another clients of us, and then the orbs as itself doesn't carry identity, but it's a shape shifter, so we can also use it for expressing, so illustrating the needs of our clients towards the thing we do.
**Dorian**：这里是另一个在巴黎地铁投放的实际案例。每天去上班的路上，能在地铁里亲眼看到自己团队设计生成的这些作品，那种感觉真的特别棒、特别有意思。
Original English
**Dorian**: And here another example in the Paris metro. It was really fun to go to work and then see that on the metro.
### 核心认知：将机器学习作为创意媒介的设计哲学
**Dorian**：接下来想和大家分享一下，在经历这整个探索过程中我们所沉淀下来的核心心得。首先，把机器学习作为创意的核心媒介，会彻底改变你做设计的方式。它重塑了你的思考维度：你不再仅仅着眼于最终的输出结果是什么，而是必须深入思考你究竟是如何把这个结果创造出来的。
Original English
**Dorian**: Um so here is what we've learned during this little process. Using machine learning as a core creative medium change how you design. It changed the way you think. You don't think about only the outputs, but you need to think about how you create that.
**Dorian**：机器学习与人工智能工具从根本上来说就是超能力，但你必须极其谨慎地去使用它们。面对你所做的每一项设计工作，切记不要轻慢对待。如今制作和生成内容的边际成本已经大幅降低，这往往会让人产生一种拼命追求快速交付、不断加速发版的冲动。但作为一个设计团队，我们深信，品质上的极致优势才是最终驱动品牌认知度提升与品牌美誉度的关键所在。
Original English
**Dorian**: Machine learning and AI tools are basically superpowers, but they need to be used carefully in every design thing that you do, don't overlook it. Now the cost of making the thing has become so much lower that we have tendency to ship fast and fast and fast, and we believe as a design team that the quality edge is going to make you actually want more of brand awareness, brand quality, and so on.
**Dorian**：因此，你必须完全掌控整个管线（pipeline）。你必须对系统生成的每一个输出结果全权负责，就像它们全都是你亲手一笔一画绘制出来的一样。设计的核心手艺已经转移到了数据和规则的定义当中。这正是我们在面对大量 AI 生成内容时维持质量标准的方式，以确保每一个成果都严格遵循既定的品牌规则与感官调性。
Original English
**Dorian**: So you need to own the pipeline. You need to own all the outputs like if they were your own. The craft shifted into the data and rules. This is how you trying to maintain all the AI stuff that we have and to be sure that they follow a branding rule, a feeling rule, and so on.
**Dorian**：在每一个阶段，都必须毫不动摇地捍卫品牌愿景。正如我前面所强调的，品牌塑造在当下正变得越来越核心、越来越关键。此外，系统性模式会揭示出单个独立样本所掩盖的问题——这意味着我们必须具备在规模化层面进行排错与调试的能力。
Original English
**Dorian**: Every stage had to preserve the brand vision. Like I said, branding is getting more and more key. And the patterns revealed what a single output hid: that means that we need to debug at scale.
**Dorian**：如果你正在构建的管线或流程未来将被部署并调用 4000 万次，那么你就必须学会在规模化场景下去做 Debug。因此，不要犹豫在较小规模下进行多次生成迭代测试，以确保你能够对管线产出的每一个具体结果都胸有成竹。
Original English
**Dorian**: If you're making a pipeline, a process that it's meant to be deployed 40 million times, you need for example to debug it at scale. So don't hesitate to run multiple run of generation at a smaller scale to be sure that you can own every single output.
**Dorian**：所以正如我一直强调的，我依然无比在乎生成的每一个结果。它们就像是我的孩子一样，我为它们每一个都感到骄傲，其中没有任何一个是我不喜欢的。但我现在的角色，不再是亲自去逐一绘制设计每一个具体图形，而是去设计能够孕育出这些结果的条件与系统规则。这极其重要，而且这种思维范式的转移，我们此前已经在代码工程领域见证过，而现在，它也正全面发生在设计领域。
Original English
**Dorian**: So I still care about every output like I said. They are all my—I'm all proud of them. There is not any one that I don't like. But instead of designing them, now I designed the condition that produce it, and that's really, really important, and it's something that we've seen on the code side and we also see it now on the design side.
**Dorian**：以上就是我要分享的全部内容。非常感谢大家的聆听！如果大家有任何疑问，请随时与我联系交流。欢迎访问我们的网站、体验我们的应用或查看更多案例。非常感谢大家！
Original English
**Dorian**: That's pretty much it for me. Thank you for your attention, and if you have any questions please don't hesitate to reach out. Check out our website or apps or everything, and thank you very much.
### 主持人串场与新讲者登台
**主持人**：太精彩了，非常感谢 Dorian！让我们再次用热烈的掌声送给 Dorian，谢谢！好的。
Original English
**Host**: All right, thank you so much, Dorian. Let's give it up one more time for Dorian, please. Thank you. All right.
**主持人**：接下来现场举手调查一下：在座的朋友里，之前用过开放权重模型（open weights models）的请举手？太棒了，大家都很熟悉。那么只要你用过，你大概率就使用过来自我们下一位讲者所在公司的开源工具。
Original English
**Host**: Okay, so show of hands. Who's here has used open weights models before? Okay, cool. So if you did, you probably have used the tools from our next speaker company.
**主持人**：下面有请来自 Hugging Face 的技术专家。他将带大家深入探究 Hugging Face Transformers 模型是如何在 vLLM 推理引擎中完成加载与运行的。大家请坐稳扶好，接下来的内容将会非常硬核、非常硬派。请大家和我一起热烈欢迎来自 Hugging Face 的机器学习工程师——Harry Miller！
Original English
**Host**: So up next is someone from Hugging Face. He's going to dive deep into how a transformers model loads in vLLM. So yeah, hold on to your seats guys, it's going to go deep. So please join me in welcoming machine learning engineer at Hugging Face, Harry Miller.
### vLLM 核心机制与架构优势
**Harry Miller**：好的，谢谢大家！大家好，我是 Harry。正如主持人刚才介绍的，我是 Hugging Face 的机器学习工程师，同时也是 vLLM 的核心维护者之一。
Original English
**Harry Miller**: All right. Okay. So, hello everyone. I'm Harry and as you just heard I'm a machine learning engineer at Hugging Face and I'm also a maintainer of vLLM.
**Harry Miller**：在今天的分享中，我将向大家详细拆解：一个 Hugging Face Transformers 格式的模型是如何能够被直接加载进 vLLM 当中的；以及为什么经过我们针对底层所做的一系列优化改造后，它的运行速度能够完全媲美在 vLLM 中拥有专属手写原生实现（dedicated implementation）的模型。
Original English
**Harry Miller**: And in today's talk I'm going to talk to you about how a Hugging Face transformers model can be loaded directly into vLLM and why, because of what we do to it, it will run just as fast as if the model had a dedicated implementation in vLLM itself.
**Harry Miller**：首先，考虑到在座可能还有一些朋友不太了解 vLLM，我先为大家补充一点背景信息。vLLM 是一个顶级的开源大语言模型推理加速引擎，最初诞生于加利福尼亚大学伯克利分校的 SkyLab 实验室，如今已经正式成为 PyTorch 基金会的托管项目。
Original English
**Harry Miller**: But first a little bit of context in case you haven't heard of vLLM before. Uh, vLLM is an open-source LLM inference engine. It came from the UC Berkeley SkyLab and is now a PyTorch Foundation project.
**Harry Miller**：vLLM 带来的核心杀手锏特性主要包括：第一是 PagedAttention，它借鉴虚拟内存分页机制，能够实现极高显存利用率的高效 KV 缓存管理；第二是连续批处理（Continuous Batching），它允许请求在抵达系统的瞬间立即加入当前执行流进行计算，而完全无需等待下一个完整 Batch 的启动；第三是高性能计算算子（Fast Kernels），并支持跨多种不同维度的并行方案实现极致扩展；第四是卓越的可移植性（Portability），能够在任意硬件环境与基础设施上顺畅运行；最后是广泛的模型支持，既支持上百种模型的专属原生手写实现，也支持即将在下文重点介绍的 Transformers 建模后端（transformers modeling backend）。
Original English
**Harry Miller**: Um, and the key things that it brings is PagedAttention which allows you to KV cache efficiently. Um, continuous batching which allows you to process a request as soon as it arrives rather than waiting for the next batch to begin. Fast kernels scale out through many different axes of parallelism, portability by running on any hardware in any environment, and then broad support for hundreds of models both through dedicated implementations and the transformers modeling back end which you'll hear more about imminently.
### 为什么需要 Transformers 建模后端
**Harry Miller**：接下来谈谈 Transformers 建模后端。补充说明一下，它的核心作用正是让你能够直接复用来自 Hugging Face Transformers 库中的模型定义代码，并将其无缝部署运行在 vLLM 引擎中。
Original English
**Harry Miller**: And so then the transformers modeling back end, um just again more context, is what lets you take this transformers modeling code from Hugging Face transformers and run it directly in vLLM.
**Harry Miller**：这意味着：模型的结构定义、配置（Config）以及模型权重全部直接来源于 Transformers 库；而底层执行运行时（Runtime）以及所有对性能至关重要的核心算子模块，则全部交由 vLLM 接管与驱动。这样一来，你就完美兼得了两者的优势。
Original English
**Harry Miller**: So the model definition, the config, and the weights all come from transformers, and then the runtime and all the performance-critical modules come from vLLM. So you kind of get a best of both.
**Harry Miller**：那么，在什么情况下你会选择使用 Transformers 建模后端呢？通常有三大核心原因。第一个原因，是你本身就已经深度植根于 Transformers 生态系统之中，自然不希望每当要支持一个新的推理引擎时，就不得不为该引擎重新手写重构一遍你的模型。借助 vLLM 以及 SGLang 等引擎所提供的这类建模后端，你在 Transformers 训练循环中编写过一次的模型代码，就可以直接原封不动地直接在 vLLM 或 SGLang 中高效跑起来。
Original English
**Harry Miller**: And so why might you end up using the transformers modeling back end? Well, there are three main reasons. One of them might be that you're already using the transformers ecosystem. And so you don't want to reimplement your model again for each inference engine that you might want to support. So through backends like this one for vLLM and then also SGLang, um you can take this code that you wrote once for your transformers training loop and run it straight in vLLM or SGLang.
**Harry Miller**：第二个原因，是强化学习（RL）场景。在传统的 RL 训练流程中，大家通常倾向于使用 Transformers 库来执行训练，而到了 Rollout 推理采样阶段，则切换使用 vLLM。在过去，由于没有统一机制，两边必须各自维护一套不同的模型实现代码。而现在有了 Transformers 建模后端，你在整个 RL 闭环中完全可以在训练和推理端使用同一份模型代码，这就彻底杜绝了两套实现之间可能产生的细微行为偏差（Implementation Drift），从而让强化学习训练过程更加平稳顺畅。
Original English
**Harry Miller**: Uh you might be using it for RL, in which case you would be using traditionally uh potentially transformers for your training and then for rollouts you would use vLLM, uh both with different modeling implementations because previously that was what you had to do. Uh but now with this you can use the same code for training and inference during your RL loops, meaning that there's no potential drift in the two implementations and the training should go a little better.
**Harry Miller**：第三个原因非常纯粹：这可能是你的模型架构在 vLLM 中获得官方支持的唯一方式。截至目前，已经有 14 种模型架构仅通过 Transformers 建模后端获得 vLLM 的官方正式支持，而且随着时间推移，这个支持名单还在持续不断地扩充。
Original English
**Harry Miller**: And then finally, it might just be the only way that your model is supported in vLLM. So today there are 14 architectures which only which are officially supported via the transformers modeling back end. Um and that list is only growing as time goes on.
### 引擎初始化与多维并行配置实战
**Harry Miller**：现在，我们开始深入技术细节，一起看看我们是如何介入 Transformers 建模后端的，以及它在底层究竟做了哪些事情。毫无疑问，首先第一步是启动一个 vLLM 引擎实例。
Original English
**Harry Miller**: So now we start getting into the into the weeds and talking about how we get into the transformers modeling back end and what it does. Uh so obviously first we start with starting up a vLLM engine.
**Harry Miller**：在这里大家可以看到一长串命令行参数。其中一个关键参数就是强制指定使用 Transformers 建模后端。之所以要显式指定，是因为我们演示所采用的这个 Qwen2.5-VL 多模态模型，在 vLLM 中本身已经存在一套专属的原生手写实现；因此我们必须明确告知 vLLM，我们这次强制要求走 Transformers 后端链路。
Original English
**Harry Miller**: Um and here we have a whole load of command line arguments. One of them is to force the transformers modeling back end because for this uh Qwen2.5-VL model that we're looking at uh there is a dedicated vLLM implementation. So we have to tell vLLM that transformers is what we want.
**Harry Miller**：接着，我们配置了一套虽然不常规推荐但完全可行的复合并行策略：我们在 8 张 GPU 组成的集群上，尽可能拉满了各种维度的并行能力，同时叠加开启了张量并行（Tensor Parallelism）、流水线并行（Pipeline Parallelism）、数据并行（Data Parallelism）以及专家并行（Expert Parallelism）。
Original English
**Harry Miller**: Um and then we've used a not recommended but possible uh parallelism scheme where we're using as many as we can across eight GPUs. We've got tensor, pipeline, data, and expert parallel all at once.
**Harry Miller**：最后，我们对多模态编码器（Multimodal Encoder）启用了 `torch.compile` 图编译优化。并不是所有的 vLLM 模型默认都会对多模态编码器进行编译，但得益于 Transformers 建模后端的通用性设计，只要原生 Transformers 代码中的多模态编码器满足可编译条件，你就可以直接打开这个编译开关；一旦模型能够顺利编译通过，它就能在 vLLM 环境下顺畅运作。
Original English
**Harry Miller**: Um and then finally we've enabled torch compile on the multimodel encoder, which not all um vLLM models will do by default, but because of the generality of the transformers modeling back end. If the transformers code uh has a compilable multimodel encoder, you can just enable this flag and if your model works it should also work via vLLM.
### 配置解析与后端分发类的动态路由
**Harry Miller**：在完成引擎启动配置后，我们从 vLLM 跨入 Transformers 体系的第一步，就是深入检查模型配置文件（Config）。在这里，系统需要解析：当前到底是一个什么模型、模型各维度的具体尺寸与拓扑结构是怎样的，以及模型具备哪些特性（例如采用的是何种量化方案等）。
Original English
**Harry Miller**: So we first cross the boundary into transformers from vLLM by looking into the config where we're looking at what is this model, what shape is it and then any other features like what quantization scheme is it using.
**Harry Miller**：一旦系统确认选用 Transformers 建模后端并全面解析了配置文件内部的信息，接下来就必须决定具体实例化哪一个 Transformers 建模后端封装类，来加载并承载该 Transformers 模型。大家可以看右侧屏幕，上面列出了当前可选的一系列后端类定义。
Original English
**Harry Miller**: And we then once we've chosen the transformers modeling back end had a look inside the config we need to choose which transformers modeling backend class to load our transformers model into. Uh and on the right you can see the selection of classes available.
**Harry Miller**：系统在选择这些类时的判定逻辑是：仔细审查配置文件中的各项特征标识，确定模型具备哪些核心架构属性，进而精准路由到最匹配的目标类。如大家所见，所有这些类的命名开头全部统一为 `Transformers`。
Original English
**Harry Miller**: Um and the way that we select them is by looking into the config and establishing which characteristics the model has which will determine the correct class to use. So first they all start with transformers as you can see.
**Harry Miller**：接着，系统检测到当前模型是一个多模态模型，因为在前一页幻灯片展示的 JSON 配置文件中，它同时包含了文本配置（Text Config）和视觉配置（Vision Config）；同时，因为模型的专家数量配置大于零，判定其包含 MoE 专家结构；最后，若未命中特殊分支，则通用的因果语言模型基类 `ForCausalLM` 将作为基础默认兜底类……
Original English
**Harry Miller**: Um and then we detect this one is multimodal because it has both a text and a vision config uh in that uh config JSON you saw on the previous slide. It's because it has more than zero experts and then ForCausalLM is the default uh if...
### 基于 Mixin 的多模态模型类架构设计
**演讲者**：你的模型是一个因果语言模型（Causal Model），或者你并未特别指定要对该因果模型执行池化（Pooling）任务，因此我们选择了 Transformers 的 MultimodalForCausalLM 类。那么，这些类究竟是由什么构成的呢？因为这里涵盖的模型类非常之多，你可能会认为，如果我们不采用某种巧妙的设计，最终代码中就会出现大量的重复代码与逻辑漂移，导致维护变得极为困难。但实际上，所有这些类都是由各个 Mixin（混入类）构建而成的，由它们来分别启用各项独立的能力。
Original English
**Speaker**: your model is a causal model or if you don't specify that you want to do a pooling task with your causal model. And so we choose the transformers multimodal for cos lm class. Uh and what are these classes made of? Because there were quite a lot of them and you'd think that if we didn't do anything clever, we would end up with lots of duplicated code um and lots of drift and it would be really hard to maintain. Um but all of these are made of mixins which enable each individual uh capability.
**演讲者**：具体到当前这个模型，我们首先从 MoE Mixin 开始，它提供了专家并行（Expert Parallel）以及专家并行负载均衡所需的全部必要协议。接下来是 Multimodal Mixin，它负责处理所有的视觉编码、视觉编码缓存，以及在模型具备时的 M-RoPE（多模态旋转位置编码，当前模型确实具备）；随后，在将数据传递给 Transformers 骨干网络之前，它还负责将视觉嵌入与文本嵌入进行融合。再往后是 Causal Mixin，用于处理 LM Head（语言模型输出头）和 Logits 的计算。最后是最底层的基类（Base Class），它承担了绝大部分繁重的工作——实际上，上述所有的 Mixin 现在都继承自该基类，它位于整个继承层级栈的最底层。该基类负责权重加载、权重映射，以及为了让模型能够以与 vLLM 原生模型完全相同的极高速度运行所需实施的各项优化改造。此外，这里还提供了大量形式多样的协议，用以支持诸如量化、LoRA、流水线并行、Eagle 以及 Eagle 3 等高级特性。
Original English
**Speaker**: So for this one we first start with the MOE mixin which provides uh all of the necessary protocols for expert parallel and expert parallel load balancing. Then we have the multimodal mixin which handles all of the vision encoding the encoding the vision encoding caching um mrop if your model has it which this one does. Um and then merging the vision embeddings into the text embeddings before passing it over to transformers. Then the causal mixin which is for the lm head and the logics computation and then finally the base class which does most of the heavy lifting and all of the mixins actually now uh inherit this and it ends up at the bottom of the inheritance stack here. Um and that handles the the weight loading, the weight mapping, um all of the modifications that enable this to run quickly um just as fast as a BLM model would. Um and then there is a whole host of supports this supports that uh protocols for things like quantization, Laura, pipeline parallel, eagle and eagle 3.
### 模型实例化与注意力机制 Shim 注册
**演讲者**：现在我们来看一下 Transformers 类是如何被实例化的，以及我们在其实例化过程中对它做了哪些处理。屏幕上的图示现在看起来颜色比较淡，但这只是因为它当时尚未被实例化。在正式执行实例化之前，我们实际上首先向 Transformers 的注意力机制注册表（Attention Registry）注册了两个注意力机制的适配层（Shims），即 vLLM 的 Attention 前向传播方法与 vLLM 的 MLA（Multi-head Latent Attention）Attention 前向传播方法。随后，我们对配置（Config）进行打补丁，使得当 Transformers 模型执行前向传播时，系统会根据模型的具体类型选出并调用这两个 Shim 之一，而不是去使用 FlashAttention、FlexAttention 或该模型声称支持的其他原生注意力选项。
Original English
**Speaker**: So now we're going to look at how the transformers class is instantiated and what we do to it uh during instantiation. So it all looks a bit faint now, but that's because it has not yet been instantiated. Um first we actually before doing any of that we register two attention shins with transformers attention registry. Um and that is the VLM attention and VLM MLA attention forward methods. Um then we patch the config so that when the forward pass of the transformers model happens uh one of those shims depending on which is correct for your model is what is actually selected rather than uh flash attention flex attention or or whichever other options the model uh claims that it can use.
**演讲者**：接着，从语言模型的角度来看，我们始终会对该语言模型添加用于 torch.compile 的装饰器。此外，由于我在前面提到过的那个命令行参数，我们同样也对视觉塔（Vision Tower）进行了装饰。这实际上是在告知 vLLM：在 torch.compile 有机会自行揣摩并盲目推断之前，你可以将这两个部分作为完整的计算图彼此独立地进行整体图编译。随后，我们在 Meta 设备上通过 `AutoModel.from_config` 方法来实例化该模型。需要指出的是，`AutoModel` 与大家在 Transformers 生态中更为熟悉的 `AutoModelForCausalLM` 存在细微差别；`AutoModel` 仅加载 Transformers 所谓的基类，即只包含语言骨干网络和视觉塔，而不包含语言建模头（Language Modeling Head）。
Original English
**Speaker**: Then we decorate uh the language model always for torch compile from the LM's perspective. Um, and then because of that command line argument I mentioned earlier, we also decorate the vision tower. Um, which tells VLM that you can compile both of these as whole graphs independently of each other before uh before torch compile gets a chance to kind of wing it and figure out on its own. And now we instantiate the model uh using the automodel from config method on the meta device. Um and for automodel is slightly different to um auto mode for cos lm which might be a class you're more familiar with if you use the transformers ecosystem. Uh and the auto model just loads what transformers refers to as the base class which is just the language backbone and the vision tower but without the language modeling head.
### 权重映射器与流水线阶段修剪
**演讲者**：我可能需要退回到笔记本电脑旁操作一下，演示设备似乎停止响应了。好，接下来我们创建 Hugging Face 到 vLLM 的权重映射器（HF to vLLM Mapper），它用于详细定义模型检查点中的权重应当如何映射转换到新构建的 Python 类结构中。如果 Transformers 模型已经从旧式的朴素 MoE 演进为融合 MoE（Fused Mixture of Experts）风格，其内部通常自带了一些此类映射。目前这个阶段该映射器还相对简陋，但随着后续流程推进以及我们对模型的持续修改，我们会逐步向其中补充映射项。
Original English
**Speaker**: I might have to step back to the laptop. This appears to have stopped working. Um, then we create the HF to VLLM mapper which is how we specify how checkpoint weights might be translated into how they now look in the Python class that gets created. uh Transformers has a few of these uh built in if they have moved from an old style naive mixture of experts to a fused mixture of experts style. Um so at this stage this is a pretty minimal mapper but as we move through this process we'll be contributing to it as we modify the model.
**演讲者**：完成这一步后，我们接着删除当前流水线并行阶段（Pipeline Stage）未使用的所有网络层。在本示例中，我们当前处于 Rank 0，因此我们直接删除了后半部分的层堆栈以及最终的 LayerNorm 层。紧接着，我们便正式开始执行递归替换（Recursive Replace）——这是实现性能飞跃并接入 vLLM 全部原生特性的核心关键所在。
Original English
**Speaker**: So after that we then delete all of the layers that the current pipeline stage is not using. And in this example, we're on rank zero. So we delete the second half of the layer stack and the final layer norm. And now we begin recursive replace which is where the bulk of the work happens to bring the performance and access to all the features that BLM has.
### 递归替换第一阶段：MoE 模块的整体替换
**演讲者**：对于 MoE 模型而言，递归替换分为两个阶段进行。因为 MoE 结构比较特殊，我们有两种不同的处理方式。针对当前这款模型，它采用的是内部路由方法（Internal Routing Method）。这意味着我们会借助 Torch FX 来自动检测稀疏 MoE 模块的图结构；一旦我们判定该结构完全可以由 vLLM 的 MoE Runner 进行等价表达，我们就会对其进行整体替换。具体而言，我们将所有的专家网络替换为这个 MoE Runner，其中封装了路由后的各个专家；而原本的路由门控则被替换为一个 Gate Linear 模块并传入该 Runner 中，由 Runner 统管全部计算流程。如此一来，我们便能针对路由与专家计算调用单体式（Monolithic）的高性能融合算子；假若该模型中存在共享专家（Shared Experts），它还支持执行异步的共享专家计算，所有这些均由 MoE Runner 进行统一管理。
Original English
**Speaker**: Um fore models it happens in two passes because thee is a little bit special and there are two different ways which we can do it. Uh for this model it uses the internal routting method which means that we by using torch FX uh detect the structure of the sparsee block. Um and if we determine that it is entirely representable by VLM's runner um then we replace it wholesale. So we replace the experts with this MOE runner. Um and that contains the rooted experts and then the routter gets replaced with a gate linear and that gets passed into the runner. Um and that handles everything. And so that lets us do uh things like use monolithic kernels for routing and experts. Uh if there were shared experts in this model, it would allow us to do asynchronous shared expert computation. Um and all of that is managed by thee runner.
### 递归替换第二阶段：子模块的深度优先搜索与算子替换
**演讲者**：第一阶段完成后，我们进入第二阶段：由基类对基础模型的所有子模块执行深度优先搜索（DFS）遍历。我们首先遍历视觉模型，将所有已知可替换的层替换为 vLLM 的高性能层以获取相应特性。vLLM 所有的并行化能力都是直接内建在这些层中的；通过改用 vLLM 的 Linear 层和专家层，我们便顺理成章地获得了张量并行（Tensor Parallel）和专家并行等能力。接下来，我们通过精准替换 `torch.nn.Embedding` 模块来完成词表并行嵌入（Vocab Parallel Embedding）的置换。这样一来，像 Gemma 这类在 Token 嵌入期间需要执行数值缩放的模型，其原有缩放逻辑便能原汁原味地得到保留，我们无需在 vLLM 端专门为其编写特殊分支逻辑。
Original English
**Speaker**: And then once that is done, we go to the second pass which is the base class doing a depth first search through all of the children of the base model. So first we walk through the vision model replacing all of the layers that we know how. Um to its features. So all of VLM's parallelization is built into its layers. So by using VLM's linear layers and VLM's experts layers, that's how we gain access to tensor parallel expert parallel etc. Then we do the vocab parallel embedding swap um by surgically replacing just the torch nn embeddings module. So if you have a model like Gemma where you do some scaling during your token embedding um that is all still preserved and we don't have to special case it on the VLM side.
**演讲者**：紧接着应用的是下一个融合器——QKV 融合器。它同样利用 Torch FX 检测出 QKV 投影层以及输出投影层，并将其替换为 vLLM 的 QKV 并行线性层以及相匹配的行并行（Row Parallel）线性层。需要说明的是，此处是利用 Torch FX 来精准检测需要替换的目标，随后利用 Python 的模块替换机制完成实际修改。也就是说，目前我们尚未执行任何 torch.compile 编译，所有的编译优化都留待 vLLM 的后续阶段处理。
Original English
**Speaker**: Now the next fuser uh is used which is the QKV fuser which again uses um FX to detect the QKV and output projections um and swap in BLM's QKV parallel linear and a corresponding row parallel linear um using a so so this is all it's using torch FX to detect what we need to swap but then it's using Python's as module to actually make the modifications. So we are yet to torch compile anything and all of that is still left up to later stages in BLM.
**演讲者**：类似地，我们还替换了 RMSNorm。尽管 RMSNorm 本身看似简单，但替换它的难度却出人意料地高。我们之所以必须替换它，是因为 vLLM 内部依赖一些全局的 torch.compile Pass 来检测能够发生算子融合的位置，而这些 Pass 是通过识别 vLLM 自身的特定模块来触发的。因此，部分涉及 RMSNorm 的编译融合 Pass 只有在我们使用 vLLM 原生 RMSNorm 类时才能被正确识别与触发。
Original English
**Speaker**: Then similarly we replace the RMS norm which is deceptively difficult for how simple uh it is but we do it because VLM uses um some global torch compile passes to detect where it can fuse layers and it does it by detecting its own modules. Um and so there are some um RMS norm passes or some compilation passes that the RMS norm is involved in that will only be detected if we're using VLM's class.
**演讲者**：接下来，我们创建 vLLM 的 Attention 实例。正如前面所提到的，这一步使我们能够全面接入 vLLM 的剩余系统能力，其中最关键的就是获取了 vLLM 的 KV Cache 控制权。这意味着我们完全不需要使用 Transformers 内部实现的任何 KV Cache 机制，vLLM 可以在 KV Cache 上施展它所有的拿手绝活，例如缓存卸载（Offloading）、跨节点的解耦 Prefill（Disaggregated Prefill）、KV Cache 复用等一系列高级特性。随后，对于任何我们尚未触碰的模块，我们确保它们均已正确放置在目标设备上；在视觉模型中，发生这一变动的仅有那两个 Norm 层。最后，我们做一次全局检查，确保所有 Transformers 坚决要求使用 FP32 精度的部分始终保持为 FP32，哪怕模型其余主体部分已经转为了 FP8。
Original English
**Speaker**: Then we create VLM's attention instances and as before this gives us access to the rest of VLM. Uh and the key thing that this gives us access to is VLM's KB cache. Um so we don't need to use any KB caching that has been implemented on transformers. Um, and VLM gets to do all of the clever stuff that it can do with the KB cache like offloading, um, disagregated prefill between nodes, um, KB cache reuse or all of that fun stuff. Then anything that we haven't touched yet, we make sure that it is on the device. Uh, so the only thing that changed there was those two norms in the vision model. And then lastly, we just make sure that anything that Transformers insists should be in FP32 stays in FP32 even if the rest of the model is FP8.
### 前向传播执行与 vLLM 回调接缝
**演讲者**：至此，整个模型已经构建完毕。当系统接收到推理请求后，该请求经由 vLLM 完成调度，并最终到达基类的 forward 方法。这里正是模型前向传播首次切回 Transformers 的关键节点。在此处，我们直接调用原始 Qwen2-VL-MoE 模型类的 forward 方法，并传入 `input_ids` 或 `inputs_embeds`。在本例中传入的是 `inputs_embeds`，因为这是一个多模态模型，vLLM 会预先对 Prompt 中的视觉部分进行特征嵌入，将其插入到文本嵌入中，并一次性打包传入，而不是交由模型内部去自行处理。与此同时，我们显式禁用了 Transformers 自身的 KV Cache，因为 vLLM 已经通过那些 Attention 模块完全接管了 KV Cache。随后，我们正常向前传递位置编码（Position IDs），针对 M-RoPE 模型在此处会有轻微的调整，这一调整在该特定类型模型的 Multimodal Mixin 中完成。在遍历 Transformers 模型的前向传播过程中，存在两个关键的性能敏感接缝会重新回调至 vLLM。其中第一个也是最引人关注的一个，就是 Attention 模块。在 Transformers 这一侧，我们再次从 Transformers 的原生类开始切入……
Original English
**Speaker**: So we've built the model um and now a request has been received. We have had it scheduled through VLN and we've reached the base classes forward method. Um and this is the first hop back into transformers. Uh so we simply call the forward method on the original Quen 3VLOE model class um and pass in the input ids or the input embeds um in this case it's the inputs embeds because it's a multimodal model and VLM pre-mbbeds the vision part uh of the prompt and inserts it into the text embeddings and passes that all in one go rather than relying on the model itself to to do that we explicitly disable transformers is um KV cache because VM owns it through those attention modules. Um and then we just forward through the position ids with a slight modification for MRO models um which happens in the multimodal mixin for this particular type of model. And so while we're going through the forward pass of the transformers model, there are two seams where we return back to VLM um in kind of performance critical ways. Uh and so the first and most interesting one is the attention module. Um so on the transformers side, we're starting again in the the transformers native class. It goes
### vLLM 注意力机制垫片与执行流衔接
**Speaker A**：通过注册表机制（registry）来选择具体使用哪种注意力方法。我们此前已经预先将其注册为 vLLM 注意力垫片（VLM attention shim），随后通过修改配置（config）选中了该实现。接下来我们进入该模块，并顺畅过渡切换回 vLLM 端。在 vLLM 内部，系统会进行一些轻微的张量形状变换与调整，随后获取并决定要调用的注意力计算内核（attention kernel）——只要是当前硬件设备针对该特定模型所支持的任何注意力内核均可。在 vLLM 体系内，它借助注意力上下文（attention context）获取对应的 KV 缓存（KV cache），紧接着真正发起底层的注意力计算调用，并将计算结果无缝传递回 Transformers 框架，以便继续执行该模型后续剩余部分的计算逻辑。
Original English
**Speaker A**: Through its registry to choose which attention method to use, which we pre-registered earlier as the VLM attention shim and then selected it by modifying the config. And then we enter that and transition back to VLM, where VLM does some minor shape manipulation, gets the attention kernel to use—which is any attention kernel supported by your device for this particular model. In VLM, it gets the KV cache using the attention context, and then it actually calls the attention and passes the result back to Transformers to continue with the rest of the model.
### MoE 内部路由机制与全块替换
**Speaker A**：第二种集成途径则是通过内部路由（internal routing）来处理 MoE 架构。正如我之前所提到的，MoE 存在一种特殊处理场景：如果我们能够成功检测出一个完全可在底层表达呈现的稀疏 MoE 模块（sparse MoE block），那么我们就可以将其整体彻底替换掉。实际上，我们此时调用的前向传播方法就纯粹是专家前向传播（experts forward），所有计算都封装在那个专门的运行器类（runner class）内部发生。因此，我们直接切入到 vLLM 端：先对隐藏状态执行门控操作（gating），获取 top-k 专家的索引编号（IDs）以及对应的路由概率得分（logits）；随后执行路由专家层面的密集计算，最后将计算输出结果重新交回给 Transformers 框架。
Original English
**Speaker A**: And the second is the MoE through internal routing. So as I mentioned, MoE has a special case where if we're able to detect an entirely representable sparse MoE block, then we can replace it entirely. And actually the forward method that we use is just experts forward, and everything happens inside that runner class. So we go straight into the VLM side. We gate the hidden states. We get the top-K IDs and logits, perform the routed experts computation, and pass the results back to Transformers.
### 计算图遍历收敛与输出采样交付
**Speaker A**：走到这一步之后，Transformers 模型的任务实际上就已经全部完成了。它已经提供了关于该模型计算图（graph）构成的一切完备信息，以及整个计算流转历程中的全部拓扑结构与指引；更确切地说，它引领完成了整趟计算之旅，因为正如我们所见，底层挂载的具体计算模块本质上全都是 vLLM 的高性能模块。在此之后，最终的 logits 计算以及语言模型输出头（LM head）的评估推理，全部都交由 vLLM 接管执行，并且后续的 token 采样流程（sampling）也一并由 vLLM 全权负责完成。
Original English
**Speaker A**: And after this point, the Transformers model is done. It has provided all of the information about what the graph of the model is and all the journey through the computation rather, because as we saw, the modules are all VLM modules. And then after that, the logits computation and LM head evaluation is all done by VLM, and the sampling is also handled by VLM.
### 外部路由与共享专家的扩展支持
**Speaker A**：对于当前这个具体模型，还有若干我们尚未启用的高级特性，在此我也想逐一阐述说明。首先是外部路由机制（external routing）。如果你的 MoE 模块结构与我们预设匹配的标准架构存在差异，这绝不意味着它无法运行；它仅仅代表我们只会针对性地替换底层专家计算单元本身。由于绝大部分密集算力开销都集中在专家层，这种替换依然能为你带来至关重要的性能加速增益；与此同时，它又保留了充分的工程灵活性，允许你在门控路由逻辑或其周边配套机制中自由实现任何自定义的创新设计。
Original English
**Speaker A**: So there are a few features that we haven't used with this model that I want to talk about. The first is external routing. If your block isn't the same as something that we've anticipated, that doesn't mean that it won't work. It just means that we will only replace the experts themselves. So that gives you the crucial performance because that's where most of the work is done, but it gives you the flexibility to do whatever you might come up with in the routing or anything around it.
**Speaker A**：另外还有共享专家机制（shared experts），这也是我前面隐约提及过的特性。如果模型中存在共享专家，并且在内部路由解析阶段被成功识别出来，它们同样会被直接纳入到 MoE 运行器（MOE runner）当中。根据你在 vLLM 中所配置的具体参数策略，它的计算方式将会比采用外部路由来得更加紧凑高效。
Original English
**Speaker A**: And there's also shared experts, which I alluded to earlier. If they had existed and they were detected in the internal routing, they would also have been included in the MOE runner. And depending on your configuration of VLM, it would be computed in a way that is more efficient than could be done with external routing.
### 微调适配器、投机采样与多元注意力形态
**Speaker A**：接下来是 LoRA 以及 Eagle 和 Eagle 3 等投机解码机制。我们今天之所以没有深入展开探讨，是因为这些能力主要取决于你所加载的具体模型检查点权重（checkpoint），以及你运行时所传入的请求参数指令。此外，系统还支持极其丰富的注意力变体：例如适用于 BERT 等纯编码器架构的纯编码器注意力（encoder-only attention），或者是你经常在 Gemma 系列模型中见到的滑动窗口注意力（sliding window attention）。针对编码器结构，我们支持纯编码器模型，但目前不支持编码器-解码器混合架构；至于滑动窗口注意力，它可以支持交错排布，正如 Gemma 最先采用的方案那样，在全局全量注意力与局部滑动窗口注意力之间交替轮转生效。
Original English
**Speaker A**: Then there's LoRA and Eagle and Eagle 3, which we just didn't touch on because these are kind of determined by the checkpoint you use and then the requests that you pass. Then there's a whole load of different attention flavors. So there's encoder-only attention which would be used by models like BERT, or sliding window attention that you might find in Gemma models. And for encoder, we support encoder-only models, not encoder-decoder models; but for sliding window attention, it can be interleaved, similar to how they started doing it with Gemma where you would alternate between full and sliding attention.
**Speaker A**：同时还有 MLA 注意力（多头潜在注意力，Multi-Head Latent Attention）。它是通过一个独立的融合算子（fuser）接入系统的，能够切实带来宝贵的 KV 缓存压缩优势；而在过去，当我们加载 MLA 注意力模型时，往往只能退化地将其简单填充对齐为全量标准注意力，无法享受到显存压缩红利。最后是池化（pooling）与分类任务支持：如果我们加载的是一个标准的池化表征模型，该逻辑就会被系统自动激活调用；我们也可以通过手动显式指定，此时 vLLM 可以基于原有的语言模型输出头（LM head）自动构建起对应的分类头，或者如果你需要直接提取特征向量，它也能直接在模型末端为你返回纯粹的原始嵌入向量（raw embeddings）。
Original English
**Speaker A**: And then also MLA attention, which comes through a separate fuser, and that actually gives us the KV cache compression benefits that we didn't use to get when we just used to pad to full attention when you loaded a MLA attention model. And then finally pooling and classification tasks, which would have been automatically used had we used a pooling model, or we could have manually selected it and VLM would have created a classification head using the LM head, or it could have given you back the raw embeddings out the end of the model if that is what you were after.
### 架构设计总结与全场致谢
**Speaker A**：总括回顾一下我们今天梳理的全流程：首先，我们深入剖析了模型的配置信息（config），精确判断出它隶属于哪种 Hugging Face 架构，并进而选定应当采用哪一个 Transformers 建模后端类。随后，我们将其无缝载入到 vLLM 内部通用的通用类中，并对其底层结构施加精巧的操作与重构，从而彻底解锁并激活了 vLLM 所具备的全部前沿进阶特性，连同其背后所附带的极致推理性能表现也一并尽数释放。紧接着，我们跟踪了基座模型的前向传播执行链路，重点辨析了两个至关重要的无缝架构衔接点（seams）：在这两处节点上，计算流重新切回 vLLM 内部以执行一系列非平凡的关键复杂计算；此后整体流程再次闭环交回给 vLLM，以完成终末阶段的 logits 映射计算与 token 采样生成。以上就是我今天分享的全部内容，讲座幻灯片已经附在屏幕上的二维码当中，如果现场有哪位朋友想要再次回顾查阅，可以随时扫码获取。非常感谢大家！
Original English
**Speaker A**: And so to recap, we looked into the config, determined which Hugging Face architecture it was, and then which Transformers modeling backend class we should use. We loaded it into this generic class in VLM and we manipulated it such that all of the advanced features of VLM became accessible, and all of the performance that comes with them came with them. Then we went through the forward pass of the base model with two important seams, where we go back into VLM and do some non-trivial work, and then we pass back to VLM to do the logits computation and the sampling. And that was my talk, and the slides are available on the QR code there if anyone wants to look at them again. Thank you.
### 中场休息与下半场议程开场
**主持人**：抱歉，现场设备难免出点小插曲。好了，我想我们现在进入中场休息环节了。非常感谢各位的光临！我们稍作休息，大约半小时后重新开始，期待与大家再次相聚。好的，非常感谢大家！
Original English
**Host**: Sorry. That's what happens. All right, I think we're on a break now. Thank you so much everybody. So, I think we were going to take a little break and we're back in half an hour, I believe. All right, thank you so much.
**主持人**：大家好，欢迎回到活动现场！好了，让我们继续为各位带来精彩内容。好的，到目前为止，我们已经一起探讨了诸如 AI 经济图景、代码自动化开发框架（harnesses）、软件工程工厂（software factories），以及生成式多媒体内容等核心主题；所有这些前沿技术与工具，归根结底都是为了赋能现场的各位开发者去构建自主智能体（agents），打造各类基于人工智能的创新应用程序。在座的各位当中，目前真正正在动手使用 AI 进行一线产品构建的请举手？好的，看来绝大多数朋友都已经在积极实践了，对吧？而我们做这些事情的核心初衷，正是为了切实给广大终端用户创造实际价值。无论你正在打磨的是哪种类型的智能体，无论你在开发的是何种工具，终极目标都是为你的用户带来真正的价值红利。
Original English
**Host**: Hello, welcome back. All right, let's get it for you guys. Okay. So so far we have spoken about, you know, the AI economy—like we talked about harnesses, software factories, we talked about generative media, and all those tools are for you guys to build agents, to build applications that use AI. Who here built actually with AI? Okay. Yeah, most of you guys do, right? And the purpose of that is to create value for people, right? Whatever agent you're building, whatever tool you're making, ultimately is to bring value to your users.
**主持人**：我们的下一位演讲嘉宾，其实一直深耕并沉浸在最真实、最具说服力的底层支付交易数据洞察之中。她来自全球领先的金融基础设施平台 Stripe；在 Stripe 团队的长期观察视角中，全球整个宏观经济体系正在因人工智能的深度介入而发生天翻地覆的根本变革。今天，她亲临现场，就是为了向大家全方位分享与揭秘 Stripe 独家海量交易数据背后所反映出的 AI 行业演变洞察与商业趋势。话不多说，请大家用最热烈的掌声，有请 Stripe 人工智能与初创企业产品负责人（Head of Product of AI and Startups）——Ariel Lubai 登台！
Original English
**Host**: Our next speaker is actually deep into the data, and she works at Stripe. And what they see is that the entire economy is changing thanks to AI, and she's going to come to tell us about and give us some insights about what Stripe's data is revealing about AI. So without further ado, please join me in welcoming to the stage the Head of Product of AI and Startups, Ariel Lubai.
### Stripe 视角下的 AI 商业新范式与增长全景
**Ariel Lubai**：大家好！谢谢大家，非常感谢各位能来到今天的分享现场。能来到这里和大家面对面交流，我感到无比荣幸和高兴。我希望我的演讲 PPT 稍后能顺利投放在大屏幕上。控制台的伙伴们，可以帮我把幻灯片切出来吗？好的，太完美了！好的，我们切回到正题。太棒了，非常高兴能来到这里。在 Stripe，我们每天都在与全球范围内增速最迅猛的 AI 头部企业深度合作；我们非常荣幸能拥有这样得天独厚的合作生态，而事实上，他们当中的许多团队核心成员今天就坐在我们这个会议室里。因此，能在此与大家相聚我格外激动。我们仿佛坐拥全局视野的最佳第一排看台，清晰见证着哪些探索正在行之有效地发挥作用、哪些做法陷入了困境，以及最顶尖的企业究竟是如何在这个全新的 AI 时代浪潮中实现规模化指数级扩张的。
Original English
**Ariel Lubai**: Hi everyone. Thank you. Thank you so much for being here. I'm super happy to be here and I hope I will get some slides at some point on the screen. Can I get the slides in the, please? Okay, perfect. Okay, let's come back there. Okay, perfect. So yeah, really happy to be here. So at Stripe we work with the fastest AI growing companies. We have this luck and many of them are actually in this room. So really happy to be here and we have a front seat of what's working, what's not, and like how the best companies scale today in this new world of AI.
**Ariel Lubai**：今天，我希望把这些珍贵的实战洞察与大家开诚布公地交流，并和各位共同剖析当下最新的商业落地法则（playbook）——因为在过去几年甚至短短数月之间，这套商业与增长战法已经被彻底颠覆并全面重写了。首先，让我们先来看一组直观的数据概览。目前全球约有 1.6% 的 GDP 总量流经 Stripe 的支付基础设施完成结算处理；正是这样宏大的宏观资金吞吐规模，让我们能够稳稳站在商业前沿的最前排，系统性地梳理出五大正在深刻发生的标志性模式与发展趋势。
Original English
**Ariel Lubai**: So today I want to share some of those insights with you guys and, you know, have some thoughts on the playbook, because this has been completely rewritten in the past few years and even months. So first, let's start with figures. First, we are working with 1.6% of the GDP globally that is processed by Stripe, and that is really something that puts us in that front-row seat at the five patterns that we're seeing.
### 头部 AI 企业的五大核心演进模式
**Ariel Lubai**：第一，顶尖的 AI 企业正在以超乎以往任何时代的速度进行产品构建与迭代，我想在座的每一位同仁对此都有切身体会。第二，它们从诞生的第一天起，就天然具备“默认全球化售卖”（sell globally by default）的基因：当你在今天着手构建一款 AI 产品时，从产品上线的 Day 1 开始，你的目标市场就不再是某个特定地域，而是整座全球市场。第三，商业定价策略（pricing）正在经历全方位的根本性重构；顺便提一句，今天大会现场安排了许多场专门探讨定价机制的专题工作坊，这是因为过去两三年里，科技产品的商业化定价逻辑已经被彻底改写了。
Original English
**Ariel Lubai**: When the top AI companies build faster, I think this is something we're all feeling. Two, they sell globally by default. When you're building today, your day one market is now the whole world. Three, pricing is evolving completely. Like there are a lot of workshops on pricing today by the way, because how you price has completely changed in the past few years.
**Ariel Lubai**：第四，它们在主动大刀阔斧地重塑其市场拓展策略（GTM / go-to-market motions）。比如渠道分销与生态销售（channel sales）在当下的商业版图中正占据着越来越举足轻重的位置，稍后我们还会就此展开更深度的剖析。第五，这些团队极其注重尽早且高频地严密监控黑灰产与欺诈风险（fraud）。为什么必须如此重视欺诈防范？因为那些借助自主智能体能够带来指数级爆发增长的裂变机制与底层机制，在客观上也恰恰为针对你企业业务的各类恶意欺诈攻击提供了一片全新的温床。稍后我们会对这些维度进行细致入微的逐一拆解；不过在此之前，不妨先让我们把视野拉远，纵观全球大盘，审视一下当今世界正在发生哪些令人震撼的宏观宏大趋势。
Original English
**Ariel Lubai**: And four, they adapt their go-to-market motions as well. We have like channel sales that are becoming increasingly important; we'll dig into that afterwards. And five, they monitor fraud early and often. Why? Is like what's creating potentially viral growth with agents is also creating just a whole new ground for fraud attacks for your business. So we'll dig into it of course, but first let's look at big picture what's happening in the world on that.
### 指数级爆发：从企业级神话到消费级普及
**Ariel Lubai**：我们从底层数据中观察到，在 2025 年，顶尖的头部 AI 企业实现了平均高达 145% 的惊人年增长率，这在以往已经是非常震撼的成就；然而更不可思议的是，到了 2026 年，这一增长数字直接狂飙跃升到了 195%——这意味着整个头部梯队的商业规模在短短一年时间内翻了整整三倍！而且尤为关键的是，这种疯狂的增长势头不仅丝毫没有放缓的迹象，反而正在当下的每一天里进一步加速冲刺。而大家耳熟能详的那些明星代表企业，其狂飙突进的发展轨迹更是令人瞠目结舌：比如 Lovable 团队，他们公布其业务仅用 8 个月时间便从零冲上了 1 亿美元 ARR 营收规模，而在仅仅又过了 8 个月之后，他们的营收体量就不可思议地膨胀到了 4 亿美元。至于大家都非常熟悉的 Anthropic，在 2023 年 1 月时他们的商业化收入还是零起点，仅用了短短两年时间便闪电冲破 10 亿美元大关，而如今，他们已经站在了高达 300 亿美元的超大规模台阶之上——这完全是一场颠覆行业认知的奇迹。
Original English
**Ariel Lubai**: We saw the top AI companies grow by 145% in 2025, which was already impressive, but in '26 this number grew by 195%. So tripling in a single year. That growth isn't slowing down, it's actually accelerating today. And the top ones that you know are even more mind-blowing of course: like Lovable, they reported 100 million in eight months; eight months later, they were actually at 400 million. Anthropic, that you all know, started in January '23 with zero and went to 1 billion in just two years, and now they're at 30 billion. So that's just completely mind-blowing.
**Ariel Lubai**：更为重要的是，这种爆发式的增长红利绝不仅仅局限于 B2B 企业服务领域。我们在 Stripe Link 的全网底层交易洞察数据中清晰地捕获到：C 端普通消费者的广泛采纳与普及度同样呈现出几何级的迅猛激增。仅仅在过去一年时间里，消费者普及规模便直接从最初的 600 万人翻倍暴涨至超过 1400 万人。不仅采纳基数呈倍数扩张，广大终端消费者的消费意愿与付费意愿也在大幅水涨船高。真正引人深思的核心现象在于，普通消费者在各类 AI 工具与服务上的实际支出正在显著攀升：现如今，个人用户在 AI 工具上的平均月度支出已经达到了大约 360 美元左右，与短短三个月之前的月均 180 美元相比，基本上实现了整整翻倍的惊人飞跃！
Original English
**Ariel Lubai**: And this isn't just B2B, like we saw this in our Link data: consumer adoption is also increasing a lot, like it doubled just from 6 million to over 14 million just in one year. Consumers are also spending more. What's actually interesting is that they're spending more on AI tools. Like today, we're around like $360 per person per month on AI, which is basically double what it was three months ago; it was around $180.
### AI 成为关键刚需与软件交付速度的重构
**Stripe 演讲者**：这基本上就相当于欧洲普通人每月在电话通讯、网络宽带以及流媒体服务上的开销总和。如今大家并没有把 AI 当成某种随时可以退订的流媒体订阅服务，而是将其视为一项绝对不可或缺的关键公用基础设施（utility）。那么，这一切背后的驱动力是什么？我们将深入探讨这些不同的发展模式，首先从“速度”开始。正如过去你不再需要耗费数月的工程时间来编写底层代码以接入支付一样，如今你同样不再需要几个月的工程研发才能交付软件。非常有意思的是，这张图表清晰地展示了转折点是在何时出现的。如果观察 iOS 应用的发布趋势，在 2024 年底时实际上处于下滑态势；然而随着智能体编程（agentic coding）工具的爆发，我们看到 iOS 应用的发布量重新开始增长，而且绝非微幅上涨，而是实现了月环比 24% 的强劲增长。如今，随着我们将支付能力直接嵌入到 Lovable、Replit、Vercel 等开发者平台中，从产生创意到完成第一笔收费的速度变得前所未有地快。从这里的数据可以看到，从最初的构想到沙箱测试再到完成首次收费，实际耗时已经缩短至不到六周。仅仅六周时间，这确实不可思议，而对于技术人员来说，这一策略甚至更为迅捷。
Original English
**Stripe Speaker**: And this is basically what the average European spends for phone service, internet, and streaming services combined. People aren't treating AI today like some streaming service that you can cancel, but they're treating that like a critical utility that they absolutely need. So what's behind all this? We're going to deep dive into these different patterns, starting with speed. Just like you no longer need months of engineering to write code and accept payments, now you no longer need months of engineering to ship software. What's actually interesting is that this chart shows when it clicked. If you take a look at iOS app releases, they were actually declining at the end of '24, and then the agentic coding tools hit. And so this is where we saw the iOS app launches just grow again—and not like small, but grow by 24% month over month. And so now that we embed payments into developer platforms like Lovable, Replit, Vercel, etc., it's actually even faster to go from your idea to first charge. What you can see here is that it's actually taking less than six weeks from first idea and sandbox to your first charge. Like only six weeks. That's really crazy. And that strategy is faster for technical folks.
### Claude 与 Stripe MCP 集成实操演示
**Stripe 演讲者**：因此，我尝试使用提示词做了一些测试，向大家展示如今借助 Claude 和 Stripe 进行开发是怎样的体验。我们在智能体开发者体验（agentic developer experience）方面投入了大量工作，确保大家在构建应用的过程中无需离开当前的开发环境，也不必特意跳转到 Stripe 控制台去配置各项事务。假设你今天是一名开发者，已经构建好了一款应用，正准备接入支付系统。让我们直接进入 Claude 界面看看具体效果。这里是我的提示词，我提出要配置支付系统，需要发票开具（invoicing）和订阅管理（subscription）功能。正如你所看到的，Claude 非常准确地推荐了 Stripe，这正是我们期望看到的；同时由于 Claude 接入了 Stripe MCP 以及我们为此创建的 Skills（技能库），它的能力会更加强大。Claude 调用了 Stripe 集成推荐工具（Stripe integration recommender MCP tool）和这些技能，从而清楚该使用哪些模块以及应该向用户询问哪些问题。它首先询问：“您的客户主要分布在哪些地区？”之所以提出这个问题，是因为 Claude 明确知道只有掌握了这一信息，才能提供最契合业务场景的集成建议。值得关注的是，它推荐了 Checkout 这一 Stripe 集成方案，而不是直接推荐 Claude Elements 等组件，因为它识别出对于更完善的订阅集成体系而言，Checkout 是最佳选择。这再次证明它能够做出精准的技术决策。
Original English
**Stripe Speaker**: So I've played a bit with a prompt to show you how it looks like today to build with Claude and Stripe. Because we've made a lot of work to make sure that we have an agentic developer experience that enables you to build without having to leave where you're building and having to go on Stripe dashboard and build stuff. So let's say you are a developer today and that you've built an app and now you're ready to set up payments. Let's jump into Claude and see what it looks like. I have my prompt here where I've asked to set up payments—I want invoicing and subscription. Here you can see that Claude correctly recommends Stripe, which is of course what we want, and that they will be more powerful because they are connecting to Stripe MCP and the skills that we created with this. Claude uses the Stripe integration recommender MCP tool along with those skills to know what to use and what to ask to the user. So it's basically asking: "Where are your customers located?" The reason it's asking is because Claude knows now that they need to have this answer to provide the best recommendation for the integration. What's interesting is that they recommend Checkout, which is a Stripe integration, and not like Cloud Elements for example, because it knows that for a better subscription integration we need Checkout. So that's again showing that it knows the right thing.
**Stripe 演讲者**：随后我回复说：我的客户分布在美国和欧盟，其中在爱尔兰尤为集中。接下来很有意思的是，系统立即针对我所需了解的税务知识给出了反馈。它明确提示：数字订阅服务在当地属于应税业务。更棒的是，它直接建议我采用 Stripe Tax，因为我显然不希望为了拓展全球业务而被迫逐一研究每一个目标国家的地区性税法规则。Stripe Tax 能够在免除我理解这些繁琐税规的前提下自动处理好一切。紧接着我看到它正在为我配置一个 Stripe 沙箱环境。这里的绝妙之处在于，你完全无需跳出当前的工作流去手动获取沙箱；智能体直接调用了最新的 CLI 命令行工具来创建沙箱，全程无需事先配置任何 Stripe 凭据，而这在以往是做不到的。该命令还会自动拉取 API 密钥并安全保存。通过这种方式，我的开发心流完全不会被打断，无需离开编辑器就能顺利推进集成。
Original English
**Stripe Speaker**: So then I responded that my customers are split between the US and the EU, with a particular concentration in Ireland. What's interesting now is that I get feedback on the type of tax knowledge I need to have. Basically, I need to know that digital subscriptions are taxable here. And it's actually interesting because it recommends me to use Stripe Tax, because I don't want to have to know all the regional tax rules for all the countries that I'm going to launch to. And so Stripe Tax is going to help me without having to know those rules. I can see that it's provisioning a Stripe sandbox for me. What's actually interesting is that you don't have to go outside of the flow here to get a sandbox. The agent uses the new CLI command to create the sandbox without needing any Stripe credentials, which was not the case before. And the command also fetches the API keys and securely stores them. That way I basically don't have to break my flow and I can build my integration without having to go away.
**Stripe 演讲者**：在搞定密钥之后，我看到 Claude 正在校验由我们 Stripe 官方提供的各类技能（skills），以确保其构建方向完全正确。系统随后提示一切所需条件均已就绪，并征求我的确认：“您是否希望继续执行 MCP Stripe 创建产品（MCP Stripe create product）并生成 Checkout 结账流程？”我在此处点击确认。我们快进一下来看最终生成的结账页面。这就是实际呈现的效果。我来测试一下这个 Checkout 页面是否能正常运行——完全没有问题。到这里集成便全部完成了。Stripe 与 Claude 的结合，本质上让无论是你还是所有的开发者，都能够在无需离开当前开发界面的情况下，极其轻松地完成构建、配置支付并开启商业化变现。
Original English
**Stripe Speaker**: And so with the key sorted, I can see that now Claude is checking all the different skills that we at Stripe provide to make sure that it's building the right thing. So it's saying that it has everything it needs and it's asking me: "Do you want to proceed with MCP Stripe create product and continue with that and create my checkout?" So I'm pressing yes here, and we're going to fast forward and try to see the checkout. This is what it looks like. I'm going to test if the checkout works properly, and so that's how it would look like. Okay, so it's done. Stripe and Claude are basically making it easy for you and all the developers to build, set up payments, and start monetizing without going away from your building interface.
### 开发者生产力新范式：并行构建、销售与迭代
**Stripe 演讲者**：那么，这对各位意味着什么？虽然你们当中的许多人可能已经在践行这一套逻辑，但我依然想再次强调：在当下的行业操盘法则中，对开发者生产力的极致追求至关重要。当今最优秀的企业无不在速度上下足功夫，全力以赴将构建时间压缩到极致。其次，必须将构建、销售与迭代三者同步推进。我们过去所习惯的那套线性打法——先踏踏实实打磨产品，再去市场中寻找客户——如今已经彻底失效了，因为同一时间有太多团队在进行构建。因此，你必须同时并行开展这三项工作。再者，关于“自研还是购买”（build vs. buy）的抉择，根据我们所观察到的现实情况，我们坚信最出色的企业往往两者兼顾：他们倾力自研能够塑造自身差异化优势的核心能力，而对于产品运转所需的底层基础设施与通用积木式模块，则果断选择采购现成方案。
Original English
**Stripe Speaker**: So what does this mean for you? Many of you are probably already doing this, but I want to reemphasize that in the playbook today, it's really important to obsess over developer productivity. The best companies today are really working on speed and just making sure that they completely collapse build times. Second, build, sell, and iterate at the same time. The linear playbook that we had before—like you build your product, then you go find your customers—isn't just working anymore because of the fact that too many people are building at the same time. So you basically need to do all three at the same time in parallel. And then on the build versus buy question, we're actually quite convinced from what we're seeing that the best companies do both: they build what's making them differentiated, and they actually also buy the infrastructure and building blocks that they need for their products.
### 全球化 Day 1：AI 企业的出海增长与本地化落地
**Stripe 演讲者**：进入市场只是第一步，紧接着面临的核心问题是：我该把产品卖到哪里？我又该如何赢得海外的那些客户？行业数据完全印证了这一点。就在几年前，增长最迅速的 SaaS 企业通常在第一年拓展至 25 个国家，到第三年覆盖 51 个国家；而如今顶尖的 AI 企业在第一年就已经触达 42 个国家，到第三年更是拓展到了 120 个国家。这正是我所强调的“全球化 Day 1”概念——从第一天起，你的市场就是整个世界。这并不仅仅是指在这些国家建立所谓的业务存在，而是要实打实地从第一天起就从这些国家获取收入。以大家熟知的旧金山团队 Gamma 为例，这是一款 AI 驱动的幻灯片演示与网站生成工具。他们在第一年就实现了 1 亿美元的营收，而且其中绝大部分收入均来自美国本土之外。这极其鲜明地印证了当前的全球化大趋势。
Original English
**Stripe Speaker**: So getting to market is one thing, but it is also a big question of: where am I going to sell, and how am I going to get those customers abroad? The data confirms it completely. A few years ago, the fastest-growing SaaS companies were basically reaching 25 countries in year one and 51 by year three. And today, the top AI companies are already at 42 countries in year one and at 120 countries in year three. So this is like what I was saying about day one market being basically the whole world. It's not about having a presence in those countries; it's actually getting revenue from these countries from day one. Gamma is a great example—you probably know them. They're based in SF, an AI-powered slides and website builder. And they reported $100 million in revenue in year one, and the majority of their revenue comes from outside the US. So that's definitely showing how the trend is going.
**Stripe 演讲者**：放眼当今所有顶尖的 AI 企业，其总营收中有 48% 来自于其本土市场之外。换言之，你收到的每一美元或每一欧元中，几乎有一半都来自于海外国家。而在几个月前，这一比例还仅为 33%。接下来我们看看究竟是哪些市场在 AI 工具上投入了大量资金。毫不意外，GDP 保持高增长的国家赫然在列，比如法国、英国和韩国；但与此同时，我们也看到了一批在 AI 消费上投入巨大的新兴国家，例如瑞士、波兰和土耳其。海外市场的强劲需求就摆在那里。因此，当你在打造自己的企业时，必须深入思考：我该如何承接这些需求？我该怎样俘获这批海外客户？
Original English
**Stripe Speaker**: And today, if you take a look at overall, like all the top AI companies, 48% of their revenue comes from outside their home market. Nearly half of a dollar or a euro that you get is actually from a foreign country. And this figure was actually 33% a few months ago. So let's take a look at which markets are actually spending a lot on AI tools. Not surprisingly, we can see that the countries with the high GDP growth are here: you can see France, the UK, South Korea. But we also have new emerging countries that are spending a lot on AI: for example, Switzerland, Poland, Turkey. And the demand is there. So now, as you're building your company, you need to think about: how am I going to get that demand? How am I going to get those customers?
**Stripe 演讲者**：从我们的实践观察来看，核心在于你必须切实贴合买家的期望，在他们习惯的场景与方式中迎合他们。具体到实施层面，就是务必支持当地货币与本地化支付方式。这听上去很简单，而且借助 Stripe 来实现也确实非常轻松，但其本质是真正贴近用户的本土习惯。以本地化定价为例，看似只是微小的调整，但实际上能够带来高达 18% 的跨境收入增长，其转化威力极为惊人；而每当你为一个国家增加一种当地主流支付方式时，该国家的支付转化率至少能直接提升 7%。因此，摆在当今每家 AI 企业面前、用于检验自身全球化就绪度（global readiness）的有三大压力测试：第一，实现价格与支付方式的全面本地化；第二，实行税务征收的自动化，毕竟谁都不想惹上合规麻烦；第三，追踪……
Original English
**Stripe Speaker**: And what we're seeing is that you really need to meet buyers' expectations and meet them where they are. The way we're doing this is that you offer local currencies and local payment methods. That sounds easy, and that is actually easy to do with Stripe. But basically it's really meeting them where they are and with their habits. Localized pricing, for example, sounds completely simple, but basically it increases cross-border revenue by 18% higher—it's crazy the impact that it has. And basically, when you add just one local payment method for a country, you have at least 7% conversion uplift in this country. So, three checks to pressure test your global readiness as an AI company today: First, you localize your prices and your payment methods. Two, automate tax collection, because nobody wants to get in trouble either. And three, track
### AI 时代价值与定价模式的根本转变
**Speaker**: 按国家细分各项表现，比如针对转化率等具体指标进行深度追踪，并全力投入到这些指标的持续优化中。接下来谈谈定价，因为旧有的商业与收费模式如今已经不再奏效了，我们必须开始思考全新的定价模式。
Original English
**Speaker**: ...performance by country, like performance, um, conversion, and obsess over optimizing those. So now pricing, um, because the old models are not working anymore. Now we have to think about like new pricing models.
**Speaker**: 非常有趣的一点在于：价值本身是具有高度弹性的。如果你稍微研究过定价策略，大概就会明白这一点。大家可能都在使用同一款 AI 工具，但每个人的使用方式和场景却大相径庭。
Original English
**Speaker**: The interesting thing is that like value is elastic. You probably know that if you've taken a look a bit at pricing and so everyone might be using the same AI tool, but they might not using the same way.
**Speaker**: 以两位截然不同的用户为例：其中一位是软件工程师，他们晚上去睡觉，第二天一早直接进行代码审查（Code Review），AI 为他们产出的是实打实的工程代码交付成果；而另一位只是普通日常用户，他们只是用 ChatGPT 代替了 Safari 浏览器进行日常搜索，产出的不过是一次体验升级后的搜索结果。
Original English
**Speaker**: Um, so take two users. One is an engineer. They're going to bed. Next day they're having a code review. Um, and the output is really engineering output. The other user is just an everyday user. They replace, uh, Safari with GPT and the output is just an upgraded search experience.
**Speaker**: 对于这两种情况而言，背后的算力与服务成本不同，创造的核心价值不同，因此最终的定价机制也绝对不应该一刀切地保持相同。
Original English
**Speaker**: So for this, like the cost is not the same, the value is not the same, and so the pricing should definitely not be the same.
### 技术范式变迁：从本地授权、SaaS 订阅到混合计费
**Speaker**: 这种因技术变革引发的价值与定价重构并非新鲜事。回顾历史，在每一次重大的技术浪潮中我们都经历过类似的过程。例如在本地主机部署时代，软件通常采用按年购买 License 授权的销售方式；后来云计算时代到来，软件能够实现在线自动更新，于是整个行业自然而然地转向了 SaaS 订阅制。那么到了今天的 AI 时代，我们又该采取怎样的模式？这可能正是大家期待我给出一个终极标准答案的地方。
Original English
**Speaker**: So this change in value isn't new. We had this with all the technology shifts that happened. Like for example, we had local host hosting at a time, so it was normal to download the software for a yearly license. Then cloud computing arrived and so we had auto updates of softwares. So it was actually, uh, normal to move to subscriptions. And today, like what do we have with AI? So that's basically where you'd expect me to tell you what the best answer is.
**Speaker**: 先说坏消息再讲好消息。坏消息是，我们目前仍处于极其早期的探索阶段，整个行业还没有形成所有公司普遍遵循、绝对最优的“黄金标准”。但好消息在于，一种明确的行业范式已经清晰地浮现出来。
Original English
**Speaker**: Um, so good news, bad news. Bad news first: it's still early, like we don't have a gold standard of what all companies are doing and it is absolutely the best. But good news is that we're clearly seeing, um, emerging a pattern.
**Speaker**: 以今天大会的主持人所在的 Postman 为例——作为大家非常熟悉的开发者工具公司，他们已经深耕行业约十年之久。当 AI 浪潮来袭时，他们敏锐地对产品进行了战略转型，开始全面引入 AI 工具与 AI 功能，同时也对定价机制进行了调整，比如叠加了点数额度（Credits）等模式。他们起步于固定的基础订阅制，以此确保产品的用户粘性并建立持久的用户连接；随后随着 AI 成为产品核心价值驱动力，进一步叠加了使用额度。这一转变让他们今年正向着 10 亿美元 ARR 的目标稳步迈进。他们绝非个例，这只是我们观察到的众多典型样本之一。
Original English
**Speaker**: Take for example Postman, our MC today, a developer tools company that you probably know that's been around for around a decade basically. And when AI tools arrived, they pivoted their product, they started adding AI tools, AI features in the product, and they also started to adapt the pricing, like layering credits, etc. So they started with flat subscriptions, which is how you stick and create the relationship with the user, and then added credits as AI became core to the value of the product. So the result is that today they're targeting 1 billion ARR, uh, this year, and they're not the only ones to do this. This is just one of the examples of what we can see.
**Speaker**: 如今，在《福布斯》AI 50 强榜单的上榜企业中，已有整整三分之二的公司采用了基于用量（Usage-based）的定价模式。其中绝大多数公司都在运行一种混合模式：一方面保留基础订阅以维系客户粘性与稳定关系，另一方面大量引入用量计费机制，以确保商业变现能够精准映射产品所释放的真实价值。
Original English
**Speaker**: So today we have like two out of three companies of the Forbes AI50, um, listing that are using usage-based pricing. So a majority of these companies are actually using a hybrid model. They're using subscriptions still for the stickiness of the relationship, but they're adding a lot of usage-based pricing to make sure that the value is actually reflected in the pricing.
### AI 定价的实操三原则：语言、透明度与价值货币
**Speaker**: 意识到必须采用某种形式的用量定价固然重要，但真正把它落地做对则是另一回事。在我们看来，打造 AI 定价策略需要重点把握三项核心法则：
Original English
**Speaker**: So it's good to know that you have to do some form of usage-based pricing, but getting it right is actually another story. So in our view, there are three things you need to do to add, uh, to your playbook for pricing.
**Speaker**: 第一，用客户的语言来定价。开发者天然以 Token 为思考单位，而企业决策层大多数时候则习惯按员工席位（Seats）来核算成本。你必须深刻理解你的目标客群究竟是谁，并确保你的计费标尺与他们在产品中感知价值的方式完全匹配。
Original English
**Speaker**: First, price in your customer's language. Like developers think in tokens, enterprises most of the time think in seats. Like you have to think about these, uh, like who are your customers and make sure that you match their pricing with the way that they experience value in your product.
**Speaker**: 第二，为用户提供近乎实时的用量消耗透明度。这绝非可有可无的附加功能，而是企业防范客户流失的最根本手段。这意味着你必须在产品体验中找到有效途径，让客户能够实时直观地看清自己消耗了什么、花费在何处。
Original English
**Speaker**: Second, you want to show like as close as real-time visibility on what they're consuming because, like you know, this is not a nice to have and this is basically how you prevent churn as a company. Like this means finding ways to make sure they're seeing real-time what they're consuming.
**Speaker**: 第三，我们强烈主张销售“额度点数”（Credits）而非直接销售底层“成本”（Costs）。客户在充值时固然是将法定货币兑换为点数，但在他们后续使用产品的整个过程中，他们不会在心中为每一欧元、每一分钱斤斤计较，而是以额度点数的形式与产品交互。这样一来，消费体验锚定的是产品带来的业务价值，而不是底层算力的成本支出。
Original English
**Speaker**: And three, we really see that it's strong to sell credits and not costs because, like at one point, your customers will exchange money for credits, but then the way they will interact with your products will not be with like each euro or each cent. It will be with credits and it will be related to value mostly, not costs.
### GTM 市场策略变革与智能体买家的兴起
**Speaker**: 接下来谈谈市场进入策略（GTM Motions）的变革。尽管很多人目前把这部分放在次要位置，但它实际上正在发生极其深刻的变化，全新的市场动作正在加速涌现。其中最显著的趋势就是渠道销售（Channel Sales）模式的重构，即你的产品究竟将在哪里销售、通过何种方式达成交易。
Original English
**Speaker**: So now we're going into go-to-market motions changing, uh, which is also a big part even if, like, a lot of people are actually taking that bit of the side, um, but that's actually a big motion changing as well. Um, actually entirely new motions are emerging. You might have seen that and the biggest one is channel sales, like it means where your product is going to be sold and how.
**Speaker**: 例如在当下的 AI GTM 实践中，大量企业实际上是通过各自的云服务商（Cloud Providers）采购 OpenAI 或 Anthropic 服务的。因此你必须充分考量这些正在兴起的多元化销售渠道。与此同时，大量 AI 企业正在构建自己的专属生态市场（Marketplaces），你的产品必须能够被纳入这些平台目录作为标准服务供客户采购。
Original English
**Speaker**: And like today in the AI go-to-market motion, for example, a lot of, um, companies are actually buying OpenAI or Anthropic via their cloud provider. So you need to think about these different channel sales that are arriving and a lot of AI companies are actually building their own marketplaces where you are going to be able to be a reference and to be bought as well.
**Speaker**: 当然，还有一个摆在所有人面前的十亿甚至万亿美元级核心命题：你该如何将产品销售给你的全新买家——也就是 AI 智能体（Agent）？过去数十年里，我们耗费了无数心血去钻研人类消费心理学，比如怎样通过把价格定在 9.99 欧元而不是 10 欧元来促进转化；但面对智能体时，它们对这些把戏毫无感觉。
Original English
**Speaker**: And of course, like the billion or maybe trillion dollar question: like how do you sell your products to your newest buyer, which is, you know, the agent? Like we've spent decades thinking about what is the best way, you know, human psychology, I will price my product €9.99 and not €10, but like agents absolutely don't care about any of this.
**Speaker**: 眼前这张图表是“必须将智能体视作采购决策者”的最直观信号。这是 Stripe 的公开开发者文档流量统计：粉色曲线代表智能体读取文档的流量，紫色曲线代表人类读取的流量。大家可以清晰看到，就在 2026 年的这个夏天，两条曲线迎来了历史性交叉——如今我们公开文档的智能体阅读量已经全面超越了人类阅读量。在 Stripe，我们每天都能切身体会到这一巨变：我们必须重新思考产品与文档该如何被智能体高效读取、解析与理解。你必须彻底改变做事的方式，重新设计展示产品与被智能体发现的机制。
Original English
**Speaker**: And today I think this chart is one of the clearest signal of like I need to think about my agent as a buyer because, like for example, this is Stripe docs, uh, so our public documentation. The pink line is, uh, how it's read by agents and the purple line is by humans. And as you can see, like this very summer in '26 we had the two lines crossed, which means that our public documentation today is more read by agents than by humans. So we live it every day at Stripe, like we have to think about, okay, how is my product, how is my documentation read and understood by agents. So you have to change the way you do things. You have to change the way you are going to showcase your products and make them discoverable by agents.
### 构建渠道中立与智能体就绪的统一体系
**Speaker**: 市场推广策略的调整仅仅是冰山一角。一旦你开始着手重塑渠道销售模式，你就必须同步对底层产品进行全面改造：确保用户入引（Onboarding）体验能够与全新的渠道旅程完美契合，定价与营收体系必须随之重构，组织架构也需要相应调整。例如，如果你拥有传统的企业销售团队，就必须推动他们主动适应全新的市场拓展路径。
Original English
**Speaker**: And go-to-market is like the tip of the iceberg, because if you think about like changing channel motions, then you also think about like adapting your product to that, making sure that your onboarding is going to reflect that new onboarding motion. Pricing and revenue of course is going to have to reflect that as well, your organization. Um, so for example, if you have enterprise sales, you are going to have to adapt this to the new go-to-market motions as well.
**Speaker**: 这是我对应对方案的系统性思考：当今的产品形态是渠道中立（Channel Agnostic）的，客户本身也是渠道中立的，他们并不真正在意具体的购买入口。一个客户可能最初通过自助服务模式（PLG）进入，中途升级为企业级销售合约（SLG），随后又通过自主部署的 AI 智能体来落地执行新功能。你必须统筹考虑整个生命周期的连贯旅程。
Original English
**Speaker**: So this is how I would think about the playbook. Like basically today the products are channel agnostic and the customers themselves are channel agnostic. They don't really care about where they buy. So they may be self-served at some point, graduate to enterprise, and then like have an agent implement a new product. So you need to think about all of this journey, uh, for your product.
**Speaker**: 具体而言包括三点：首先，设计一条清晰的演进路径，实现从产品驱动增长（PLG）到销售驱动增长（SLG）的平滑衔接；其次，确保后台系统的绝对统一，维护统一的产品目录（Product Catalog）、统一的客户实体对象（Customer Object）和统一的数据模型，切忌各行其是、四分五裂；第三，正如我刚才所强调的，在架构设计上必须全面面向智能体就绪（Agent Readiness），确保你的产品在完全没有人工介入的情况下，就能被智能体自主发现、评估与采购。
Original English
**Speaker**: So have a clear graduation progress from product-led growth to sales-led growth. Uh, then make sure that you have unified system, you have like one product catalog, that you have one customer object, um, one data model, and that you don't go away in all directions. And third, like as I was saying, please design for agent readiness. Like make sure that your product is ready to be discovered and bought by agents without a human in the loop.
### AI 时代的欺诈防御与批量多账号滥用挑战
**Speaker**: 最后一个核心模式是欺诈防御（Fraud）。这是一个至关重要且必须严肃对待的课题：必须尽早、常态化地严密监控欺诈与安全风险。欺诈之所以极难对付，是因为它绝不会局限在单一环节，在产品生命周期的任何节点上都潜藏着遭受攻击的脆弱面。从新账号注册、绑定支付方式、支付清算处理，一直到服务使用后的账单结算，处处都是潜在的欺诈风险暴露点。
Original English
**Speaker**: Okay. So last pattern: fraud. Um, this is actually something very important that you need to think about. Monitor fraud, um, and risk like early and often. So what makes it hard basically is that like it doesn't show up in one place. Like at all time of your product lifecycle there's going to be a potential for an attack. So if you think about account creation, payment method added, payment processing, and then after that like post-usage, all of these places are potential, um, places for fraud.
**Speaker**: 我们先来看看处于漏斗最顶端的注册环节（Top of the Funnel）。这里的典型困境在于，恶意黑客会通过自动化脚本不断批量注册新账号，瞬间将你提供给新用户的免费试用额度席卷一空。给大家一组直观的数据：在当今的互联网环境下，每六次新账号注册中就有一次涉及“多账号滥用”（Multi-account Abuse）。黑客们实际上是在规模化薅取你的免费试用额度，而你却永远无法从他们身上赚到哪怕一分钱。
Original English
**Speaker**: So let's take first like sign-up top of the funnel. The thing is, you're going to have someone, like a bad actor that signs up again and again, and so they're basically just stealing all of your user credits, um, at once. One in six signups, to give you an idea, today, uh, involves multi-accounts abuse. So, they're basically farming you free trial, um, without you ever earning a dollar from them.
**Speaker**: 更严峻的挑战在于，生成式 AI（GenAI）的发展让识别虚假账号变得异常艰难，因为这些机器人账号伪装得与真实人类别无二致。值得关注的是，这种多账号滥用正是规模化免费试用欺诈最常见也是最核心的运作机制。从数据上看，这种攻击正在呈爆发式剧增，正如我们在 Stripe 支付网络中所监测到的惊人增速一样。到 2026 年，免费试用欺诈的激增已经达到了令人瞠目的程度——因为在互联网世界里，只要你的业务取得成功，你就必然会成为黑产极具吸引力的首要攻击靶标。不过，值得庆幸的好消息在于……
Original English
**Speaker**: And the problem also is that GenAI today has made it very hard to, you know, find fake accounts because they actually look real. And what's interesting is that multi-account abuse is often like exactly how free trial, um, abuse happens at scale. So you can see that it increases dramatically, like this is how it accelerated in the Stripe network. So you can see that in 2026 we have this crazy increase of free trial abuse, because like if you're having success, like you're actually also an attractive target on the internet. Um, what's good news...
### 应对免费试用与按量付费中的欺诈滥用
**Ariel**: 不过值得庆幸的是，这是一个完全可以解决的问题。就像你会看到针对产品的攻击呈指数级增长一样，只要你采用了正确的应对方案，同样也能看到攻击数量迅速且断崖式地回落。例如在短短四个月内，我们就在免费试用环节拦截了大约三百万次此类攻击。因此，防范此类风险绝对是我们最为关切的核心事项之一。
Original English
**Ariel**: ...though is that it's a solvable problem. Like just as you can see an exponential growth of the attacks, you can also see a very strong and steep decline of the attacks if you have the right solution. Like just in four months we blocked like three million attacks like this for free trial. So that's definitely part of top of mind for us.
**Ariel**: 当然，还有另一种大家可能非常熟悉的攻击路径——它伴随着“按量付费”（pay-as-you-go）定价模式而生，也就是针对按量付费机制的滥用。具体发生的情况是：某个恶意用户在单月内消耗了价值数千美元的 Token，但到了月底结算尝试扣款时，你却发现绑定的信用卡根本无法扣费，甚至直接失效。这有点像法国人常说的“米其林霸王餐”，只不过这次消费的是模型算力与 Token。等你知道卡扣不掉款的时候，算力已经耗尽了，你根本无法向那个用户追讨任何费用。按量付费确实是一种非常棒的定价策略，但采用它时，你必须深思熟虑该如何防范这种欺诈。那些处理得当的公司通常都会实时监控使用行为信号，确保能捕捉到各种异常指标，从而在事态失控前有效阻断损失。
Original English
**Ariel**: And then, of course, there's another vector you might be familiar with: pay-as-you-go pricing. Pay-as-you-go abuse goes with that. So what happens is that a consumer would basically consume, let's say, thousands of dollars of tokens in a month, and then at the end you try to get the money and the card is just invalid. So it's kind of like, you know, a "Michelin dine-and-dash" like we say in French, but for tokens, right. So by the time you know, the compute is gone and you can't charge anything to that user. Pay-as-you-go is an amazing tool for pricing, but you have to think about how to manage fraud in that case. So the companies handling that well are monitoring usage signals in real time and make sure that they see all the different signals and avoid that from happening.
**Ariel**: 那么面对这些风险，你应该做些什么？这里我再次给出三点建议：第一，请在用户全生命周期的每个阶段都进行持续监控，从注册创建账户一直贯穿到使用之后，全力为你的正当客户守护业务安全；第二，我建议尽可能缩短实际用量与账单结算之间的时间差。你绝不能等到一个月后再去集中扣款，必须让扣款与消耗尽可能紧密贴合；第三，请在最初制定产品发布上线方案时，就把防欺诈机制内置进去。这或许不是产品构建过程中最令人兴奋的环节，但它极其关键。在规划产品的每一步时，你都需要换位思考：如果我是攻击者或恶意分子，我现在想钻空子套利，具体的攻击形式会是怎样？在我看来，如果你还没考虑过这个问题，或者目前根本没有明确的防范答案，那你其实还没有做好向市场发布产品的准备。
Original English
**Ariel**: Okay. So what do you do about this? Three things again: please monitor this at every stage of the life cycle, from account creation to post usage. Defend all of this for your customers. Then I would recommend to close the gap between usage and billing. You don't want to wait a month to bill; you want to make sure that you close that gap as much as possible. And then, please build fraud into your launch plan. It's not necessarily the most exciting part about building, but that's actually incredibly important. At every step of your product, you need to think about: okay, if I am a bad actor now and want to take advantage, what does the attack look like? And so in my view, if you haven't thought about this or you don't have the answer yet, you're not ready to ship it.
### AI 时代重写商业规则：第一天即全球化与按价值定价
**Ariel**: 商业的打法正在被全面重写，这一点我们心知肚明。但真正发生蜕变的，不仅仅是底层工具，而是你如何构建软件、如何走向全球市场、如何设定价格、如何调整进入市场（GTM）的节奏，以及如何管控欺诈风险——事实上，是创立一家公司的整个起跑线被彻底重塑了。如大家所见，你现在可以在数小时内构建出核心产品，第一天就能覆盖全球 42 个国家，并且能够直接按照产品创造的真实价值进行定价。在我们看来，这或许是全世界历史上构建公司最有趣、也可能最疯狂的时代。我们希望今天分享的洞察能为大家的创业与产品构建带来启发。在 Stripe，我们的职责就是陪伴在大家身边，帮助你们平稳扩展规模。非常期待看到各位创造出的产品！如果你们面临规模化扩张的需求，请记得我们随时都在这里提供支持；若想深入交流，欢迎随时给我发信息。非常感谢大家！
Original English
**Ariel**: So the playbook is being rewritten. We all know that. But how you build, how you go global, how you price, how you adapt your go-to-market motions, and how you manage fraud—that's what's changed. Not just the tools, but actually the whole starting line of building a company. And you know that you can build in hours, you can be present in 42 countries on day one. As we can see, you can price for value. So that's actually, in our view, the funniest and maybe craziest time for building a company in the world. So we hope that these insights give you some idea of how to build this. And yeah, our job at Stripe is to be there with you to help you scale. So I'm really excited to see what you build. And just please know that we are there if you need to scale as well, and drop me a message if you want to chat more. Thank you.
### 从文字到语音：为人与 AI Agent 同时设计产品
**Rau**: 好的，非常感谢 Ariel！让我们再次把掌声送给 Ariel！太棒了。在刚才的整场演讲中，最触动我、让我深有感触的一点是：大家不仅需要为人类设计产品，同时还必须为 AI Agent 设计产品，因为未来很可能正是这些智能体在代表用户直接购买你们的服务，我认为这个视角极其震撼。之前我问过大家一个问题：“在场有多少人是构建者（Builders）？”当时很多人都举了手，对吧？你们大部分人都在一线做产品，可能平时也在频繁与各类智能体对话。以我个人的亲身体验来说，我已经基本不再用键盘给我的 Agent 敲字了；我现在主要只通过语音与智能体交流，我发现这是一种更加自然的交互方式。虽然我阅读文字的速度确实比听语音要快，但与此同时，我开口说话的速度也远远快过我用双手打字的速度。不知道现场有没有人和我有同感？在座平时会用语音和自己的 Agent 对话的人请举个手？好的，看来人数相当不少。
Original English
**Rau**: All right. Thank you, Ariel. Let's give it up for Ariel once again, please. All right. Okay. I think the piece of information that stayed with me through this presentation is that you need to design for humans, but you also need to design your products for agents, because they might be purchasing your products as well, which I think is awesome. I asked earlier a question: I said how many people are builders here? And many of you raised their hands, right? Like yeah, most of you are builders, you're building products. And you're maybe chatting with agents. My personal experience is that I kind of stopped typing to my agent. So I only chat mostly with voice, and I find it like a more natural way to interact with an agent. I read faster than I hear, but I talk a lot faster than I type. I don't know if anybody shares that experience here. Who talks to their agent? Yeah. Okay, so it's actually a good number.
**Rau**: 那么，我们的下一位演讲嘉宾正是专门来为大家剖析语音 AI（Voice AI）与实时交互技术的。话不多说，请大家与我一同热烈欢迎 Piano AI 的首席科学家兼联合创始人——Erve Ba 登台！
Original English
**Rau**: Well, our next speaker is here to talk to us actually about voice AI and real time. So without further ado, please join me in welcoming to the stage chief scientist and co-founder of Piano AI, Erve Ba.
### 走进说话人日志：揭秘 Voice AI 的关键感知层
**Erve Ba**: 好的，让我试一下这个设备。没问题，声音正常。大家好！谢谢 Rau 的介绍。这是我人生中第一次在舞台上被专业主持人这样隆重报幕，感觉今天在巴黎有点像席琳·迪翁（Céline Dion）登台一样。不过别担心，我今天不会唱歌，而是要和大家聊聊语音 AI（Voice AI），特别是其中的核心技术——说话人日志（Speaker Diarization）。首先做个简短的自我介绍：我是 Piano AI 的联合创始人兼首席科学官。在过去职业生涯的大部分时间里，我一直是一名学术界的研究人员，直到两年前我们创立了这家公司。在过去的 15 年里，我几乎一直在专注深耕说话人日志领域。大家似乎觉得我对这个领域略知一二，所以我希望今天能向大家清晰阐述究竟什么是说话人日志，讲解它的底层运作原理，也希望大家至少能从今天的分享中有所收获。
Original English
**Erve Ba**: Okay, let me try this thing. Yeah, it's working. Hi everyone. Thank you Rau for the introduction. It's the first time I've been ever announced on a stage by an MC. I feel a bit like Celine Dion in Paris tonight—or today at least. Don't worry, I won't sing anything, but I will talk to you about Voice AI and in particular about speaker diarization. Quick introduction about myself: I'm the co-founder and chief science officer at Piano AI. I've been an academic researcher most of my career up until two years ago when we created this company. I've been mostly focusing on speaker diarization for the last 15 years, and it looks like people think that I know a few things about speaker diarization. I hope today that I'll tell you what speaker diarization is and that you'll learn a bit about how that works, and yeah, that at least you take something out of this talk.
**Erve Ba**: 那么，究竟什么是说话人日志？本质上，说话人日志的任务是获取一段多人对话的录音，并输出相关的结构化元数据。面对一段连续的对话录音，你首先能做的基础处理是语音活动检测（Voice Activity Detection，简称 VAD）——即自动检测出音频中哪些时间段有人在说话，哪些时间段无人发言。在 VAD 的基础上，第二步是将那些较长的连续语音片段进一步切分成属于单个人发言的独立区间，这一任务被称为语音切分（Segmentation）。它既包含说话人切换检测（Speaker Change Detection），也包括重叠语音检测（Overlapping Speech Detection）。比如大家看中间这个示例，这里明显显示至少存在两位说话人：其中一人在打断另一人，或者是在前一个人的发言回合结束后立刻接话；而再往后一点的对话中，出现了一个非常短暂且与主发言重叠的语音片段。这类重叠通常就是我们所说的反馈声道（Backchannels），比如当别人和你交谈时你点头附和发出的“嗯嗯”、“好的”、“对”。对于语音 AI Agent 来说，精准捕捉这些事件至关重要，因为只有这样智能体才能真正理解对话的核心动态——单纯识别出字面单词往往是远远不够的；了解某人在何时打断了对方，或者在何时对对方的观点表示附和，能为整场对话提供丰富得多的上下文信息。
Original English
**Erve Ba**: So what is speaker diarization? Basically, speaker diarization is the task of taking a recording of a conversation between multiple speakers and outputting some metadata about it. What you can do starting from a conversation is basically do voice activity detection first, where you would detect the parts where someone is speaking and when nobody's speaking. That's voice activity detection. The second step that you can do on top of that is splitting those very long speech turn regions into single speaker regions. This task is called segmentation. So it includes both speaker change detection as well as overlapping speech detection. In this example here in the middle, you see that it looks like there might be at least two speakers because there's someone interrupting another one or starting speaking after the end of another speech turn, and a bit later in the conversation there is a very small speech turn that overlaps with the other one. These are usually what we call back channels, like "mhm", "okay", "yes", that you say when you are nodding at someone talking to you. Those events are very important for voice AI agents to basically understand really the core of the conversation. The words are usually not enough; knowing when someone is interrupting someone else or nodding at what they are saying gives a lot more information about the conversation.
### “谁在何时说了什么”：核心挑战与关键应用场景
**Erve Ba**: 接下来最关键的一步，也就是我们严格定义的说话人日志：在将对话准确切分成各个发言轮次之后，将每一个发言轮次精确归属到对应的说话人身上。而这项工作必须在完全不预先获知说话人总数的前提下完成。这就使得该任务比常规的监督机器学习任务要困难得多；与此同时，我们事先也完全不知道说话人的真实身份，模型必须具备强大的泛化能力，能够处理任何说话人，包括那些从未在任何训练集中出现过的声音。这正是让说话人日志极具技术挑战性的原因。
Original English
**Erve Ba**: And the next step, really what we call speaker diarization, is then once the conversation is segmented into speech turns, to actually assign each speech turn to the right speaker. And this is done without knowing in advance the number of speakers. So that makes the task more difficult than usual supervised machine learning tasks, and we don't know the identity as well of the speaker. We have to basically generalize to any speaker, even those that are not part of any training sets. So that's what makes speaker diarization difficult.
**Erve Ba**: 说话人日志在众多实际场景中都能发挥巨大价值，其中很多任务都可以归结为一个核心逻辑：搞清楚“谁说了什么”，其重要性丝毫不亚于搞清楚“说了什么内容本身”。举例来说，在视频配音应用中，假设你想拿一段原始法语视频放到互联网上向更广大的国际受众传播，你可以借助 AI 视频配音工具将法语翻译成其他语言，同时必须确保译后对话听起来依然自然流畅。为此，你必须把精准克隆的声音模型对应绑定到正确的原片人物身上，这就要求你必须在原始音频中分秒不差地获知究竟是谁在什么时候开的口。另一个显而易见的典型场景是会议纪要助手，尤其是在混合办公的多人会议中——当多名参会者共处同一间实体会议室时更是如此。如果每个人各自单独接入独立音频流，说话人日志或许并不那么迫切；但当所有人围坐在同一个房间使用共享麦克风时，这项技术就变得不可或缺。它能帮助系统生成结构清晰、高质量的会议总结，明确区分出在会议结束后具体指派谁去执行什么任务等等。此外还有……
Original English
**Erve Ba**: And so there are many tasks where speaker diarization can be useful, and lots of those tasks can fall actually under the "knowing who said what is just as important as what was said." So for instance, for video dubbing applications, when you want to start from a video in French for instance, and you want basically to put it on the internet and reach a larger international audience, then you can use video dubbing tools to basically translate from French to other languages and make sure that the conversation remains natural. So you have to put the right speaker clone voice onto the right speaker, and for that you need to know exactly who speaks when in the original audio. Other obvious tasks are meeting note takers, especially in hybrid meetings where some people are joining from the same meeting room. When there are multiple streams, speaker diarization is not necessarily useful, but when people are in the same room, that's where it's really useful. And this allows to make some nice summaries of the conversation of who's supposed to do what after the meeting, for instance. And other...
### pyannote.audio 开源工具与说话人日志应用
**演讲者**：比如播客智能分析这样的应用场景，你可能希望在多个播客单集中追踪某一位特定的发言者，或者跨多期节目持续追踪播客主持人的发声。为此，我们开发了一个名为 pyannote.audio 的开源工具包，它能够帮助你直接从原始音频录音生成完整的说话人日志（Speaker Diarization）。多年以来，它受到了社区非常广泛的欢迎。借助于我们在 Hugging Face 平台上的合作伙伴，这里展示了一些值得骄傲的展示指标：它在平台上非常受关注，平均每月下载量达到 1000 万次。基本上，只需几行代码，你就能从原始音频直接获得实际的说话人日志结果。在屏幕上的这个代码示例中，第一块代码主要是从 Hugging Face 下载开源的 Community-1 模型。在 Python 代码中获取到这个名为 Community-1 的流水线对象后，你只需将该流水线直接应用到对话录音上，就能得到预测结果；接着你可以遍历该结果，精确获取每个语音片段的起始时间、结束时间以及具体的说话人是谁。这一切真的只需三行代码即可搞定。
Original English
**Speaker**: ...applications like podcast intelligence where you want to track one particular speaker in multiple podcasts or track the host of a podcast across episodes. And um so we have this uh open source toolkit called uh piano to audio uh that uh allows you actually to get from raw audio recording to uh uh an actual speaker deration. So it's it's become very popular over the years. Uh I have some vanity metrics here uh uh thanks to our friends at hugging face. It's also very visible there. Uh it has uh on average 10 million downloads every every month. And basically in a few lines of code you can get from the audio to the actual speakerization. So here in this example we start for the first block of the code as basically downloading uh the community one model the open source community one model from hugging phase. You get this uh object called community one here in the P Python code and then you simply apply uh this uh uh pipeline on a recording of a conversation and you get the prediction that you can then iterate over to know exactly start time, end time and the actual speaker time and this really uh in in three lines of code.
### 从批处理到实时：语音智能体的现实需求
**演讲者**：我刚才描述的这一切属于批处理（离线）模式，也就是说，整个处理过程是在对话完全结束之后才进行的。那么，我们为什么非要走向实时呢？想要实现实时处理，背后其实有很多驱动因素。对于传统的语音智能体（Voice Agent）来说，这种需求可能还不算特别明显，因为通常只有一个人在对着智能体说话，场景中只有一位说话人，你自然不需要关心说话人日志。但很快——实际上现在已经出现这种情况了——智能体需要在我们开会等场景中实时旁听对话。一个典型的应用案例就是：实时标记某些频繁打断他人的发言者，或者提醒某位发言者发言时间过长。要在会议中实现这类功能，实时处理是必不可少的。再往前推进一步，机器人很快就会走进千家万户。当它要对家里的孩子或是成年人做出针对性回应时，首要前提就是必须明确“当前说话的究竟是谁”。这正是实时语音说话人日志真正大显身手的地方。
Original English
**Speaker**: Um yeah so this uh that I described was batch in the sense that uh the whole processing happened after the conversation took place. So why uh go real time? So there are many uh reasons why we we would want to go real time. Uh for voice agent it's not that obvious because usually there's one person talking on an or to an agent and uh you you don't really care about uh speakerization because there's only one speaker. But very soon and it's already there actually the agent will listen to to the conversation while we're having a meeting for instance and to a use case would be to flag some speakers interrupting another speaker too often or to flag a speaker talking too much. Uh and doing that in real time is is necessary. And uh the next step is also that we soon are going to have robots in our house. And uh to basically answer to uh a child or to the adult in the in the household, you would uh basically need to know who's who. And this is really where uh real time voice darization speakerization is useful.
### 批处理说话人日志的技术内幕：分块与嵌入表示
**演讲者**：既然我们已经清楚了实时处理的实用价值，那么我们在 pyannoteAI 又是如何实现实时化改造的呢？首先，我需要退回一步，先为大家拆解一下刚才提到的、只需三行代码调用的 Community-1 批处理说话人日志在内部究竟是如何运作的。因此，我打算在这张幻灯片上多花一点时间。假设有一段时长为 1 小时的对话录音，目前 Community-1 的工作流程是：首先将整个对话切分成若干小片段，比如 20 秒或 30 秒的时长。这张幻灯片中的每一个绿色矩形，实际上就代表了 30 秒的语音片段；大家可以看到，它们之间是有一定重叠的。我们以滑动窗口的方式向前推进，然后应用第一个神经网络，对每一个音频块（chunk）分别独立执行说话人日志分割。
Original English
**Speaker**: So now that we know that it's actually useful, how uh did we at PenAI uh manage to go real time? Uh I need first to take a step back to explain how the batch speakerization that I just described earlier in three lines of code called community one works internally. So uh I'm going to spend some time on this slide a bit more. So um basically given a let's say a 1 hour conversation uh a recording of a 1 hour conversation how it works how community one works today is we start by splitting the conversation into pieces let's say 20 seconds or 30 seconds pieces. So each uh green rectangle here in in this slide uh actually uh represent 30 seconds of speech let's say you see that they are overlapping a bit. So we we move that in a in a sliding uh manner uh uh in a sliding window manner and then we apply the first uh neural network that basically performs uh speakerization on each chunk separately.
**演讲者**：我们为什么要这么做？为什么要逐个片段处理对话，而不是直接进行全局处理呢？原因在于，在局部音频块上操作可以将输出的说话人数量限制在一个固定且较小的范围内。这样一来，机器学习任务的求解就会更加简单、更加稳健。具体来说，该模型的训练目标是输入一段 30 秒的音频，返回一个矩阵——可以理解为时间步与说话人维度的矩阵——并输出 0 和 1，以此标记每个说话人在各个时刻是否处于活跃发声状态。但这种初始方法存在一个痛点：在当前阶段，同一个说话人在第一个音频块中的标签，到了另一个不同的音频块中可能会被赋予完全不同的标签。
Original English
**Speaker**: So why do we do that? Why why do we uh process the conversation pieces by pieces rather than uh trying to do that globally? It's because uh working on chunks allows to uh have a limited number of speakers as output. So the the the machine learning task is actually easier and safer to to solve like that. So basically the model is trained to take 30 seconds of audio as input and returns uh basically a matrix of let's say uh the time and the number of speakers uh and um it will output zero and once whether each speaker is active over time. And the problem with this kind of a first approach is that at that point uh the same speaker on the first chunk might have a different uh label on the same for the in a different chunk.
**演讲者**：因此，我们必须依赖第二步：利用说话人嵌入（Speaker Embeddings），将每个音频块中的每位说话人——在图中用三角形、圆形和星形等符号表示——投影到一个高维特征空间中。我相信大家对什么是“嵌入”都非常熟悉了，简而言之，它就是对某种事物的高维数学表示。在这里，这个事物就是一个个发声轮次（Speech Turn）。每个发声轮次都会被投影到该空间中。在幻灯片中为了直观展示画成了三维形式，但在实际工程中……哎呀，幻灯片翻掉了。实际中……画面回来了吗？回来了，我还是老老实实站在这里吧。这个提取说话人特征的模型同样也是一个神经网络。它的训练目标是：让同一说话人的两个发声轮次在特征空间中靠得足够近，而让不同说话人的发声轮次彼此远离。通过这种方式，我们就能在这个空间中寻找并形成聚类簇。这也正是我们最终调和所有音频块的依据所在，使我们能够将所有这些细碎的发声轮次拼接在一起，得出最终的全局日志结果。这就是批处理模式下说话人日志的基本运作机制。
Original English
**Speaker**: So we need to rely on a second step uh where each speaker in each chunk here represented by symbols the triangles the circles and the stars are projected into a highdimensional space using speaker embeddings. I'm pretty sure you're all familiar with what embeddings are, but basically those are uh highdimensional representation of a thing. Here the thing is a speech turn. So each speech turn is uh projected in into the space. Uh here it's represented as 3D but in practice it's uh Oops. Uh in practice it's um it's coming back. Is it? It is. I should probably stay here. Um and uh this speakering model, it's a neural network as well. It's trained to have two uh speech turns of the same speaker close to each other and two speech turns from two different speakers far away from each other. And this way we can actually find uh clusters in this space. And this is how basically we managed to reconcile all the uh the chunks finally and be able to basically stitch uh uh all those small speech turn together and and get the final results. So that's the starting point of um um of speaker dization in a batch manner.
### 增量聚类挑战与 diart 开源库
**演讲者**：那么核心问题来了：我们是如何从这种离线批处理日志走向真正的实时落地的？我们着手改进的第一项工作，就是刚才提到的聚类部分——将全局聚类改造为增量聚类（Incremental Clustering）。屏幕上的这段动画可以非常直观地说明一切。在批处理模式下，面对两位说话人之间的对话，我们首先从所有发声轮次中提取说话人嵌入；你会得到所有的点，这些点实际上就是特征嵌入向量，然后你在所有数据上统一执行聚类算法即可。然而，一旦切换到流式（Streaming）环境，你根本无法获知未来的音频信息，因为后续的对话尚未发生。这就要求聚类算法必须以增量方式运作，而这使得任务难度急剧上升，因为算法在每个瞬间都只能掌握局部对话视野。
Original English
**Speaker**: So the question is how did we get from uh this uh batch diorization to an actual uh realtime implementation. So the first thing that we uh worked on is basically the clustering part that I just mentioned and instead of having it uh global having it incremental. So uh this uh um animation will will say it all. So uh so basically starting with a conversation uh between two speakers here in batch dorization we start by extracting speaker embeddings from all those speech turns. So you get them all those points are actually speaker embeddings and you simply do uh clustering uh into that. But if you turn to streaming you don't have access to the future because the conversation did not happen yet. So the idea is that uh the clustering algorithm needs to work incrementally and that makes its job way harder because you only have a limited view of the conversation.
**演讲者**：大家在这里可以看到——虽然这是一个模拟示例——聚类算法需要经过一段时间的观察，才能真正确认当前场景中其实存在两位说话人。正因如此，如果我们依然沿用刚才提到的基于音频块（chunk-wise）的框架，从批处理切换到实时就会变得异常棘手。尽管算法最终确实能够收敛到正确的聚类结构，但大家也能发现，在对话进行的实际过程中，它不可避免地产生了一些识别错误。这种处理范式已经在开源工具包 diart 中得以实现——diart 是我们首席技术官 Juan Manuel Coria 的心血之作。基于这种方法，我们成功将系统延迟控制在 5 秒左右，但代价是日志错误率（DER）出现了轻微上升；之所以延迟在 5 秒左右，是因为每个处理块大约长达 5 秒，或者说音频块之间的步长至少是 5 秒。
Original English
**Speaker**: So uh you see here uh this is a fake example but it takes some time for the clustering algorithm to realize that there is actually two speakers and that's why it makes um you know switching from batch to real time quite difficult if we if we were to stay with the you know chunk wise approach that I just mentioned. it still managed in the end uh to find the right clustering but you saw that during uh the actual uh conversation it made some mistakes and so with this kind of approach uh this is actually implemented in the dieart open source toolkit which is uh a creation of our CTO Juan Manuel Ka and we managed to uh basically get slightly uh higher diorization error rate uh at around 5-second latency because the chunks are around 5 seconds long or at least the steps between the chunks are 5 seconds.
### 因果分割与不可避免的前瞻延迟
**演讲者**：于是我们对自己说：很好，这确实迈出了重要一步，但接下来的攻关目标，是必须设法降低这一说话人日志错误率，至少使其达到与离线批处理相同的性能水平；同时，这 5 秒的延迟对于诸多交互场景来说依然有些偏高。因此，我们接下来投入研发的核心方向，就是把原本基于分块的分割模式转变为更加彻底的流式架构，我们称之为“因果分割”（Causal Segmentation）。在因果分割中，系统不再等待整个音频块全部就绪才开始计算，而是以纯流式的方式进行持续处理。然而，这又引入了许多全新的技术难题。
Original English
**Speaker**: So we said okay uh that's great but the next step would be to manage to lower this diation error rate at least to be as good as a batch speaker derization and also this 5-second latency might be a bit too much. So um what we worked on next is basically uh turning this uh chunk wise segmentation into something a bit more streaming that we call causal segmentation where instead of waiting for the whole chunk to be available to process it basically pro processing in in streaming. So that that brings uh a lot more uh of other difficulties.
**演讲者**：以批处理模式下的说话人切换检测（Speaker Change Detection）为例。在离线批处理中，我们通常可以纵览整段对话的全部音频。因此，说话人切换检测可以通过这种方式轻松实现：沿着整场对话的时间轴滑动一个窗口，对比窗口左半段与右半段的声学特征并计算二者差异；在发生说话人交替的位置，就会形成一个非常明显的特征峰值。但在流式处理中，核心障碍在于你无法预知未来。因此，模型必须经历一定的时间积累，才能获得足够的置信度来做出判定。模型在未来方向上为了判定是否发生说话人交替所必须等待的这一小段时间，我们称之为“前瞻窗口”（Lookahead）。而这种延迟在物理上是无法彻底消除的——因为在对话进行到某一特定时刻时，除非你拥有极其强大的韵律（Prosody）预测模型或类似机制，否则你必须切实等待并观察是否有另一位说话人开始发声，才能确凿地做出切换判断……
Original English
**Speaker**: So uh let's focus for instance on speaker chain detection uh in batch mode. uh what we you we usually have access to the whole conversation. So speaker chain detection can be done like that. You slide a window uh around the course of the conversation. You compare what happens to the left of the the segment and to the right and you compute the difference and you get some nice peak where there is a speaker change. But in streaming uh the problem is that you don't have access to the future. So it takes some time for the model to actually reach uh you know confidence enough uh to um enough confidence to basically take a decision and this uh small amount of time uh that it needs in the future to to decide whether there was a speaker change or not. We call that the look ahead and that's basically a a a latency that we can't get rid of because if we at one particular point of time in the conversation you need to decide whether there is a speaker change uh unless you have a very strong modeling of the pro or something like that. You really have to wait whether someone else is starting speaking to
### 流式说话人日志的延迟挑战与 400 毫秒架构
**Alve**: 为了真正做出判定，这就会带来某种不可压缩的固有延迟，并直接叠加到整个流程中。因此，我们继续推进并落实了这项方案。在研究团队针对这一特定流式分割模型开展的大量研究支持下，我们成功采用了比分块模型大得多的模型，并引入了一些损失函数的设计。这里我就不展开细节了，会后我很乐意深入交流。最终，我们基本做到了在仅有 400 毫秒极低延迟的情况下，达到与批处理说话人日志相当的准确率水平。而这 400 毫秒实际上涵盖了方方面面，它还包括了客户端到服务端再返回的往返网络传输。因此，我想花点时间详细拆解一下这 400 毫秒的延迟构成：这 400 毫秒里面究竟包含了什么？
Original English
**Alve**: to actually take the decision. So that brings this incompressible latency that is added to the thing. And so we went ahead and implemented that and with basically lots of research on the research team on this particular segmentation streaming model we managed to actually using much larger model than the chunk wise one and also some implementation of loss function. I won't go into the detail. I'm happy to discuss that after that. But we managed to basically reach the same level of accuracy as the batch speaker diarization with a very low latency of 400 milliseconds. And this 400 milliseconds actually involves many things. It involves also round trip from the client to the servers and back. So I'd like to spend some time on the actual 400 milliseconds latency. So what's in those 400 millisecond latency?
**Alve**: 首先，整个流程始于客户端在本地进行的一场对话。你遇到的第一层延迟，基本上是因为对话被切分成了 100 毫秒的数据块。也就是说，你必须先等待 100 毫秒。假设在时间点 t，你必须等待 100 毫秒来构建出这个音频块。因此早在时间点 t，我们就已经产生了 100 毫秒的延迟。我这里列出的这项延迟本身就是 400 毫秒总延迟的一部分，而在平时大家所看到的各种指标数据中，这项开销往往并没有被计算在内。但实际上，它是一个必须纳入考量的重要因素。
Original English
**Alve**: So we start by a client having a conversation locally. The first latency that you get is basically that the conversation is split into 100 millisecond blocks. So you already have to wait 100 millisecond. Let's say at time t you have to wait 100 milliseconds to build this block. So already at time t we already have 100 millisecond latency. This latency that I put here is part of the 400 millisecond latency and it's not something that is often counted in the numbers that you can see here and there. But it's actually also something that is important to take into account.
**Alve**: 接下来的一步，我们在底层依托 Cloudflare 的 Durable Objects 技术，将这个分块发送到最近的 Cloudflare 服务器。这一步平均耗时约 20 毫秒，不过这完全取决于客户端所处的地理位置以及 Cloudflare 的网络状况。在这台边缘服务器上，我们接收多个音频流并将它们打成批次。我们将许多 100 毫秒的数据块聚合成批次，以便能够并行处理多路并发流。这一步非常迅速。一旦完成批次构建，我们就会将它们发送给模型，也就是由 GPU 提供核心算力的地方。
Original English
**Alve**: Then the next step basically is we are relying on the Cloudflare Durable Object technology to basically send this block size to the closest Cloudflare server. So this on average takes 20 millisecond but it really depends on where the client is and how good Cloudflare is basically. And on this server what we do is we receive many streams and we batch them. So we batch lots of 100 millisecond blocks so that we can process multiple streams in parallel. So this is very fast and then once we've built those batches we send them to model where the GPU horsework lives.
**Alve**: 对这些 100 毫秒的数据块进行分割处理时，假设批次大小为 32 个并发流——这里只是举个例子——仅模型推理大约就需要 50 毫秒。此外，你还必须把我在前面提到的 200 毫秒前瞻窗口（look-ahead）计算在内。也就是说，当时间推进到 t 时，神经网络模型实际预测的是 200 毫秒前所发生的事情。这些开销累加起来，与此同时，我们在并行处理中巧妙地找到了一个技巧：以完全异步的方式运行特征嵌入提取（embedding extraction）与聚类（clustering）。这样一来，这部分运算完全不会对预测产生任何阻碍或延迟。这就是为什么我在这里标为 0 毫秒。接着是返回的往返网络开销，大约需要 5 毫秒。最后，我们利用保存在 Cloudflare 中的上下文状态，将所有片段拼接缝合并序列化，再发送回客户端。综合所有环节，总共就是这 400 毫秒的延迟。
Original English
**Alve**: So basically for segmenting those 100 millisecond latency a batch size let's say of 32 streams let's say that's just an example here but it takes around 50 milliseconds just for the inference and also you have to take into account this 200 milliseconds look ahead that I mentioned earlier. So basically when you reach time t what the model the neural network does it predicts what happened 200 milliseconds in the past. So this adds up and then in parallel we find this trick of actually running the embedding extraction and the clustering in a totally async manner. So this does not delay the prediction whatsoever. So that's why I put it zero millisecond here and then we do the round trip back so five more milliseconds here. Then we basically using the state that is stored in the Cloudflare we basically stitch the pieces together and serialize it to send it to the client. So in total you get those 400 milliseconds latency.
### 延迟拆解与多维权衡取舍
**Alve**: 如果进一步深入探究，你会发现延迟其实并不只是单一维度的数字，这里交织着多种不同类型的延迟。首先是网络延迟，它高度依赖于客户端环境——客户端与网络的物理距离有多近、他们的网络连接速度有多快。这部分平均占用了大约 50 毫秒，但这属于动态波动的变量。其次是我刚刚提到的推理延迟，约占 50 毫秒。推理延迟同样存在一定的调节空间：显然，如果你增大批次规模，单次处理耗时就会增加，但整体单位成本会更低，因为你能在同一块 GPU 上并发处理更多的数据流；而如果你追求更快的速度，就必须调小批次规模，这样硬件成本就会上升。因此，推理延迟正是掌控成本与延迟权衡（cost-latency trade-off）的核心所在。
Original English
**Alve**: And if we go into more details so basically there are just not just one latency basically there are many latencies here. There is the network latency that basically really depends on the client how close the client is to the networks how fast their connection is. So that accounts for like 50 milliseconds on average but this can be moving parts. Then there's the inference latency that I mentioned the 50 millisecond. So same this inference latency you can tweak it a bit. Obviously if you make a higher batch a larger batch it will take more time but then your cost will be lower because you can process more streams with the same GPU but if you want to be faster then you have to lower the batch size and your cost will go up. So really this inference latency this is where you control a cost latency trade-off.
**Alve**: 再者就是算法延迟，它占据了整个延迟预算的大头。同样，它掌控着另一层全新的权衡取舍——准确率与延迟之间的平衡（accuracy-latency trade-off）。你在前瞻窗口中等待的时间越长，模型做出的预测判断就会越精准。因此正如我所说的，这完全是两者之间的权衡。当然，降低延迟的一个直观手段就是缩小音频块的大小，我们目前使用的是 100 毫秒；但这种改变会连锁波及推理延迟，因为单位时间内需要处理的音频块数量会急剧膨胀，进而导致整体系统权衡变得更加难以平衡。
Original English
**Alve**: And then there is this algorithmic latency that takes basically most of the latency here and same it controls a new trade-off. This is a trade-off between accuracy and latency. The more you can wait in this look ahead the better the prediction you can make. So this is really as I say a trade-off between the two and an easy way to reduce the latency would be to reduce the block size obviously here it's 100 millisecond but that has repercussion on basically the inference latency because you get way more blocks that you need to process and so yeah that makes the whole trade-off a bit more difficult to achieve.
### 现场实时演示与重叠语音挑战
**Alve**: 讲到这里，我想现在应该进行现场演示了。我想请我的联合创始人 Vanson 一起上台，我们简单聊几句。你可以拿这支麦克风，我来切换到演示界面，让大家切实感受一下 400 毫秒在实际体验中究竟意味着什么。关于现场的收音配置说明一下：虽然台上放着两支麦克风，但实际只是我的笔记本电脑在拾音并捕捉对话。
Original English
**Alve**: So yeah now I've now reached the point where I think it would be good to make a demo. So I'm going to ask my co-founder Vanson to join stage just so that we have a chat quickly and you can take a mic here and I'll switch to a demo so that you get really what 400 milliseconds really means in practice. So just about the setup even though we have two microphones here it's really my laptop that captures that captures the conversation.
**Vanson**: 所以你这是把底层机关全揭秘了，全被看光了。
Original English
**Vanson**: So you're revealing the tricks you see everything.
**Alve**: 是的。
Original English
**Alve**: Yeah.
**Vanson**: 大家好，很高兴见到各位。
Original English
**Vanson**: Hi everyone. Nice to meet you.
**Alve**: 刚才我的演讲感觉怎么样？
Original English
**Alve**: So how did my talk go?
**Vanson**: 相当棒！希望你分享的这些内容能让大家深入了解我们所做的事情。我一直都很感叹这背后的技术难度有多大。用起来好像很顺理成章、轻而易举，但背后……
Original English
**Vanson**: Quite good. I hope you share and people have been learning things about what we do. I'm always amazed how difficult it is. It sounds easy when you use it but...
**Alve**: 没错，而且我们甚至还能支持重叠语音的分离。毫无疑问，这也是我们日常交流中的常态。
Original English
**Alve**: Yeah but we can even do overlapping speech. Absolutely. That's what we usually do.
**Vanson**: 通常有 20% 的语音是相互重叠的，要准确识别并区分出来确实非常具有挑战性。
Original English
**Vanson**: 20% of the speech usually is overlapping, which is kind of crazy to detect.
**Alve**: 对，在我们每次开会的时候他都老打断我，这简直是个完美的测试场景。
Original English
**Alve**: Yeah, he keeps interrupting me in every meeting we have. It's an ideal effect.
**Vanson**: 不知怎么回事，后台的 Raul 好像说了什么话，因为我们界面上检测到了“说话人 02”。
Original English
**Vanson**: For some reason, Raul says something in the backside because we found speaker 02 here.
**Alve**: Raul，请别在后台插话呀。
Original English
**Alve**: Please don't back channel, please, Raul.
**Vanson**: 好的，那我就先下去了，谢谢大家。
Original English
**Vanson**: Okay. I leave you there. Thanks.
### 总结要点与闭幕致辞
**Alve**: 好的，大家现在应该对 400 毫秒的实际响应感知有了直观概念。我的演讲也基本接近尾声了。希望大家能从这次分享中有所收获。回顾今天的内容，我认为有三点最具价值的核心结论，欢迎大家与我探讨：第一，我希望让大家认识到，离线批处理与实时流式说话人日志本质上是两个完全不同的机器学习问题。它们面临的系统约束截然不同，即便输入与输出的形式表面上一致，其求解途径也必须采用不同的架构与方案。第二，我想着重强调的是，延迟从来不是孤立的单一数字，它受到多种复杂因素的共同制约。大家对此可能有所了解，但在说话人日志领域中，核心延迟主要源于算法层面的前瞻机制，这是由该任务本身的设定所决定的。我们研发团队在攻克这一挑战上付出了巨大努力，我为这项成果感到无比自豪。第三，在延迟、准确率和算力成本之间，存在着丰富的权衡调节手段。非常感谢大家，我的演讲到此结束。如果大家有兴趣，我很乐意在会后与大家进一步交流，谢谢！
Original English
**Alve**: So you get the idea of what basically 400 milliseconds feels like. And yeah, so I've reached almost the end of my talk. I hope you learned a few things in this talk. And at least the three things that are I think interesting you'll tell me about this talk is that I tried to convince you that batch and real time diarization are actually two different machine learning problems. They have very different constraints. Even though the input and the output are the same solving it needs different approaches. Wanted to highlight as well that latency is just not one number. It really depends on many things. I mean you all knew about that but in particular that for speaker diarization you see that the main latency is coming from the algorithmic part because of the way this speaker diarization task is set up and yeah very proud of the work that the research and tech team did to handle this and yeah there are many ways of trading between latency accuracy and cost. And yeah so thank you very much that's the end of my talk and happy to have a chat after if you want. Thank you.
### 上午场结语与下午议程预告
**Host**: 非常感谢 Alve！让我们再次为 Alve 献上热烈的掌声！太精彩了，好的。现在正好告一段落，这是今天上午的最后一场演讲。接下来大家可以前往展区，我们由 Sierra 赞助了午餐，让我们感谢 Sierra 的大力支持！非常感谢他们为我们提供能量补给，让大家在这场盛会中保持活力，在下午以更加充沛的状态回来，因为下午我们还有极其丰富的内容要探讨。我们将在下午 2 点准时回到这里，届时将有一场 Mistral 与 Nvidia 之间的炉边对谈，这绝对是不容错过的重磅环节。那么我们下午 2 点见，祝大家午餐愉快！女士们先生们，有请本次 2026 巴黎 AI 工程师大会的主持人——来自 Replit 的开发者关系工程师 Raul Chevrey 登台！
Original English
**Host**: Thank you so much, Alve. Let's give it up one more time for Alve, please. Nice. All right. Okay. So, this is a good moment because this was the last talk of this morning. Okay. So, please you can go to the expo. We have a lunch that is being sponsored by Sierra. So, shout out to Sierra, please. Yeah, thank you for offering us some fuel, you know, to get going with this event and to come back stronger in the afternoon because we still have so many things to talk about. We're going to be back here at 2 p.m. We're going to have a fireside chat between Mistral and Nvidia. You should not miss this at all. So, I'll see you here at 2 p.m. All right, enjoy your lunch. Ladies and gentlemen, Please join me in welcoming to the stage your MC for the AI engineer Paris 2026, developer relations engineer at Replit, Raul Chevrey.
**Raul Chevrey**: 好的，我们回来了！太棒了，祝贺大家坚持到了这里！大家已经进入了今天大会的最后冲刺阶段，也就是 2026 巴黎 AI 工程师大会的第二天。非常感谢大家重新回到会场。接下来我们有一场极其精彩的圆桌讨论……
Original English
**Raul Chevrey**: All right, and we're back. Okay. Well, congratulations to us all, right? You guys are making it for the final stretch of today's conference, day two of AI engineering Paris. Thank you so much for coming back here. We actually have a great panel that
### 开场与嘉宾介绍
**主持人**：我们马上就要开始了，大家感觉怎么样？好的，你们现在应该都很清楚流程了。来吧，我们从昨天就已经开始了，所以必须把现场的热情调动起来！今天我们请到了两位重量级嘉宾来到台上。请大家和我一起，用热烈的掌声欢迎 Mistral 的算力副总裁（VP of Compute）Yan Leuger，以及来自英伟达（NVIDIA）的高级解决方案架构师主管（Senior Manager Solution Architect）Adolf Hohl！让我们热烈欢迎他们！
Original English
**Host**: We're going to start in a second. But how are you feeling? Yeah. No, no. Okay. You know the drill by now. Come on. We've been doing it since yesterday. So, we got to bring up the energy. Okay. So we have two gentlemen here that are going to come on stage. So please I would like you to join me in welcoming to the stage VP of compute at Mistral Yan Leger and also senior manager solution architect at Nvidia Adolf Hohl. So please let's give it up for them. All right.
**主持人**：对了，我差点忘了说明我们今天的主题——我们今天将要探讨的是“构建下一代 AI 背后的基础设施”。我们将以稍微带有一点未来视角的眼光，去探讨为了构建下一代人工智能到底需要些什么。非常感谢两位能来到这里，Adolf、Yan。Yan，你去年也参加了 AI Engineer 大会，你还记得吗？感觉那仿佛已经是十年前的事了。
Original English
**Host**: And I kind of forgot to say that. Um, yeah, we're going to be addressing like building the infrastructure behind the next generation of AI. So, we're going to have like a little bit of a futuristic view of what's needed to build the next generation of AI. So, thank you so much for being here, Adolf, Yan. Yan, you were here last year at AI Engineer. Do you remember? It feels like it's been 10 years ago.
**Yan Leuger**：过去这一年确实发生了翻天覆地的变化，比如在这期间我们公司就被收购了。
Original English
**Yan Leger**: Kind of a big year went by like our company was acquired in between.
**主持人**：确实发生了太多的事情。
Original English
**Host**: A lot has happened.
**Yan Leuger**：去年我们还在一起筹办这届大会，而现在我们又再次做到了。
Original English
**Yan Leger**: Organizing like this conference together last year. And now we're making it again.
**主持人**：没错，正是这样。但真的感觉……在 AI 的时间尺度里，这一年的时间感觉就像过了几个世纪一样漫长。
Original English
**Host**: Yeah, exactly. But it really feels like yeah, one year in AI time, it feels like ages ago.
**Adolf Hohl**：真希望我去年也能在这里。
Original English
**Adolf Hohl**: I wish I would have been here as well.
**主持人**：明年一定邀请你来！
Original English
**Host**: Yeah, you're invited next year.
### 从沙箱到智能体工作流：AI 基础设施的演进
**主持人**：太棒了。那么 Yan，去年你站在这个台上的时候，分享的主题是“为智能体时代的未来构建基础设施”。你能为我们梳理一下，在这过去的一年多时间里，究竟发生了哪些变化吗？
Original English
**Host**: Yeah, that's nice. So, Yan, when you were here, you were here on stage, you were talking about building for the future of agentic era. So, can you walk us through what do you think has changed in the interim?
**Yan Leuger**：好的。其实在过去的两年里，我一直在探讨 AI 基础设施，甚至比这还要久。去年我的重点特别放在了沙箱（sandboxes）上，以及它如何从根本上开创了一个全新的基础设施品类。如今，我们正在亲眼目睹它的实际落地。我认为过去这一年的关键变化就在于：当时我们还处在智能体时代大潮的起步阶段，各项技术的采用才刚刚拉开序幕；而到了今天，已经有大量的软件直接由智能体编写，智能体工作流整体上实现了极高的渗透率。
话虽如此，GPU 归根结底仍然是核心。不过现在除了 GPU 基础设施之外，CPU 也开始加入到这套组合当中。我认为这是此前几年与当下之间的一个关键区别。所以我们依然在朝着同一个方向演进：大模型训练并未止步，在机器学习领域，一方面我们仍在持续训练业界前沿的尖端模型；另一方面我们也在进行大规模的推理。而智能体时代的到来，实际上进一步引爆了推理算力的消耗，这也意味着整个基础设施技术栈变得比以往更加复杂了。
Original English
**Yan Leger**: Yeah, I think like over the last two years I've spoke about infrastructure for AI. I mean, and over actually like more than this now. Last year was specifically a focus on sandboxes and how basically it's creating a new category of infrastructure. And we're seeing it in action now. I think that's what happened like the one year we were in the beginning of this motion of the agentic era and like the adoption kicking off. And now a lot of the software is written by agents, and we have a huge penetration of the agentic workflows overall.
The but the GPUs are still like the essence after all. I mean like we have in addition to the GPU infrastructure we have CPUs which play into the mix now. That's a key difference between the previous years and now I think. So we're still going in the same direction like training large models it didn't stop. At ML we're still training like state-of-the-art models on one side, we're doing large scale inferencing and the agentic era is actually like kicking off even more the inference consumptions I think, which means the stack is even more complex than before.
### 算力边界：CPU 角色、MoE 架构与新模态仿真
**主持人**：所以你的意思是，去年你提到了沙箱，并且见证了这类基础设施的兴起，同时看到了负载的持续增长，这在当今世界毋庸置疑。那么 Adolf，从硬件视角以及英伟达的角度来看，你认为在我们与下一代 AI 之间，目前横亘着哪些关键挑战？
Original English
**Host**: So you're saying yeah, you were talking about sandboxes and you've seen the rise of that type of infrastructure. So you're seeing like the progress in the load and I think there's no doubt in this new world. What do you think is standing between us and the next generation of AI from a hardware perspective and from Nvidia's perspective?
**Adolf Hohl**：如果你在这个行业里待得足够久，过去大家谈论的通常就是算力（compute）、网络（networking）和存储（storage）。但实际上，鉴于如今智能体工作流占据了如此主导的地位，那么问题来了——这些工作流究竟应该在哪里执行？我认为在当下的各种讨论中，人们往往忽略了这一环：智能体工作流到底跑在哪里？其实这是一个 CPU 的问题。
在模型开发层面，这一点可以说是非常明显的。我们有一些既定的成熟架构，它们具备独特的特定需求，而这些需求在根本上驱动了下一轮创新周期中基础设施与芯片的设计方式。这其中无疑受到了混合专家模型（Mixture of Experts, MoE）的深刻影响，因为 MoE 对通信能力提出了极其严苛且独特的要求。
如果我们再把目光投向新的模态，我认为整个社会和产业界在数据利用上已经做得非常出色了。我们基本上已经训练了全世界所有的书籍，很快我们可能还会训练所有的电影和多模态数据。但这绝不是探索的终点。我认为未来还会有其他维度的突破需要去尝试，比如说进入具身智能与机器人领域，它必须具备即时反馈机制。而提供这种闭环反馈，本质上是一个物理仿真的问题。因此，我认为这将会带来崭新的挑战。我们目前还远没有达到终点，这是一段令人极其兴奋的旅程。
Original English
**Adolf Hohl**: When you are longer in this business, then you used to speak with compute, with networking and storage. And actually since this agentic workload is so dominant, well where is it going to be executed? And I think often these days in this equation it's forgotten: where does the agentic workload go? And that's a CPU question.
And that is I would say pretty obvious if we speak in model development. So we have certain established architectures, they have particular needs which drives basically how the infrastructure and the chips are designed for the next innovation cycle. And they are certainly influenced by mixture of experts, and with its unique demands of communication.
If we look into new modalities, I would say we have I think the society and the industry has made very good use of data. We trained on every book in the world. Likely we soon train on every movie and every multimodal. But that's not the end of the game. I think there are other things coming which need to be tried out from let's say if we go into this robotic domain, it needs to have feedback. So giving this feedback is a question of simulation. So I think that will pose new challenges. So we're not there yet, it's a very exciting journey.
**主持人**：我想对你刚才提到的“混合专家模型（MoE）”做个进一步的探讨——这基本上就是让我们能够构建超大参数规模模型，同时又能在推理阶段高效运行的一种方式，对吧？
Original English
**Host**: So I just want to double click on something you mentioned: mixture of experts. And it's just a way for us to have very, very large models, but who can be efficiently being used at inference time, right?
**Adolf Hohl**：是的。
Original English
**Adolf Hohl**: Yes.
### 芯片演进与软硬件协同：打破训推割裂的资源池
**主持人**：好的。两位如果愿意的话，接下来我想深入探讨一下底层芯片硬件。大家现在都知道了英伟达最新一代的芯片架构 Rubin，目前已经全面投入量产了，对吧？官方的一些宣传宣称 Rubin 的能效提升了 10 倍，同时所需的 GPU 数量也减少到了原来的四分之一。随着 AI 的演进，我们拥有了更高效的系统来训练混合专家模型。面对这样的迭代节奏和规模，负责构建 AI 产品的工程技术团队应该如何思考下一代架构的设计？
Original English
**Host**: Okay, all right. If gentlemen if you'd like to follow me, I would like to talk a little bit about silicon hardware. And so yeah, we know about Rubin, the latest generation of Nvidia chips, and it's now in full production, right? So yeah, some of the claims are like that Rubin is 10 times more efficient and it's four times fewer GPUs also. So as AI is evolving, like we're getting more efficient systems to train a model of experts. So how should teams, you know, how should these engineers be building AI products thinking about the next generation of architecture given this cadence and this scale?
**Adolf Hohl**：我的建议是尽量紧密地跟上这股浪潮，因为我认为整个行业目前都正处于一个高强度的学习周期中。想当年我刚加入英伟达的时候，当时的芯片架构还是 Pascal 和 Volta，也许台下很多人甚至都没听说过。但如果谈到数据中心经济学与效率——正如你所强调的生成更多 Token、让 Token 单价变得更便宜等等——我们在那一两代芯片时就已经开启了技术演进路径，距离现在大概是七到八年前，而这条路线一直在延续。
因此，我们需要顺应这一规律来开展工作，软件层同样也在遵循这些设计范式。软件和硬件是紧密协同、相辅相成的。为了让硬件真正发挥价值，你必须通过软件手段去充分利用它。所以当你想要运用最新一代的技术创新时，就会面临一些依赖关系。而对于提供大规模推理服务的从业者而言，这些显然都是关乎成本与商业回报的经济学考量，必须在全局层面实现最佳杠杆效应。整体上，我们需要优化技术栈的每一个层级。
最近我们观察到的一个显著趋势是：过去人们习惯于将训练基础设施和推理基础设施截然分开、各自独立规划；而我认为，出于经济效益的原因，这两者正在走向深度的融合。因为如果你划分出固定的资源池，它就是静态的，而在每一个静态资源池中，必然都会存在一定程度的算力闲置与资源浪费。
Original English
**Adolf Hohl**: I would say try to follow as best because the entire industry I would say is on a learning cycle. So when I joined Nvidia, there was an architecture called Pascal and Volta, maybe you don't know about that one, but if we talk about data center economics and efficiency, what you also stress with more tokens, cheaper tokens etc., then we started a technological path in that generation, and that is now seven or eight years ago and that continues.
So we need to work with that, and the software follows these paradigms. So software and hardware goes hand in hand. And in order to be of use, you want to use it by means of the software. And so there are some dependencies when you say I want to make use of the latest innovation. For someone who provides inference at scale, obviously those are economic arguments and they need to be leveraged at best overall. We need to optimize all layers of the stack.
What we see recently is that we used to think in infrastructure for training and inference, and I would say that is going a bit more together also for an economic reason, because if you make a pool it's static, so in each pool you have some waste.
**Adolf Hohl**：因此，我认为这种割裂的状态正在逐渐消失，我预期这两个领域的基础设施将会越来越紧密地走向融合。
Original English
**Adolf Hohl**: And so that is I think something which is go away, and the infrastructures for both domains I expect goes more together hand in hand.
**主持人**：非常有意思，明白了。
Original English
**Host**: Interesting. Okay.
**Yan Leuger**：是的，我非常赞同 Adolf 关于推理实际使用模式的观点。其实如果大家回顾一下我两年前做的一些演讲，当时我就已经在探讨针对不同业务场景采用不同基础设施的话题了。
训推集群所面临的物理要求截然不同：在训练端，你需要的是绝对顶级的极致性能；而在推理端，工作负载的容忍度相对要高一些，它们并不一定非要依赖最新一代的硬件。在工程实践中我们也确实能看到这种情况：对于模型训练，我们可能会部署最新一代的硬件；而对于推理服务，我们完全可以使用前几代的硬件设备。
这意味着，当我们在物理层规划和设计基础设施时，必须时刻牢记：这类基础设施的实际用途是会随着时间推移而不断演进的。举个典型例子，你上线了一个算力集群，前两年可能会将其专门用于某一项特定任务（如大模型训练），而在第三、第四年则转而将其用于另一种用途（如日常推理）。正如 Adolf 所提到的那样，这带来了极大的资源灵活性。在理想状态下，你甚至可以做得更加动态——让整个训练基础设施能够根据实时负载，动态地复用并承接推理任务。要实现这种灵活性，归根结底需要在编排调度层依赖极其大量的软件能力来支撑……
Original English
**Yan Leger**: Yeah, I do agree with Adolf on the usage of the inference. There is actually two kind of—I mean in the past if you look at some of the talks I did two years ago I was already speaking about different kind of infrastructure for different purpose.
Training clusters don't have the same physical requirements. They need the highest performance on the training side. And inference workloads are a bit more tolerant. So they don't necessarily need the latest generation of hardware. And we see this in practice where for training we might use the latest generation of hardware, and for inference, we can use older generation of hardware.
Which also means that when we design infrastructure on the physical side, we need to keep in mind that the usage on this kind of infrastructure is going to evolve over the time. You might deploy a cluster typically and get the two first years used for one purpose and the three four years used for another purpose, as Adolf mentioned in terms of flexibility. And ideally you can even go be more dynamic and get your training infrastructure to be reused dynamically for inference. And so this is something which is a lot of software at the end in terms of orchestration to...
### 训练与推理基础设施的分流与统一资源池
**Yan**：……能够动态地重新分配各个工作节点（workers）。
Original English
**Yan**: ... be able to reallocate dynamically the walkers.
**主持人 (Host)**：好的，非常有意思。这么说来，如果我们理解正确的话，随着时间推移，我们在后续最终会需要两种截然不同的基础设施——一种专门用于推理，另一种专门用于训练。因为两者的需求存在本质差异，推理并不一定非要使用最新一代的硬件。
Original English
**Host**: Okay, very interesting. So, we'll need eventually two different types of infrastructure for um for inference and for training down the line if I understand it correctly because the needs are different because we don't need necessarily the latest type of hardware.
**Yan**：实际上从长远来看，我们可能正在走向一种更加收敛的模式。一方面，确实存在专门构建的推理基础设施，如果你完全只为推理而构建它，你确实可以容忍不采用最新一代的硬件；但这也取决于你所训练或运行的模型类型。正如我们所知，如今的模型规模和形态各异。比如训练一个拥有 1 万亿（1 trillion）参数的模型，跟训练一个 80 亿（8 billion）参数的模型完全不可同日而语，对吧？在这两种情况下，你所需的训练基础设施是截然不同的。因此从长远角度来说，我认为我们在某种程度上反而是更加收敛的——我的意思是我们构建一个全局统一的算力资源池（global pool），然后在不同类型的训练任务与推理任务之间进行动态重新分配。
Original English
**Yan**: So we I mean we actually were more converging maybe I mean we have specialized inference in infrastructure on one hand which uh if you build it like just for inference uh you can like tolerate like having not the latest generation of hardware it also depends on the kind of models you're training like models now are as we know are old of um different sizes and shapes. If you uh train a 1 trillion parameter model, it's not the same as like an 8 billion billion parameter model, right? You don't need the same training infrastructure in both cases. Um and actually we're more I think like converging on the long term in a way. I mean on having a global pool that we reallocate uh between the kind of trainings uh and inferencing.
### NVL72 机架系统与机架级计算架构
**主持人 (Host)**：好的，很有启发。接下来的问题是想请教 Adolf。我想聊一聊 NVL72，不过在深入探讨之前，您能否先向现场可能还不太熟悉的观众解释一下，究竟什么是 NVL72？
Original English
**Host**: Okay. Interesting. Um okay. So the next question is to for you Adolf. Um, I'm going to talk about uh NVL72, but before I I dive into that, can you please explain what NVL72 is to the audience maybe who are not familiar with?
**Adolf**：如果大家还不太熟悉的话，我先做个对比：当最初涉足人工智能领域时，我当时的入门成本大概只有 150 美元，买一块 GPU，搭一个工作台，就能参与到 AI 的研发当中。而如今，一切都以极其惊人的速度膨胀发展。我想说，目前像 Mistral 这样面临的最基础的问题，都已经不可能塞进单台服务器里了。与此同时，硬件必须高度契合软件所提出的严苛要求，因此多个组件必须以最经济、最高效的方式协同工作。而高效的核心，在于通信必须非常非常快。正是出于这个目的，我们设计了 NVL72——一个整机架系统（system in a rack）。它与传统系统的核心区别在于，NVL72 内部组件之间的互联带宽和紧密程度，远远超越了 TCP/IP 网络所能达到的水平。如果你想运行现代的 AI 工作负载，这种架构是绝对必不可少的。可以说，它是目前应对高度并行化训练任务中最具经济效益的解决方案。
Original English
**Adolf**: In case you're not familiar, um, AI is in the meantime, when let's put it this way, when I started, my entry cost to AI was, I think, $150 with a GPU, I had the workbench, I could participate. Now, everything grew. Um, fascinating speed. So uh I would say that the smallest problem what Mistral has I think it's it's it's not going to fit on a single server anymore and uh um so at the same time uh the hardware pays attention to what is requested by the software and so uh multiple components need to work together in a most um economic and efficient way. Efficient means communication needs to be very very fast. And for that purpose um we designed NVL72 which is a a system in Iraq. And u um the the difference to a classic system is um that NVL72 is an interconnect between them much closer than than anything of TCP IP. And this is uh necessary if you want to run the modern workloads. Um well and uh um I would say it's the most economic way to address uh highly parallel uh trainings at this point.
**主持人 (Host)**：明白。所以 NVL72 是一个整机架系统。那么这对技术栈的其余部分意味着什么？这是否意味着我们正在迈向“机架即新一代服务器”（the rack being the new server）的时代？
Original English
**Host**: Okay. So yeah. So uh Nv72 is is a rack system and um so yeah, what does that mean for the rest of the stack? Does it mean that uh yeah, we're headed towards the rack being like the new server?
**Adolf**：英伟达有一句名言：“数据中心就是计算机”。如果站在更高的视角来看，当你面对具体的工作负载，需要在各个计算单元上调度各种任务作业时，最经济合理的调度方式是什么？NVL72 本身就是一个不可分割的任务分配基元（unit of assignment）。从经济学角度来说，没有必要去把它从中间强行切分；虽然你确实可以这么做，但那样做绝对会脱离该系统原本的最优性能平衡点（sweet spot）。因此，我非常赞同你刚才的总结——计算机的定义实际上已经从单台独立系统升维到了机架级规模（rack scale）。
Original English
**Adolf**: Well, there's a saying from Nvidia that the data center is the computer. So but uh thinking even bigger um uh but uh when you think about the workload you have to schedule various uh jobs on the units then um what is the most economic way of doing it and the NVL72 in itself is a unit of assignment and um so u economically there's no need to say I'm cut it in the in the in the middle you can do that um but you're definitely leaving the sweet spot um of that uh of that system so in terms of I I would uh assign your your um your summary that it the computer lifts basically from a single system to a rack scale.
### 异构算力演进与前沿模型训练
**主持人 (Host)**：是的，为了满足未来的需求，系统体量势必会变得更加庞大。好的，那么我想请教 Yan 一个问题：因为你们身处算力层，承载着海量的推理需求。从芯片或硬件的角度来看，您认为在不久的将来，你们需要些什么来更好地服务客户？
Original English
**Host**: Yes, it's it's going to be just uh like yeah just a bigger systems to serve our needs moving forward. Um, okay. So, yeah, I have a question for you, Yan. Uh, since I mean like you're you're at the compute level, so you serve a lot of uh inference. And what do you think um that you will need moving forward from maybe from the chips or from a hardware perspective in order to maybe better serve your customers in let's say in the near yeah in the near future?
**Yan**：这是一个挺微妙且复杂的问题。我们在硬件使用上有一种固定的模式：对于训练前沿模型（frontier models），我们一定会采用最新一代的硬件。这是因为前沿训练普遍需要最新架构所带来的显著能效提升；如果你想保持在前沿水平，对更强算力的渴望是永无止境的。因此总的来说，我们采取的是一种混合策略：我们不仅部署像 NVL72 这样配备超高性能互联专用于模型训练的系统，同时也进行专门针对推理的硬件部署，而推理集群所配备的系统可能完全不同。实际现场的现实情况是，基础设施的多样性正在与日俱增。因为本轮 AI 浪潮已经持续了数年，我们开始拥有不同世代的硬件共存，并逐步进入类似于过去 CPU 基础设施的迭代更新周期。虽然 AI 芯片在架构设计上与 CPU 完全不同，但硬件层面上同样存在长期的轮换更新，你必须应对这种异构硬件环境。总的来说，芯片制造商和英伟达正在规划和设计下一代产品，而我们将借助这些更强大的硬件，训练出性能更加卓越的模型。
Original English
**Yan**: It's uh I think it's well it's a it's a tricky question like uh we I mean we have a way of consuming hardware which is like we take the latest generation of hardware for training frontier models. So which is like that's uh what we need in general they they provide significant gain of efficiency and like to be at the frontier you actually still need it's still like uh you still want more power. Um so in general we have this this mix and and we don't only deploy NVL72 systems actually we do have a mix of both um system which is which with this high performance interconnect for training uh but we do also some deployments dedicated to inference and they might not be equipped uh with exactly the same system. Um so I think like we have um the the reality of the field is like our the diversity of infra infrastructure is increasing because um the AI wave has started like several years ago. So we are starting to have like different generation and getting to actually a cycle of infrastructure which might actually resemble CPU in a way. Um the designs are completely different. uh but you do have a long-term like rotation in terms of hardware and you need to deal with this heterogeneous hardware. So um yeah I think in general like u um the chip manufacturers and Nvidia are designing what is like coming next and uh we're going to leverage this to to to train even more performant models.
### 推理经济学与智能体工作流驱动的爆发
**主持人 (Host)**：好的。既然我们谈到了推理，如果各位不介意的话，我想进一步探讨一下推理的经济效益问题。当英伟达谈论所谓的“AI 工厂”（AI factories）时，衡量指标往往是每秒每美元每瓦特生成的 Token 数量（tokens per second, per dollar, per watt），对吧？我们希望以最快的速度提供推理服务，同时要求成本最低、能耗最小。但正如刚才提到的，训练和推理往往使用不同的基础设施。那么在您看来，从什么时候开始，我们能够真正凭借推理带来的商业营收证明这些投资的合理性？而不是仅仅停留在“我们目前建设算力主要还是为了训练”的说法上？什么时候我们才能证明推理业务创造的收入，足以支撑并兑现此前庞大的基础设施建设投入？我看……
Original English
**Host**: Okay. Yeah. Um okay I think uh I want to move on since we're talking about inference I would like to talk a little bit about inference economics if uh if that's okay with you guys. Okay. So, um, uh, yes. So, when Nvidia basically talks about, you know, AI factories, you always measure with tokens per second, per dollar, per watt, right? Like we want the fastest way to serve inference. We want it at the cheapest cost and the lowest energy possible, right? Um, but yeah, you mentioned it's different infrastructure for uh, yeah, for training and inference. But what do you think at what point we're going to be able to justify the the the revenue of inference rather than just saying like um so we're building basically for for training mostly. But when when are we going to uh to be able to justify the the revenue for inference and say that yeah all that build out is not going to is going to materialize at some point. looking
**Adolf**：你在看着我呢。
Original English
**Adolf**: you're looking at me.
**主持人 (Host)**：是的，我正看着你。不好意思，如果大家刚才没看出来的话，这确实是一个抛给英伟达的问题。
Original English
**Host**: Yes, I'm looking at you. Yes. Sorry, it was a question for if uh nobody understood, but yes, it's an Nvidia related question and
**Adolf**：凡是你用来生产和创造商业利润的东西，其底层的经济规律是不会改变的。过去可能存在这样一种模式：你先采购系统用于训练，等下一代硬件推出后，再把退下来的老设备转去跑对算力要求相对没那么苛刻的推理任务。但无论如何，经济效益始终是必须衡量的核心要素。而现在，推理工作负载的需求正在呈现爆发式增长（going through the roof），市场需求比以往旺盛得多。我自己就在高频使用它，我非常喜欢用，而且无时无刻不在使用。我如今已经不再用传统的方式去进行搜索了，我想现场的许多人也一样。
Original English
**Adolf**: well, everything what you what you what you use for for producing and making money that there needs to be some some it the economics will not change for this one. So um it might have been uh a pattern in the past that you say I'm using training uh systems and uh when once the next generation is available I'm using that one for for inference for a less demanding workload. Um but still um the economics must be taken into account. Um and uh um now um I would say the inferencing workloads with the um it's going through the roof. Um and um there is there is much more demand. I myself I'm using it. I love to use it. I use it all the time. It it I don't I don't search in a classic way anymore likely many of you don't do this.
**Adolf**：我完全离不开它了，所以我每天都在留下大量的使用痕迹。正因如此，推理对我们来说也极其关键，我们在软件优化上投入了巨大的工程研发精力，持续不断地提升软件效能。推理如今绝对属于最核心、最高优先级的工作负载之一。因此我认为在当下这个时代，你从规划的第一天起，就必须把推理需求充分纳入考量。
Original English
**Adolf**: Um and I don't want to miss it. Uh and so I'm leaving a trace um of this one and uh um so inferencing is uh um is also very important uh for us because we spend a lot of uh engineering cycles in making the software better all the time. Um so it has a um it is really at a top workload and so I would say um these days um you need to take it into account um right from the first day.
**主持人 (Host)**：明白了。
Original English
**Host**: Okay.
**Yan**：是的。顺着这个思路来看，我认为在此之前确实存在一种风险或者说担忧，大家担心推理端的商业化规模迟迟无法落地。但在过去一年里，随着智能体工作流（agentic workflows）和相关工作负载的爆发式涌现，我们亲眼见证了推理需求的全面起飞。比如业界热议的一个焦点是：在软件工程领域，开发者究竟愿意为加速软件研发的各类 AI 工具投入多少预算？
Original English
**Yan**: Yeah. And so if you go in this root like I think there was a risk which was before which was um a fear because the inference side didn't materialize at scale and I think in the last year with the emergence of actually actually agentic workflows uh and workloads um we see inference taking off like I think like one one of the the thing which is discussed is like how much like for instance if you look at software engineering how much do you spend on like AI tools to accelerate your software engineering efforts
**Yan**：围绕这一话题，外界固然会有关于“Token 最大化使用”（token maxing）等各种角度的技术探讨；但不可否认的客观现实是，对于如今从事软件工程的任何人来说，AI 已经成为日常工作中必不可少的生产力工具。因此我们看到实际的营收正在源源不断地产生，早期基础设施投资的成果如今正在真正兑现商业回报。
Original English
**Yan**: and so obviously there are discussion around token maxing or whatever like to to on this angle but um there is also just a reality that it's a daily tool uh for anybody who is doing software engineering these days so um and we see the the revenue uh generation like uh from the the result of the previous investments are paying off now
**主持人 (Host)**：没错，所以推理正在全面爆发，而且在未来的发展中必然会持续激增。因此所有这些大规模的基础设施建设实际上……
Original English
**Host**: yeah so basically the inference is is taking off and we're going to see it's just going to be uh increasing moving forward. So all this buildout is actually
### 衡量智能：从追求 Token 数量到追求业务结果
**Host**: 这些算力在某个时刻终究会被推理所消耗，即便它们随时也可用于训练。不过，方才你谈到了“追求 Token 数量最大化（token maxing）”。在我看来，我们其实不那么信奉单纯去拼 Token 数量，而是更相信“追求业务结果最大化（outcome maxing）”。我们希望为客户提供真实的价值。我认为这正好引出了我的下一个问题：因为抛开成本不谈，或者说当你在量化成本时，归根结底这并不仅仅关乎成本本身，而是关乎它对最终使用者的价值，对吧？那么你们究竟是如何量化这一点的？我的意思是，跳出单纯粗暴的基础商品属性以及纯粹的经济学考量，我们该如何去量化“智能”？
Original English
**Host**: going to be used at some point by inference even though if it's for for training at any time but uh yeah you spoke about token maxing uh and um yeah I reflect like we we don't believe so much at uh at token maxing but we believe more um about um outcome maxing. So we want to provide value for the customer and I think this is a good segue for my next question actually because yeah despite or when you quantify you know the the uh like we were talking about cost etc but at some point it's not just a matter of cost it's a matter of value to to the people who use it right so h how do you guys actually quantify that so I mean beyond just raw bruter commodities ities and you know economics in this case. So how do we quantify intelligence? Uh
**Yan**: 这个问题是问我的，还是问……？
Original English
**Yan**: >> so is a question for me or is it for
**Host**: 这是问你的。
Original English
**Host**: >> this is a question for you?
**Yan**: 如何量化智能？我认为这是个棘手的难题。探讨如何量化智能，归根结底更多是在看业务结果。所以它具有极高的可变性。如果你把 AI 应用得当，有时仅凭极少量的 Token 消耗，就能创造出巨大的商业价值。我认为这也是我们在 Mistral 正在践行的理念之一：这绝不仅仅关乎规模或调用量。有些特定业务活动在比例上会消耗极其庞大的 Token，这就引出了一个核心问题——我们依然需要应用所有的效率手段和工程认知去优化这些流程，因为就最终产出的结果而言，部分工作负载消耗的 Token 数量可能是不合常理的。
因此我认为，整个生态系统在这一层面上大有可为。审视当前的现状，这不仅关乎基础大模型本身，不仅关乎模型训练，更关乎整套系统的运行框架与评测体系（complete harness），以及你究竟如何合理地利用这些 Token，从而在 Token 成本与最终创造的价值之间取得理想的投入产出比。至于底层的基础设施架构，我们所做的是对每一层进行极限优化。但归根到底，基础设施层只是在对外交付 Token；我认为真正的商业价值，实则是由技术栈上层捕获并体现的。
Original English
**Yan**: >> How do we intelligence? I think it's a tough one like how do you quantify intelligence is more about yeah it's about business outcome. So it's uh and it's highly viable. you can have like um with a low amount of token you can generate tremendous amount of of value if you apply AI right I think I mean it's one of the thing that we are doing at Mistro it's not only about volume there are some activities which are um consuming a lot of tokens uh proportionally and there is a question there is all the efficiency and all the actually knowledge we still knew to to apply to optimize this processes because some of the workloads might be actually like using an abnormal amount of tokens uh for the outcome. Um and so I think that's where um all of the ecosystem can deliver a lot of value. If we look at like the the situation and like it's not only about the M it's not only about training it's about like the complete harness and how do you actually like leverage the tokens properly uh to have a ratio of like token cost versus uh value. Then on the infrastructure layer, we're just optimizing each layer. Uh but we are just providing like the tokens. Uh the value is really in the uh it's captured by the higher levels of the stack I think.
### 推理时代的缓存架构与分布式演进
**Host**: 好。既然你刚才谈到了 Token 的消耗量，以及伴随系统框架与智能体不断攀升的 Token 需求，我想这也非常自然地引申出我对 Adolf 关于“推理（Reasoning）”的提问。确实，增加 Token 消耗量的一大途径就是引入推理机制，但这也是获得更好业务结果的一种极佳方式，对吧？那么在你看来，针对由推理智能体所产生的海量 Token 冲击，整个技术栈中哪一部分目前的准备是最不充分的？
Original English
**Host**: >> Okay. Um well since I think since you are talking about you know uh the consumption of tokens and the ever rising the the consumption of tokens and harnesses etc. So I think like um it's a good segue for me for my next question for Adolf about reasoning. So um yeah, one way to increase the number of tokens that we consume is reasoning, but it's also a good way also to to have a better uh a better outcome, right? So which part you think of the stack that is maybe the least prepared uh for all that uh you know tokens and that is generated by by reasoning agents.
**Adolf**: 我必须说，我对推理技术非常推崇。但它也是有代价的，这个代价就是我们会消耗更多的 Token。对每一位基础设施提供商而言，这意味着我需要足够的显存与内存来承载超长上下文；同时我需要强大的缓存（Caching）能力，以便将这些对话交互的复用价值发挥到极致。因此我可以想象，目前业内尚未在所有地方都部署最先进的技术，或者说还没完全穷尽一切可能的技术边界。
在我看来，我们当前正处于一个缓存技术愈发至关重要的阶段。在分布式系统中跨节点进行缓存至关重要——当前已经出现了非常明确的技术信号，表明整个体系正迈向分布式模型服务（distributed serving），而这一缓存范式对分布式服务来说极其关键。我想，如果我们明年再坐到这里交流，大家一定会重新审视缓存机制在这个方程式中所具备的核心价值。
Original English
**Adolf**: I would say I love it. It comes at a price. price is um we we consume more tokens. Um that means for everyone who provides infrastructure um I need the uh the memory to keep uh the the long contexts uh and I I need uh the caching um to take the best out of uh reuse of these conversations and uh um so I could imagine that uh um we haven't uh um deployed the latest uh or the out of possible everywhere. Um I see we are basically in in a phase where uh where caching um gets gets more important caching across um um in in a distributed systems um there are clear uh clear signs that uh it goes into u um distributed serving um and this caching paradigm is important to this one and um well I think uh um we if we sit here next year I think uh we will we will review um the value of caching in that equation.
### 欧洲 AI 工厂、最被高估与最低估的基建趋势
**Host**: 啊，非常引人深思。好，最后我还有一个问题想问你们二位。我们刚才提到明年还会回到这里，Yan 也向大家发出了邀请；不过希望五年后我们还能再次相聚于此。所以，你们认为未来会有顶尖的前沿模型（frontier models）在欧洲本土的“AI 工厂”中训练出来吗？这个问题你可以先来回答，Yan。
Original English
**Host**: >> Ah, super interesting. Okay. Um, I have one final question for the two of you. Okay. Uh, we said, yeah, we're coming back next year. You're invited according to to Yan, but we'll probably be come back coming back hopefully in five years from now. And, uh, um, so yeah. Do you think that we're going to have any frontier models going that are going to be trained in uh, European AI factory? This is you. You can start with this one. uh Yan and uh
**Yan**: 对，我认为毫无疑问。在 Mistral，我们已经在付诸行动了：我们明年将上线 200 兆瓦（MW）的算力容量，到 2030 年更将达到 1 吉瓦（GW）。而且其中相当大的一部分都在欧洲本土。我们正是在这套基础设施上进行训练的；事实上，当前最新的 Mistral 系列模型就已经在我们自己的基础设施上完成训练了。因此我相信在五年之内，欧洲必定会诞生本土训练出的前沿模型。
Original English
**Yan**: >> yeah yeah I think I mean we're acting on it at Mistro like we're bringing up like 200 megawatt of capacity next year and 1 gawatt uh until 2030 so and um um and a lot of it is in Europe so um and we are training on this infrastructure and actually like the current uh MR models are already trained on our infrastructure so I think in five years for sure we'll have frontier models Uh,
**Adolf**: 我们其实已经在对外发布了。
Original English
**Adolf**: >> we're announcing already.
**Host**: 太棒了。那么，在 2026 年，最被高估的基础设施趋势是什么？
Original English
**Host**: >> Okay, great. Okay, so what's the most overrated infrastructure trend in 2026?
**Adolf**: 哈哈，这也是问你的，也是问你的。突然想到的答案：“一切皆可自愈”（self-healing everything）。
Original English
**Adolf**: >> That's my love. That's for you. That's for you as well. Out of the blue, self-healing everything.
**Host**: 什么？好吧。那什么又是最被低估的趋势呢？
Original English
**Host**: What? Okay. And what's the most underrated one?
**Yan**: 我觉得在基础设施领域，这取决于我们审视的是哪一个技术层级。这是个很有难度的问题，但某种程度上说是“软件”。如果在纯粹的基础设施层面来看，“基础设施即软件”很可能是最被低估的方向之一——也就是用软件去赋能和盘活基础设施。
Original English
**Yan**: I think like uh in in infrastructure like it depends on which uh which uh layer we're we're looking at. It's a a tricky question but like software uh in a way. Um yeah I mean if you look at the purely infrastructure level uh infrastructure is software is probably like the the one of the underrated one uh software enabling infrastructure.
**Host**: 好的，这个回答非常精彩，我很赞同。非常感谢两位先生。我们本次对话就到这里，再次感谢 Adolf，感谢 Yan！请大家为 Adolf 和 Yan 热烈鼓掌！
Original English
**Host**: >> All right that's a good answer. I'm happy with that. So with this thank you so much gentlemen. Uh this is it for for for this chat and uh yeah thanks Adolf and thanks Yan. Please let's give it up for Adolf and Yan. Oops. All right.
**Yan**: 现场的观众也是这五年之约的见证者，对吧？
Original English
**Yan**: >> Well, the audience is part of the five years so mortal exercise, right?
**Host**: 绝对如此。是的，五年后我们所有人都会回到这里。非常感谢大家！让我们再次以热烈掌声送给 Yan 和 Adolf！好的，接下来我们有请下一位演讲嘉宾登场。我们的下一位演讲嘉宾来自 Modal。女士们先生们，请热烈欢迎 Charles Frye 登台！
Original English
**Host**: >> Absolutely. Yes. We're all going to be coming back in five years. Thank you so much. >> All right. Let's give it up for Yan and head off once more, please. Okay. And with that, we're ready for our next speaker. Our next speaker comes from Moto. Please, ladies and gentlemen, please welcome to the stage Charles Fry.
### 低延迟大模型推理：突破 UI/UX 的全新范式
**Charles Frye**: 大家好，近来可好？刚才的出场音乐太棒了，我现在整个人热血沸腾，简直想当场跳一段有氧操。想要搞定低延迟推理，你确实需要极其充沛的能量——这不仅需要高瓦数的电力支撑，更需要坚韧的工程毅力。所以这个开场非常契合。
我是 Charles，在 Modal 负责推理工程。我们是一家基础设施平台公司。今天我想向大家分享的，正是我们为优化大语言模型（LLM）低延迟推理所总结出的实战指南。
不过，在告诉大家如何将低延迟推理做到极致之前，我想先聊聊究竟什么是低延迟推理，以及为什么它会如此关键。
最近有一款新发布的技术产品叫做 Jev。今天会场上大概每隔五分钟就会有人提到它。Jev 是一个决策模型，也是一种运行速度极快的通用零样本分类器（generic zero-shot classifier）。人们对它反响极为狂热，并且基于它构建了许多令人惊叹的 Demo。其中我在 Twitter（X）上最喜欢的一个演示是：你只要随手键入一个词，它就能立刻为你匹配并生成一组与之契合的色彩调色板。
制作这个演示的开发者 Matt DesLauriers 评价道：“天哪，这东西不仅极度廉价，而且速度快得惊人。这感觉就像是开创全新 UI 与 UX 交互范式的一次巨大飞跃。”
然而对于许多长期深耕机器学习和自然语言处理领域的资深从业者来说，大家的反应其实截然不同。许多人的第一反应是：“咦，我挺纳闷大家为什么会对这个东西兴奋成这样？”
这里最本质的答案在于：开发者把这种过去一直存在的智能能力，在同一时间做到了前所未有的更便宜、更快速。这种极致的低成本与高速度，瞬间解锁了过去无法想象的全新应用场景。如果开发者不用时刻提心吊胆担心触碰使用限额，或者不用再为了让界面上的一个图标转动一下就白白烧掉 1000 美元，更不用为了等那个图标开始转动而苦等 30 分钟——那么眨眼之间，呈现在你面前的产品想象空间将呈爆炸式增长。
顺便提一句，我差点忘了念赞助声明：没错，Jev 和 Typesafe 这款应用正是完全构建在我们的 Modal 云平台之上的。因此，这也正是我今天想协助大家弄明白的核心命题——如何在你们自己的业务中打造出这种体验。
在人们基于大语言模型序列推理所构建的各类应用中，Jev 大致归属于三大基础类别之一。而从我作为偏底层基础设施架构师的后端视角来看，你实际上需要对这些应用进行分类剖析，明确评判其性能的核心指标究竟是什么。低延迟性能在这里具有决定性的意义；正是性能的突破，才真正开启了诸如 Jev 乃至当年初代 ChatGPT 发布时所展现出的全新 UI 与 UX 范式……
Original English
**Charles Frye**: All right. How's it going? All right. Um, that was great intro music. I'm very pumped. I'm ready to do like a jazzer size or something. Um, and you need a lot of energy for low latency inference. Um, not just wattage, but also uh grit. So, it's an appropriate start. Um, so I'm Charles. I work on inference engineering at Modal. We're a infrastructure platform. Um, and what I'm gonna tell you about today is our playbook that we've developed for optimizing uh low latency LLM inference. Um, so but before telling you how to make it good, uh, I want to talk about like what low latency inf low latency inference is and why it's so important. Um, so there's this thing that got released recently called Jev. Um, that probably hasn't been brought up every five minutes uh, today. Um, so, uh, Jev is a decision model, a a generic zeroot classifier that works really fast. Um, and people have gotten very excited about it. People have built these really cool demos. One of my favorite from Twitter is you like type a word and you get a pallet associated with it. Um, and the um, the person who created this demo, Matt Deloreier, said, "Oh, this is very cheap and fast. It feels like a leap forward for creating new UX and UI paradigms." Um, and the response, I think, for a lot of people who've been in this field in machine learning and natural language processing for a long time was actually kind of different. It was like, "Oh, I'm surprised that everyone's so excited about this." Um, and the sort of like basic answer here is that they made this intelligence that was always there cheaper and faster at the same time. and all of a sudden that unlocked new use cases if you aren't sweating that like you might run up to a usage limit or you might like spend $1,000 just trying to make an icon spin um and you won't wait 30 minutes for that icon to start spinning then like all of a sudden there are more opportunities available to you um so um oh I forgot my corporate sponsorship statement yeah this is uh application uh Jeff and Typesafe building on top of our cloud platform So this is the kind of thing u that I'd like to help you figure out how to build yourself. Um and so Jev sort of sits in like one of three basic categories for the kinds of things that people build with uh with language model inference with sequence inference. Um and my sort of like infrared take thinking on the back end here um is that you kind of want to divide these out into what is the kind of like figure of merit for their performance. The performance is so critical here. It's what unlocks these UI UX u paradigms that something like you know Jev for the original release of chat GPT was able
### 推理工作负载的性能指标与低延迟聚焦
**Speaker A**：……从而打开此前完全不存在的全新可能性。因此这种性能表现是极其关键的。对于聊天机器人（chatbot）而言，核心的衡量指标（figure of merit）是每秒能够向单个用户返回多少个 token，也就是每用户每秒 token 数（tokens per second per user），对于决策模型（decision model）或者零样本场景也是如此。而最后一类则是所谓的高延迟推理（high latency inference）场景，例如那些可能针对后端海量数据运行的数据处理器，或者像 Reduct 这样的案例中可能需要处理成千上万、甚至数以百万计的 PDF 文档。它们有着完全不同的性能目标与衡量指标——关注的是每一美元能够处理几百万个 token，甚至数十亿个 token。这类场景属于高延迟推理，因此我们今天不展开讨论，而是把重点完全放在低延迟推理上。
Original English
**Speaker A**: ...to just like open up that was not there before. Um so this performance is super critical and for chat bots the sort of figure of merit is how many tokens per second can you get back to a user. So tokens per second per user um for a decision model or a zero. Um, and then the sort of final category, the sort of high latency inference category, these like data processors that might operate on a whole bunch of data from your backend or might consume in Reduct's case like you know thousands, millions of PDFs, they have a different sort of figure of merit, a different performance target, the number of millions of tokens that you can that you can process for a dollar or maybe billions of tokens. Um, so this is higher latency inference, so we won't talk about it. Um we're going to focus on the low latency inference.
**Speaker A**：在智能体（agent）这一类别中，我们一直在负责月之暗面（Moonshot AI）的 Kimi 模型的推理服务。直接拿开箱即用的方案——比如直接照搬 SGLang 或 vLLM cookbook 中的配置来跑——与你真正坐下来深入剖析具体工作负载并进行系统性优化、做大量的配置调优以及一些底层性能工程，这二者之间能取得的效果是有天壤之别的。在我们的实际案例中，我们不仅使每张 GPU 每分钟产出的 token 效率提升了约 5.5 倍，而且每用户的端到端生成速度也提升了约 3 倍——每位用户每秒能够获得大约 400 个 token，而不是原先的略高于 130 个 token 左右。这让我们能够通过 OpenRouter 等多种平台以及与客户的紧密合作，实现超大规模的模型推理服务。
Original English
**Speaker A**: Um so in the sort of agent category um we've been working on serving the uh Kimmy models from Moonshot AI and the like the difference between what you get from a sort of like offtheshelf let me take the description in SG Lang's cookbook or VLM's cookbook and serve it versus what you can get if you sit down and look at the workload and optimize um do a lot of configuration tuning do a little bit of uh um uh like performance engineering you can get um in our case we were able to get about you know five and a half times more uh efficiency tokens per minute per GPU but also three about three times faster per user. So um about like 400 tokens per second per user instead of a little over uh like 130 or so. Um and this has allowed us to like serve this at large scale through a variety of platforms like open router and in partnership with our customers.
**Speaker A**：在极致低延迟这一端，显而易见，正如你前面提到的，Jev 运行在 Modal 之上；此外，我们也撰文详细记录过我们如何与 Decagon 展开合作，为其语音智能体流水线中的智能体操作规程（agent operating procedures）——也就是负责实时决策的核心环节——实现尽可能低的极致延迟。那么接下来，我们就来具体聊聊这背后的整套打法，也就是我们总结出的一套优化实战手册（playbook）。
Original English
**Speaker A**: Um and then on the low latency side, obviously you mentioned that Jev is running on modal, but also um we've uh written about um uh how we worked with Decagon to get the like lowest possible latency for their like uh for their agent operating procedures that sort of do um this decision work um inside of their voice agent pipelines. Um so let's talk about how that's actually done. Um so the sort of like playbook we've developed.
### 推理优化第一步：读懂负载与硬件
**Speaker A**：优化的第一步，是你必须透彻理解自己所服务的具体工作负载（workload），以及它所运行的底层硬件（hardware）。一旦你对这两者有了清晰的全貌，才能着手开展具体的性能优化。工作负载界定了你必须完成的任务是什么，而硬件则是为你执行这些任务的底层承载实体；硬件会直截了当地告诉你：这件事情会很轻松，那件事情会非常困难，而另外某项操作的速度可能会比其他方式快上 10 倍。
Original English
**Speaker A**: Um so the first step is that you need to understand the workload that you are serving and the hardware that it runs on. Uh and then once you have a clear picture what that is then you can start optimizing the performance. The workload tells you what you have to do and the hardware is what sort of does it for you and it it says this thing will be easy. This thing will be hard. This thing will be 10 times as fast as this other thing.
**Speaker A**：为了透彻理解工作负载，我们首先需要确保大家的认知统一——因为我认为现在很多人基本上只是通过 API 来调用和消费 AI，并不一定会深入思考模型内部到底发生了什么。为了确保我们都在同一语境下：比如你向大语言模型推理服务器发送一段输入，比如“不可造出……（thou shalt not create）”。在 Prefill（预填充）阶段，模型会针对这一整段输入计算出内部表示，生成最初的几个 token；随后进入 Decode（解码）阶段，通过不断单步循环迭代生成后续输出，直到补全《沙丘》中《奥兰治天主教圣经》的那句戒律：“不可造出具有人类心智形态的机器（thou shalt not create a machine in the likeness of a human mind）”。在此过程中，系统会将这些中间内部表示保存在所谓的 KV 缓存（KV cache）中，以便后续步骤可以直接读取，而不必重新计算。
Original English
**Speaker A**: So in understanding the workload just to make sure we like I think a lot of people have been substantively consuming AI through APIs and not necessarily thinking about what's going on inside. So just to make sure we're all on the same page. Um we like you send in something like this thou shalt not create uh into a language model inference server. Um this uh the uh internal representation of the model is calculated on this input in the prefill phase. Um that generates a few tokens and then you iterate that over and over again in the decode phase to generate the outputs and complete the commandment uh from the orange Catholic Bible that thou shalt not create a machine in the likeness of a human mind. And along the way, you store those internal representations in something called the KV cache so that you can uh retrieve it later instead of having to recalculate it.
**Speaker A**：KV 缓存是一个极其关键的基础组件，尤其是当你想要提供低延迟的智能体推理服务时。如果你服务的系统更类似于 Jev，由于其工作负载特征和接收请求的结构有所不同，KV 缓存的相对重要性可能稍弱一些。但在典型的智能体工作负载中，你所看到的交互形态是这样的：用户的首个输入到达，LLM 给出响应；紧接着来自同一个用户的下一次请求中，会同时包含该 LLM 的响应以及最开始的用户输入；接着模型可能会发起一次工具调用（tool call），而工具调用的执行结果返回之后，又会将之前全部的历史上下文完整囊括进来。因此，在每一轮交互中，输入序列都在经历着迭代式、递增式的构建。
Original English
**Speaker A**: Um, and that's a pretty important piece especially if you want to serve low latency agent inference. If you're serving something that looks a little bit more like Jev, this is less important because of the sort of structure of the workload, the structure of the requests that you're going to receive. Uh so with an agent you see something like this where you have the the first user input comes in the LM responds and then later you get another request from the same user that has that LM response and their first input in it and then you get another maybe a tool call from the LM and once that tool call comes back that also is going to have the whole prior context in it and so you have this iterative construction of the input sequence turn after turn.
**Speaker A**：在这个领域以及大语言模型与人工智能的发展过程中，很多技术热点可能都只是昙花一现（flash in the pan）。但这种交互模式给人的感觉非常深刻、具有根本性，似乎是一种会长期持续存在的架构模式。本质上，用户、模型以及它所访问的外部系统，都在协同且迭代地构建这一段上下文，并随着时间推移创造出价值不断递增的沉淀物。因此我们可以预见，这种高度互动的上下文增量构建，将成为此类序列模型运行机制中不可或缺的核心部分。即便未来这些模型被植入到在你家中四处走动的机器人体内，或者全面转向接收视频输入并输出视频，它们的交互架构很大程度上依然会非常类似于这种形态。
Original English
**Speaker A**: Um so some things about this um like this field and about the way LM and artificial intelligence work are kind of like flash in the pan. Um but this actually kind of feels very deep and seems like something that'll stick around for a while. Essentially the user, the model um and the external systems that it's accessing are iteratively constructing this context and they're creating something of increasing value over time. And so we should expect these kinds of interactive um like constructions to be part of how these sequence models end up working. Even once they become, you know, something inside of a robot moving around your house or uh once they start taking in videos and outputting videos, they'll still probably look quite a bit like this.
### 硬件拓扑：高交互性无需超大规模 GPU 集群
**Speaker A**：好，刚才讨论的是偏长期架构层面的规律，现在我们来看一些相对聚焦当下、具备时效性的工程现实。一个好消息是：在当前阶段，如果你专注于高交互性、低延迟的推理，实际上通常最多只需要单机 8 张数据中心级别的 GPU。如果你参考过 SemiAnalysis 极其出色的基准测试工作——比如他们的 InferenceX 或 InferenceMAX 基准测试——这些图表数据非常密集，可能稍显晦涩，我已经挑出了最简明的一张呈现给大家。你可以看到，如果你追求的是单芯片极致的吞吐量（throughput per chip），你确实需要那种兆瓦级（megawatt scale，虽然可能没到整兆瓦）、体量极其庞大且在极紧密的 NVLink 域内互联的集群机器，动辄需要 40 张甚至 64 张 GPU 联动。
Original English
**Speaker A**: Um yeah, now for a little something that's a little bit more narrow in scope or momentary. Uh the good news is that right now um if you're focused on high interactivity, low latency inference, you actually can usually run on no more than eight data center GPUs. So if you check out the really excellent benchmarking work from semi analysis and their inference X or inference max benchmarks, um this is maybe a little hard to see um because they're very dense charts. This is like the least dense version I could pull out. Um, you can see that if you want the highest throughput per chip, you want these big like megawatt scale, not quite a megawatt, these really huge machines that have a bunch of GPUs in a in a tight domain with each other, an envy link domain. U, you need like what it says here, 40 of those GPUs, 64 of those GPUs.
**Speaker A**：但如果你的优化目标是极致的交互性，也就是追求每位用户每秒获得最高的 token 数（tokens per second per user），你的选型实际上应该完全落在图表下方的这个区间。仔细观察这块区域的数据点就会发现，它们其实只需要 4 张或 8 张 GPU。这意味着你根本不需要那些极其昂贵复杂的庞大拓扑，这在实际工程运维（operational win）上是一个巨大的胜利。
Original English
**Speaker A**: Um, but if you want to serve at the highest interactivity, if you want the highest tokens per second per user, you actually want to be all the way down here. Um, and if you look at the points that are down here, you can see that they have four or eight GPUs. Um, so you don't need um these fancy things, which is a huge operational win.
### 性能优化三步法：先抓大收益，再做 Host 端，后写 GPU Kernel
**Speaker A**：好了，现在我们已经搞清楚了我们所面对的工作负载，以及我们打算运行的硬件平台。接下来让我们思考如何展开性能优化。整体而言，优化的推进逻辑是：首先必须锁定那些能带来大幅收益的重大突破点（big wins）。其中投机解码（Speculative Decoding）就是最大的一项，它往往能带来 5 倍甚至更高的加速效果；量化（Quantization，降低数值计算精度）是另一项重大加速手段，不过它会牵涉到输出质量上的权衡。把这些重大收益斩获之后，接下来便是在 Host 端（主机 CPU 侧）进行一系列艰苦的中等收益任务攻坚——也就是传统的 CPU 系统性能工程，这类调优通常能带来 50% 到 2 倍左右的性能提升；你所处理的业务逻辑越特异（weird），Host 端的优化空间往往越大。直到把这些全部做完，你才应该把精力投入到编写自定义 GPU kernel 上——尽管那往往是大家最感到兴奋的领域。接下来让我们在高层面上快速过一下这些手段的具体样貌以及何时应当采用它们。
Original English
**Speaker A**: Um, okay. So, we've understood like the workload that we're thinking about and the hardware that we want to run on. Now, let's think about how we're going to optimize the performance. Um, so roughly the way that things work, um, is you want to target a bunch of big wins first. So speculative decoding is one of the biggest ones. Um you can get a five time speed up or more. Uh quantization reducing the precision is another big speed up has quality implications. Um once you get those big wins out of the way, there's like kind of a grind of mediumsiz tasks on the host side. Um like sort of regular CPU systems performance engineering stuff you got to do. Um that you wins 50% maybe as large as 2x. um the weirder the thing you're doing is the more wins they're probably going to be on the host side. Um and then only then do you spend your time you know working on uh GPU kernels and the sort of thing that like everybody gets most excited about. Um so let's go through that real quick at a high level like what do all these things look like and and when should you use them?
### 投机解码机制详解
**Speaker A**：先从收益最大的投机解码说起。投机解码针对的是我们前面提到的 Decode（解码）阶段——在正常情况下，大模型生成一个句子，比如从“Modal 是一个……”逐词生成到“Modal 是一个无服务器计算平台（Modal is a serverless computing platform）”。你不再使用大模型一步一个 token 地逐步串行推演，而是引入一个运行成本极其低廉的轻量草稿机制来快速先行生成一系列候选输出 token，对后续可能出现的 token 做出前瞻性猜测，也就是“投机（speculate）”。其核心理念与现代 CPU 内部的分支投机执行（speculative execution）如出一辙：当系统中存在空闲计算余量（slack）时——这正是大模型单步逐个 token 解码时的典型状态——你就可以利用这些松弛余量去提前执行那些最终未必会被采纳的工作。类似于投机执行中执行可能不在实际分支路径上的指令，在投机解码中哪怕处理了最终未出现在输出序列中的预测 token（比如示例中猜错的“company”这个词），但能够实现高度并行的验证和多 token 同时提交，尤其是在以高交互性为目标时，这种机制带来的性能飞跃是极其惊人的。这也是为什么它成为了……
Original English
**Speaker A**: So big wins first. Speculative decoding takes that decode process that we talked about where you generate um uh an output sentence like the LM is going to generate from modal is a to modal is a serverless computing platform. The thing you do instead of running this guy one step at a time is you take something much cheaper to run and you run it um and generate the output tokens and you take a guess at what those output tokens might be. you speculate and the idea here is the same as speculative execution inside processors. If you have some like slack in a system um which is what happens when you run looms one token at a time uh then you can uh sort of use some of that slack to do work that might not be accepted. So run instructions that might not actually be on the code path in speculative execution or process tokens that might not show up in the output like the word company here. Um, but you like it's such a big win to be able to run these things in parallel, especially when you're targeting high interactivity that this um is
### 投机解码与工作负载定制化加速
**演讲者**：这在绝大多数情况下都是一个极其巨大的胜利。而且这种加速效果基本上与你能让投机模型（speculator）猜出多少个 token 呈线性相关。这出自我们关于投机模型撰写的一篇博客文章。基本上，随着平均被接受的 token 数量增加，系统的加速比也会随之提升。而且这些加速幅度都是实打实的整数倍数字，比如 4 倍加速、8 倍加速。如果你平时从事性能工程相关的工作，你可能早就习惯了为了 5% 或 10% 的性能提升而费尽心思、大汗淋漓；但在这里，你所看到的却是 500% 这样的巨大飞跃。
Original English
**演讲者**: like very frequently a humongous win. Um, and it's sort of linear in how many tokens you can get the speculator to guess. This is from one of our blog posts about speculators. Um, basically as you increase the number of tokens that are accepted on average, your speed up goes up as well. And these are these are integral numbers like 4x faster, 8x faster. If you're you know do performance work, you're used to like you know sweating over 5% or 10% but these are like 500%.
**演讲者**：作为从一开始就从事机器学习训练工作的人，在这个技术中我最喜欢的一点是：像这样的性能点（即某个投机模型与目标模型对）与另一个点之间的本质差距，在于你可以针对希望投机模型或目标模型处理的具体工作负载进行深度定制。你训练投机模型，让它精准掌握目标模型在特定数据分布上的行为模式。转瞬之间，原本只有 3 倍加速的效果就能一举跃升到 5 倍乃至 6 倍加速。这正是机器学习领域常说的“苦涩的教训”（The Bitter Lesson），但被富有成效地应用到了性能工程之中，这让人感到非常振奋。
Original English
**演讲者**: Um, and then one of my favorite things about this as somebody who worked on machine learning training from the beginning is that the difference between a point like this and as a speculator and model pair like this and one like this is that you um you customize to the specific workload that you want the the the speculator uh or the target model to work on. you train the speculator to know how that model behaves on that data and all of a sudden you go from six times faster with it three times faster to five times faster. Um and this is you the bitter lesson of machine learning but apply to performance engineering which is exciting.
### 投机解码的机理：输出结构与内部表征复用
**演讲者**：人们经常会好奇地问：等等，一个相对较小、较弱的模型，到底怎么可能提前猜出更聪明的大模型接下来要说什么？这套机制之所以行之有效，背后有几个核心原因。其中最重要的一点在于，模型的输出本身存在大量的结构化模式（structure）。举例来说，如果你观察编码智能体（coding agent）会话中的内容（比如上面展示的实际交互），再将其转换为模型底层实际看到的格式，这种规律性就会变得更加明显。你拥有包含大量特殊标记的对话模板（chat templating），而这些特殊标记通常是高度可预测的。再比如这里直接引用的用户原话，它很可能在之前的上下文记录中就已经完整出现过了，因此非常容易猜中。此外，可调用的工具名称数量是极其有限的；当你刚明确表示要读取某个文件之后，接下来可能采取的操作选项就只有寥寥几种。所以这些内容都非常容易准确预测。
Original English
**演讲者**: Um people often ask like wait how can some dumber model guess what the smarter model is going to say? There's a couple reasons why this works. Um the biggest one is that there's a lot of structure in outputs. Um, and so if you take something from your coding agent sessions like the uh the thing above and then you translate it into what the model sees, this becomes even more obvious. You have chat templating that has all these special tokens which are usually very predictable. And then this right here is a quote from the user which probably appears earlier in the transcript. So that's easy to guess. There are only so many tool calls. um there's only a few of them that are something you would say you would do after saying you were going to read that file. And so this is like pretty easy to predict.
**演讲者**：但目标模型真正在进行的计算，是在处理这些 token 的同时，逐步构建出极其丰富的内部表征（rich internal representation），这些表征在后续生成大量 token 时都会被反复依赖。因此，通过能够以较高命中率准确猜测出那些具体的 token 值，你实际上就能够将目标模型构建内部表征的过程进行并行化加速，这非常精妙。不仅如此，这些深层表征还会直接在当前先进的投机模型内部得到复用。如今的投机模型不再单纯依赖外置逻辑，它们既不是马尔可夫模型，也不是 n-gram 猜测器，更不是从零开始运行的独立外置语言模型；它们会直接提取并复用目标模型计算出的丰富内部隐藏表征，充分借助目标模型内部正在发生的激活状态。这样一来，投机模型就能够以此为基础进行自举（bootstrap）。我们具体采用的核心架构是 Dlash，这是我们在实践探索中发现效果最好、投入研发精力最多的一种投机架构，无论面对特定模型自带的原生投机解码实现，还是对比基线模型，它都能带来巨大的推理速度飞跃。
Original English
**演讲者**: Um but what the model the target model is doing is actually like processing these tokens and creating this rich internal representation that is going to be used for many tokens in the future. Um so you by being able to like guess what those specific values are with a good rate, you're able to parallelize that um construction of their representation. Um which is pretty cool. Um, and that representation actually gets reused inside of current speculator models. They don't just have to they're not like a marov model, an engram speculator or another language model from scratch. They take those rich internal representations from the target model and they uh they use that they use what's what's going on inside the model. So they're able to kind of like uh bootstrap from that. So the specific architecture there was Dlash. um that's the the sort of speculator architecture that we've found uh works best um and the one that we've invested the most in and gives us like large speed ups over both built-in speculative decoding in certain models and of course over the baseline.
### 模型量化与全栈软硬件协同
**演讲者**：在搞定投机解码并实现深度优化之后，下一项能够带来巨额性能收益的技术就是量化（quantization）。所谓量化，就是将模型权重或激活值的浮点数表示位宽降低。比如从 16 位浮点数压缩到 8 位，甚至进一步降至 4 位。这意味着内存带宽需求得到了线性的削减，同时因为数字位数更小、数据更加紧凑，硬件每秒所能执行的浮点运算次数（FLOPS）也会迎来线性的增长。
Original English
**演讲者**: Um after uh speculative decoding and getting that right, the next big win is quantization. So with quantization, you take the floatingoint representation of the weights or activations of the model and you decrease the bit width. So it goes from 16 bits to 8 bits to maybe even four bits. Um so this is a linear reduction in memory bandwidth demand and a linear increase in floatingoint operations per second because the the the bits are smaller or the the the numbers are smaller.
**演讲者**：之所以强调这需要全栈协同，是因为你必须深入底层硬件去核验：我当前运行的 GPU 是否具备真正利用这些低精度格式的硬件加速能力？针对 FP8，我手头是否有 H100 或更新一代的 GPU？针对 FP4，我是否有 B200 或更新的芯片？但与底层硬件支持并存的另一个关键是，量化会实质性地改变模型的输出行为。投机解码的神奇之处在于它丝毫不改变模型原始输出的概率分布，但量化则必然会带来分布偏移。因此你必须一路向上追溯到应用层，严谨评估并确认：经过量化的模型是否依然保持了足够的智能水平？各项核心能力是否依然能满足业务场景的实际需求？
Original English
**演讲者**: Um so this I say this requires full stack coordination because you have to go down to the hardware layer and be like okay am I running on a GPU that can actually make use of these things? Do I have a H100 or later for FP8? do I have a B200 or later for uh FP4? Um but it also changes model behavior. The magic of spec of decoding is that it changes nothing about the probability distribution of the model's outputs. Um but quantization does and so you kind of have to go all the way back up to the application layer and ask is this still sufficient intelligence? Is this are the capabilities still what I need if I quantize the model?
**演讲者**：在量化评估方面，我们的《LLM 工程师指南》（LLM Engineers Almanac）提供了一个非常直观的小型可视化工具，用于展示量化所带来的直观影响。在这个演示中，它对比了一张鹦鹉图像在进行 4 位量化前后的视觉差异。我觉得这很好地展示了量化在何时能够真正发挥作用：坐在会议室前排的人可能能够分辨出量化后的图像稍显粗糙、有些微瑕疵；但对于坐在后排的观众，或者那些眯着眼睛用手机看图的人来说，他们大概完全看不出图像已经被进行了量化压缩。这就是在工程落地中你可以合理依托的感知特性，当然它始终取决于你具体的业务场景与精度宽容度。
Original English
**演讲者**: Um, so with quantization, there's this nice little visualizer on our LM engineers almanac that shows you like the effect of quantization. In this case, 4bit quantization on this uh image of a parrot here versus the original. I think maybe as a demo of when quantization does work. I think the people in the front row can probably see that this is a little bit um a little jank, but the people in the back row and maybe um you know people squinting at their phones like maybe can't see that that's um that that's been quantized. And so that's the effect that you can rely on. Um sort of depends on what you're doing.
### 主机端性能调优与 CUDA Graph 捕获
**演讲者**：上述这些都是能带来巨幅加速的重大技术手段。接下来，让我们深入探讨主机端（Host/CPU）性能调优这一艰苦的磨砺过程。这里的关键症结在于：所有的重度计算任务实际上全是由 GPU 承担的，但 CPU 却常常在半路形成严重阻碍。CPU 负责协调与调度 GPU 上进行的所有计算流程，因此你必须确保 CPU 侧的调度逻辑绝不会阻塞 GPU 的正常运转。如果出现这样的情况：GPU 刚完成了一段繁重的计算，此时 CPU 却必须停下来花费时间去计算和决定下一步该干什么，等决策完毕后才慢吞吞地重新向 GPU 下发新任务——这在系统设计中被称为“气泡”（bubble）或计算停顿（stall），其负面影响极其恶劣。如果你曾经编写或调优过事件循环（event loop）系统，你肯定非常熟悉这个基本常识：永远不要让慢速逻辑去阻塞那个跑得最快的核心组件。
Original English
**演讲者**: So those are big wins. Let's talk about the host performance grind. So the key thing here is that the GPU does all the work, but the CPU can get in the way. the CPU is coordinating all the work that's going on in the GPU. And you want to make sure that the CPU work that is going on does not block the GPU work. Like this is finished. Oh, now I need to do some work to decide what to do next. And then oh, I'm going to start running GPU stuff again. That's really bad. It's a bubble or a stall. Um if you've worked with an event loop, then you're familiar with this idea like don't block the fast thing.
**演讲者**：针对主机端调度的这一瓶颈，有一项极其关键的核心技术叫做 CUDA 图捕获（CUDA Graph Capture）。它的核心原理是改变过去逐个向 GPU 串行发射算子核函数（kernel）的做法，转而让你在第一次执行时完整录制下整套需要运行的核函数调用关系图，之后在实际调度时便能将整张图一举打包下发并发执行。这是一种能够彻底消除大量主机端调度与发射开销的极其有效的技术手段。
Original English
**演讲者**: Um there's a key technique for this called CUDA graph capture that allows you to take like instead of launching like one GPU kernel at a time, you can just record all the kernels that you ran one time and then just launch them all at once. Um which is a pretty effective way to like cut a lot of this host work out.
**演讲者**：在监控和追踪这种流水线停顿方面，我个人最喜欢的一种工具或手段其实非常直观——那就是直接观测 GPU 的实时功耗（power usage）和核心温度。如果你的 GPU 功耗没有紧贴着 100% 的满载功率运转，那么极其大概率说明你并没有完全榨干这块 GPU 的计算潜能。现代 GPU 在达到巅峰峰值算力时，无一例外都是处于严重的功耗受限（power-limited）状态的。在我们的具体优化实践中，仅仅通过肉眼观察监控图表就迅速发现，某一特定设备的总功耗远远低于正常水平；在正常的对比设备功耗早已大幅突破 2000 瓦的同时，出现瓶颈的这一块却显著低于它，或者说卡死在了某个异常的低阈值之下。
Original English
**演讲者**: Um one tool for keeping track of this is actually my my favorite is to just look at the power usage and the temperature of the GPUs. if you aren't close to 100% power, you probably aren't using the whole GPU. They're very much power limited uh when they're achieving peak performance. Uh and so in this case, we were able like just by eye noticing that this thing was way below its total GPU power usage even at this uh uh between this one which is at well over 2,000 watts and this one which is uh below it or kind of capped at it.
### Profiler 追踪排查与主机端开销优化
**演讲者**：仅凭功耗异常本身并不能直接揭示根本症结所在。你必须借助专门的性能分析工具（profiler）深入底层，抓取详尽的系统执行轨迹（traces）。性能工程本质上是一门高度依赖经验与实测数据的学科。在这方面，NVIDIA 官方出品的 Nsight Systems 以及 PyTorch 体系内的 torch profiler，就是你最必不可少的核心排查利器。在针对上述那个案例的深入分析中，我们通过分析追踪轨迹最终锁定，确实有一块 GPU 严重拖慢了整体进度。界面中所有高亮显示的算子块都在并行执行 Deep GEMM 矩阵乘法，而图中的另外三块 GPU 却全部在原地空转，苦苦等待那块运行异常缓慢的 GPU 完成任务。经过深入底层排查，发现这竟然是一个 NUMA 节点感知（NUMA region awareness）层面的配置失误——由于那块 GPU 被错误地绑定到了不匹配的 CPU 插槽上，产生了跨 NUMA 访问惩罚，导致它每次执行都不可避免地比其他卡慢上一截。
Original English
**演讲者**: Um that doesn't tell you what the problem is. You want to use something like a profiler to get in there. You want these traces. It's a very empirical discipline. Um, and so the endsite systems or torch profiler, the tools that come from Nvidia for this are kind of your uh your go-to. And if you taking a look at that specific case, we're able to determine that one of these GPUs is actually falling behind. All these guys here are all doing the same thing. The the highlighted blocks, they're all running this deep gem. And these three GPUs are all waiting on this one that's slow. turned out to be like a NUMO region awareness thing. Um that was causing one of them to always be a little slower because it was on the wrong socket.
**演讲者**：在主机端层面，还有许多 Python 运行时的开销值得深挖，你可以利用 Py-Spy 等性能分析工具进行检测。鉴于底层调用链是运行在 Python 环境中的，借助这些工具能够透彻理清 Python 解释器内部的动态。我们曾写过一篇博客详细记录过一个典型案例：系统中此前存在某种重复创建和分配资源的低效操作，我们直接将其重构为了一个轻量级的缓存层——实质上就是一个 Python 字典。虽然在 AI GPU 基础设施的语境下，你完全可以高大上地称其为“CUDA IPC 进程间通信句柄池缓存”（CUDA IPC pool handle cache），但它的核心实现就是一个常驻内存保存这些句柄对象的 Python 字典。仅仅引入了这样一个简单的缓存结构，整个系统性能便不可思议地当场提升了 10%。
Original English
**演讲者**: Um there's a lot of host level things that you can use PI Spy like the things are running in Python. So you could look at what's going on in Python and figure things out pretty well. Um we have a blog post that talks about one case in particular where we took this like repeated sort of allocation and turned it into just a cache, a Python dictionary. Um, but I guess you get to call it a CUDA IPC pool handle cache if it's uh AIG GPU stuff. Um, but a Python dictionary that just held on to handles for these and suddenly that made the thing run 10% faster.
### 集群横向扩展与 KV Cache 感知路由挑战
**演讲者**：上述讨论大多局限在单个副本（单实例，通常覆盖几块 GPU）的微观调优层面。而当你开始将系统横向扩容（scale up）到大规模分布式集群时，你就必然会迎面撞上请求路由（routing）所带来的全新棘手难题。你很可能会发现，在扩容之前预先测算的优异性能表现，在真正扩容到大规模集群后却根本无法顺利兑现。其中一个尤为典型的故障表现就是：系统会出现远超预期的长尾延迟（tail latency），而且这些突发延迟仿佛完全是随机出现、毫无规律的。然而事实的真相是，它们并非纯粹随机，其根源恰恰在于你的底层请求路由策略究竟是如何制定的。对于智能体（agent）长上下文这类工作负载而言，这一点因为 KV 缓存（KV cache）的存在而显得尤为致命。如果你的调度路由没有针对 KV 缓存的命中状态进行极其精细的感知与对齐，那么缓存命中与缓存未命中之间的巨大性能鸿沟就会……
Original English
**演讲者**: Um, so this was mostly focused on like a one single replica like a couple of GPUs at a time. When you go to scale up, um, you're going to run into routing problems. So you might predict performance before scaling that that just doesn't uh happen once you scale up. Um, so one particular thing that might happen is you might uh end up with like tail latencies that are much higher than you expect and that kind of show up seemingly at random. Um, and the answer is that they are kind of at random. They come from the way that you do your routing. Um, and this is especially important for agent workloads because of the KV cache. If you're not routing carefully like with respect to the KV cache, then the difference between like a cache hit and a
### 路由与系统层优化：解决请求过散与 KV Cache 状态感知
**Charles**：当缓存未命中时，延迟可能高达 30 秒，而缓存命中时只有 500 毫秒。我们在实际生产部署中对其进行了深入测量，并排查出多种导致请求偏离预期的诱因。我们原本期望维持大约四到五个并发请求，但在监控中却观察到了严重的请求过度分散（over-dispersion）。最终的解决方案并非盲目堆砌硬件，而是让路由层变得更加智能。路由器必须演进为更具状态感知（stateful）能力的系统，深度感知应用层的运行状态，实时跟踪系统负载以及各节点的 KV Cache 驻留状态。正是通过这种智能调度，我们成功收敛了请求的异常发散，彻底消除了延迟突刺，从而大幅提升了整体吞吐与响应表现。
Original English
**Charles**: Cache missed is maybe 30 seconds versus 500 milliseconds. Um, and so uh measured this in one of our deployments and determined that there was variety of reasons causing instead of the you know four or five uh concurrent requests we were trying to aim for, we we saw this major over dispersion of requests and the solution was just to make the router smarter. had to become much more stateful um and aware application aware of things like the load and the KV cache state but that was able to sort of pull in this uh over dispersion and uh massively you know get rid of these uh spikes and massively improve performance.
### GPU 算子性能与白板第一性原理分析
**Charles**：只有在解决了上述系统架构与路由调度层面的问题之后，你才真正有必要去考虑 GPU 内部的计算性能以及针对 GPU 硬件本身的极致调优。从总体来看，目前的 GPU 算子（Kernel）已经相当标准化了，它们的执行效率已经无限接近硬件的物理极限（speed of light），而且底层的模型架构近期也没有发生颠覆性的剧烈改变。业界已经拥有非常出色的开源算子实现，比如来自 Nvidia 官方团队、Treehouse（Tri Dao 团队）以及其他顶尖研究群体的成果。因此，继续在算子层面死磕，通常往往只能挖掘出几个百分点的边际性能提升。这种微幅优化在超大规模集群中确实至关重要，但如果你刚刚起步搭建属于自己的自建推理服务，这绝不是你最先需要焦虑的核心瓶颈。
Original English
**Charles**: Um so then and only then are you allowed to think about you know GPU performance and optimizing things on GPUs. In general kernels are relatively standard. They're close to the speed of light. architectures are not changing that much. Um, and so there are really good kernels out there from Nvidia and from like treehouse group and others. So there's only a few percentage points to gain there. Um, so that matters a lot at scale, but it's not something to worry about at the like, you know, at the beginning of serving your own inference.
**Charles**：如果确实需要深入分析 GPU 性能瓶颈，英伟达官方提供的专用工具 Nsight Compute 是极具杀伤力的利器，它允许你一路向下深潜至汇编指令级别，精准定位性能劣化的根本原因。然而，在你真正打开并运行 Nsight Compute 之前，我强烈建议你先拿出一支白板笔和一块白板。如果你仔细研读过 Tri Dao 与 Together AI 团队关于 FlashAttention 算子优化工作的技术博客，就会发现他们衡量研发方向与识别核心瓶颈的方法极其质朴——就是在白板上清晰推演数据吞吐率与吞吐指标（feeds and speeds）：当前输入了多少数据？数据在流经管线时需要经历多少次算术运算？我们以多快的吞吐速率将计算结果写回显存？这种基于第一性原理的系统推演，是你可以在写下任何复杂优化代码前就完成的。这种分析框架不仅适用于底层算子，更贯穿了整个系统栈。一旦你对底层硬件的物理限制与吞吐特性了然于胸，就能在工程实现中少走很多弯路，既不会把自己绕进死胡同，更不会平白无故烧掉巨额的算力账单。
Original English
**Charles**: Um, the specific tool for this ensite compute allows you to like go all the way down to the assembler level and figure out where performance problems are coming from. Um, but before you pull out insight compute, I'd suggest you pull out um a marker and a whiteboard. Um, if you look at uh Tree Dow's uh and together's blog about their flash attention for kernel work. Um, the sort of the way they decided what to work on, the way they decided what problems mattered was just to write down the feeds and speeds like how much data is coming in, how many operations need to come uh happen on it and how quickly can we move it back out. And uh that is something that you can do you know before um you know from first principles and this sort of applies across the stack. If you understand these things about the like hardware that you're operating with then uh you can really um you can get a lot further um without confusing yourself and wasting money.
### 高吞吐量推理新范式：AI SQL 与关系型 Join 提示词
**Charles**：好的，我之前曾说过今天不会展开讨论高吞吐量推理的场景，但实际上我撒谎了。今天恰好有一项最新的重磅工作正式发布，我非常渴望借着这次在巴黎 AI Engineer 大会的机会，第一时间与现场的各位同行交流。在 Modal 内部，我们此前投入了海量精力来攻坚极低延迟推理，因为大家都对智能编程助手（Coding Agents）以及类似 Jev 这样的实时交互工具展现出了极致的热情。然而，在工业界现实场景中，还有另一批形态完全不同、对批处理高吞吐量推理存在着巨大刚性需求的业务模式。
Original English
**Charles**: Um great. So I said that we weren't going to talk about high about high throughput but I actually lied. Um there's a little piece of work that actually is just coming out today and that I wanted to share with folks here at AI Engineer Paris. Um we've been thinking a lot about low latency inference at Modal because of how much people, you know, love coding agents, how much they love things like Jev. Um but there's actually quite a bit of demand for high latency inference for things that look very different.
**Charles**：这类工作最典型的代表，也是我们今天要重点介绍的技术，就是“AI SQL”。请注意，这绝对不是指“让 AI 帮人类写 SQL”。据我所知，现在全世界大部分业务 SQL 代码基本都被 AI 包揽了；我本人写这个演示时也是让 AI 生成的，但我绝不会把那类原始脚本直接扔到生产环境去裸跑。我们这里谈论的恰恰是完全相反的维度：它是利用 SQL 的关系代数能力来为 AI 构建提示词并编排计算流。其背后的核心诉求非常直观：你的数据库中沉淀了极其庞大的客户数据，同时也记录了巨量的产品维度信息。你是否曾想过直接让大语言模型来做复杂的商业意图匹配，比如“分析究竟哪类客户群体最有可能购买我们的哪一款新产品”？
Original English
**Charles**: Um, so the sort of prototypical example of this and that we're going to talk about is AI SQL. So this is not the same as AI writing SQL, which my understanding is AI now writes all of the SQL in the world. Um, certainly I wrote this um, but you know, this is this is not something I run in production. Um, but yeah, so this is not AI writing SQL. It's actual it's actually the other like direction, the opposite. It's using SQL to write prompts for AI. Um, so the idea is, you know, you have all this information about your customers and about your products. Have you ever wanted to just ask Chad GBT like which of my customers might buy which of my products?
**Charles**：在传统的数据库体系中，这种跨维度的业务关联本质上就是一个标准的关系型联结（Relational Join）操作，是你在传统的事务型数据库（OLTP）或分析型数据仓库（OLAP）中每天都会执行的基础查询。然而，一旦把大语言模型介入其中，灾难就可能降临：如果你直接调用第三方商业闭源 API 来执行这种基于 Join 展开生成的提示词，它会在瞬间让你彻底破产。试想一下，当你在包含大量记录的数据库表上直接触发这种跨表关联计算时，可能一次性就会向外迸发出多达 800 万个独立的推理请求。紧接着，这 800 万个请求被狂暴地并发投递到 API 端点，你的 Token 预算额度瞬间就会被彻底打穿。虽然这也是达成所谓“Token 消耗最大化（token maxing）”的一种极端途径，但我相信在座的各位都已经深刻意识到，这种无节制的消耗完全是一场工程灾难。
Original English
**Charles**: Um, and that can be expressed as like a relational join, the classic kind of thing that you would normally do in your database or your and in your transactional or analytic database. Um, but the problem here is that this guy here, this prompt is going to like bankrupt you if you're running against like um a proprietary API. So like running this on your database might produce like 8 million queries all at once. Um, and then you know fire them all off and there goes your token budget. Um, yeah, that's one way to achieve token maxing, but I think we now all agree that that's a bad idea.
### 查询计划与推理引擎融合：14倍加速与每分钟十亿 Token
**Charles**：之所以说高吞吐量推理和低延迟交互式推理遵循着截然不同的系统优化法则，最本质的区别就在于：在这类离线批处理查询中，你天然拥有极其宏大且确定的计算结构信息。屏幕上展示的便是一个相对复杂的查询执行计划（Query Plan），其中包含了多层关系型 Join 操作。这个查询计划清晰地定义了一切：为了拼装提示词，底层需要抽调哪些文档与实体数据；各个实体上下文之间应该如何排列组合并相互交互；以及系统后续即将生成并喂给大模型推理引擎的总计 800 万到 1000 万个确定性 Prompt 全集。然而，当今市面上主流的大模型推理引擎对于 SQL、关系代数或者查询执行计划完全一无所知。现有的系统仅仅理解简单的请求-响应（Request-Response）模式，或者最多支持朴素的批量请求语义（Batch Request）。那么，如果我们借鉴经典数据库系统设计的精髓，将数据库成熟的查询优化与执行计划编译技术，直接与大模型推理引擎进行深度纵向协同，会产生怎样的化学反应？
Original English
**Charles**: Um so uh the key thing that makes this a totally different optimization problem from the low latency inference is that you have this tremendous amount of structure. So this is a query plan for a little bit more complicated of a query that has a couple of joins in it. Um and this query plan says these are all the sort of documents I'm going to use to construct prompts. This is how I'm going to have them interact with each other. Um, and this is all the 8 million or 10 million or whatever prompts I'm gonna I'm gonna generate and send to the inference engine. And the existing inference engines have no idea about SQL. Um, they like they only know sort of like request response maybe batch request kind of semantics. Um, so what if you took the sort of uh page out of the database book and you did like query planning and combine that with the inference engine.
**Charles**：我们正是顺着这个思路付诸实践的。我们与卡内基梅隆大学（CMU）的 Shreya Shankar 团队紧密合作，共同攻关完成了这项工程实现。在刚才展示的那个典型复杂查询用例中，相比于业内通用的基线多模态大模型推理栈（baseline vLLM），我们取得了惊人的 14 倍端到端加速。在单张英伟达 H100 GPU 上，系统的吞吐速率突破了每分钟十亿级别（over a billion tokens per minute）的 Token 生成与处理能力。更为震撼的是整个查询任务的算力成本：借助我们的优化架构，跑完这整个超大规模查询仅仅花费了约 2 美元；而如果采用原始的基线 vLLM，则需要花费近 30 美元；若是换成调用外部商业专有模型，哪怕是选用 OpenAI 旗下最轻量、最便宜的小模型，所需成本也会呈数量级倍增。
Original English
**Charles**: So we did that with um partnered with Treya Shankar at CMU uh to get about a 14 times speed up over the like baseline VLM on that specific query that I just showed you and achieve like over a billion tokens per minute on a single H100 and finish this query at just like $2 to execute instead of like uh almost $30 with VLM and like uh quite a bit more with um something like uh even the smallest OpenAI models.
### Modal Serverless 平台与自动化端点
**Charles**：令人振奋的是，这项成果完全是开源的，我们就在今天刚刚将其正式公开，现场的分享正是该项目的首次全球发布。大家可以在 Full Stack Data Labs 的官方主页以及他们的 GitHub 仓库中查阅完整的项目源码。同时，我们也在 Modal 官方技术博客上发布了详尽的深度解析，欢迎大家扫码或访问链接查看。今天我在这里所分享的各项理论依据、系统架构细节与测试基准，在博客中都有更为系统而详实的展开阐述。此外，我们在 modal.com 上也沉淀并开源了大量关于如何在生产环境中自建并运行大模型推理集群的技术资源。
Original English
**Charles**: Um, so this is actually open source. We just released it today. Um, this is the first announcement of it. Um, you can find it at the full stack data labs website and their GitHub. Um, and you can read about it on our blog on uh, yeah, check it out here. Uh, all the information that I shared today is in greater detail on our blog and a bunch of resources about running your own inference on modal.com.
**Charles**：我看到工作人员已经在示意我大概还有 30 秒就要走下讲台了，所以最后我想总结一下我今天向大家全盘托出这些思考的初衷。这一方面是为了鼓励更多的开发者摆脱对黑盒闭源 API 的盲目依赖，积极探索并在生产中掌握自建大模型推理的核心控制权；另一方面，也是因为我坚信，Modal 团队历经多年打磨出的 Serverless 容器基础设施平台，正是运行这些高并发、高弹性异构推理负载的最理想底座。当你真正迈出在生产环境中自建推理服务这一步时，你一定会真切体会到 Modal 的独特优势。无论你是倾向于完全自主掌控代码与底层容器执行，还是希望直接复用 Modal 提供的自动化弹性端点（Modal Auto Endpoints）这一专门为终端用户打造的定制化推理部署系统，它都能为你带来无与伦比的工程效率。非常感谢大家的倾听！
Original English
**Charles**: Um, I'll I'm gonna get pulled off the stage in about 30 seconds. So, I'll just say the reason why I'm telling you all about this is both because I want people to run their own inference and because I think the contain the uh um serverless infrastructure platform that we've built at Bodal is the best place to run this stuff. So, when you go to try and run your own inference, I think you'll realize that modal is the right place to do it. um whether that's uh by running code yourself or by taking advantage of modal auto endpoints um our automated system for producing custom inference deployments for uh for users. All right, thank you very much.
### 主持人串场：从数据中心高吞吐转向端侧计算
**主持人**：非常感谢 Charles 精彩绝伦的分享！好的，让我们再次用热烈的掌声送给 Charles！太精彩了。听完刚才的案例，我举双手赞成——我可绝对不想盲目跑一个巨型查询然后把自己的银行账户干到破产，这绝对是所有工程师的共识。紧接着我们要请出的下一位重量级演讲嘉宾，所要分享的方向则恰好与此截然相反。他将带领我们深入探讨模型如何下沉并在极其微小的端侧边缘设备上稳定高效运行，也就是如何让轻量化边缘 AI 落地生根。我们的下一位演讲者是来自 ARM 公司的资深总监 Dominic Pajak。现在，请大家随我一起，用最热烈的掌声欢迎 ARM 资深总监 Dominic Pajak 登台！好的，有请！
Original English
**Host**: Thank you so much, Charles. All right, let's give it up for Charles once more. All right. Yeah, I totally didn't don't want to run that query and bankrupt myself. That's for sure. Um yeah, our next speaker is actually going to talk to us about the exact opposite. Uh he's going to talk to us about how to run models in uh and in uh in smaller devices. And so he's going to talk to us about edi. Our next speaker is Dominic Payak uh and uh he's a senior director at ARM. So please join me in welcoming to the stage senior director at ARM, Dominic Payak. All right.
### 端侧 AI 的三大契机与 ARM 的全栈算力布局
**Dominic Pajak**：好的，谢谢大家，各位朋友大家好！正如主持人介绍的那样，我是来自 ARM 公司的 Dominic Pajak。今天我们将把目光聚焦在边缘端，共同探讨当我们将现代 AI 推理完整搬到终端边缘设备上运行时，究竟会碰撞出怎样的技术火花。纵观当下的产业技术格局，我认为正有三股相互交织的技术浪潮在同时爆发，它们将合力开启前所未有的巨大创新空间。第一股浪潮，是体积小巧、完全开源开放权重（open-weight）且具备强大认知推理能力的小模型，正以令人窒息的速度日新月异地涌现。
Original English
**Dominic Pajak**: All right. Hi everyone. So yeah, my name is Dominic Pike. time at ARM and today we're going to talk about what happens when you run inference at the edge. So there's there's three things going on right now which I think are going to unlock some incredible opportunities. One is just the speed that small openweight capable models are becoming available.
**Dominic Pajak**：第二股浪潮在于，如果你将这些层出不穷的高效小模型，与物理世界中无处不在、极易获取的普适边缘硬件算力相结合，就会发现我们拥有极其充沛的部署载体。由此便催生了第三个层面的巨大机遇：整个行业迎来了前所未有的可能性，我们能够构建出更具情境智能、能够深度理解用户真实意图的全新硬件终端形态，从而在五花八门、异构多样的物理现实场景中，交付出以往人类从未体验过的交互与智能体验。
Original English
**Dominic Pajak**: You combine that with the fact there's tons of accessible hardware you can then put these things on and there's this potential for devices that are a little bit more intelligent, a little bit more um aware of the intention of the user and deliver experiences that have not been seen before in all kinds of different environments.
**Dominic Pajak**：大家知道，我来自 ARM 公司。这里我也想向大家简要勾勒一下 ARM 在计算格局中的全貌。ARM 本质上是驱动我们周围整个物理世界设备运转的核心计算底座：从能效极高的大规模现代云端数据中心基础设施，到涵盖智能驾驶汽车与机器人在内的具身物理 AI（Physical AI）——这些对硬实时（hard real-time）与功能安全有着极其严苛要求的关键任务应用；再一路向下延伸，几乎涵盖了你人生中所拥有过的每一部智能手机、广阔的客户端 PC 个人计算设备、工业物联网（Industrial IoT）节点，以及各种层出不穷的前沿智能硬件新品类。而我和我所带领的技术团队一直以来在 ARM 内部的核心使命，就是打造全栈软硬件赋能方案，推动这些全新维度的智能设备能够真正被工程团队孵化并顺利推向全球市场。
Original English
**Dominic Pajak**: And so yeah, I'm I'm from ARM. And so a little bit about ARM. ARM is the compute platform at the heart of uh devices all around you everywhere from highly efficient data center infrastructure. There's physical AI with automotive and robotics, so hard real time safety critical applications all the way down to pretty much every smartphone you've ever owned, client computing and industrial IoT and emerging devices. And that's the the stuff that me and my team have been working on enabling new classes of devices to be created and come to market.
**Dominic Pajak**：从骨子里来说，我是一个对计算机发展历史极度痴迷的技术极客。顺便提一句，我特别喜欢今天我们开会的这个场地——Station F 创业孵化园恰好坐落于“艾伦·图灵前广场（Parvis Alan Turing）”。我不确定我的法语发音是否足够标准，但伫立在以图灵命名的土地上，回望整个计算机工程漫长而激荡的发展史，并深思当下计算革命究竟在将全人类引向何方，这本身就是一件令人无比神往的事情。
Original English
**Dominic Pajak**: And so I am a computer history geek. I love the fact by the way that this uh station F is on Parve Allen Turing. I don't know if I pronounced PV correctly, but um it's it's fascinating to look at the history of computing and where it's taking us
### 人机交互演进与 AI 带来的范式反转
**演讲者**：回顾过去 80 年人类与机器交互的历史，从最早的时期开始——大约 80 多年前的打孔卡和磁带，随后发展到键盘，再到鼠标、指针设备以及触控屏。每一代技术变革都让计算变得更加易于人们访问和交互。但无论技术如何演变，在过去每一个时期，用户都始终必须将自己的注意力完全集中在设备上，并且承受着一种心智阻力：必须把内心想要完成的事情，自行转化成机器能够理解的步骤，并以符合设备输入规范的形式输入进去。而人工智能时代真正令人惊叹之处，就在于彻底颠覆了这种关系。突然之间，机器开始理解我们了。我并不是说机器具备了意识，而是它们能够理解自然语言，能够将我们的意图转化为具体的操作和工具调用（tool calls），从而帮我们把事情办成。
Original English
**Speaker**: Looking back over the past 80 years of how people have interfaced with machines starting in the very first days—maybe starting 80 or more years ago with punch cards and tape, and then keyboards, and then mice, pointers, and touchscreens. Each generation has made computing easier for people to access and interact with. But irrespective, every time it's still the case that the user is having to give their attention to that device and have the mental friction of converting the thing they want to get done into steps that the machine understands in a form that can be input into that device. And the amazing thing about the AI era is it inverts this relationship. Suddenly the machines understand us. I'm not saying in a conscious way, but they can understand natural language and they can convert our intent into actions and tool calls to get things done.
**演讲者**：从某种角度来看，当我们谈论边缘推理（edge inference）时，这其实算不上全新的事物。多年以来，目标检测（object detection）和卷积神经网络（CNN）一直都在边缘设备上运行。可以设想诸如工厂质检、停车场车位监控等形形色色的成熟应用。然而，最近随着嵌入向量（embeddings）、大语言模型（LLMs）以及随之而来的智能体应用（agentic applications）的发展——所谓智能体，也就是拥有足够强大的推理能力、能够调用外部工具，并且能够在足够长的上下文中进行处理、从而代表用户自主行动的大模型——这些在如今才刚刚成为现实。我认为很有必要退后一步，冷静审视一下这一切发生得究竟有多快。
Original English
**Speaker**: In some ways, when we're talking about edge inference, this isn't new. There's been object detection and CNNs running on edge devices for many years. You can envision applications in factory inspection or parking bay monitoring and all this kind of stuff. More recently, embeddings, LLMs, and then agentic applications—which I guess is LLMs that can reason strongly enough, call tools, and deal with context long enough to act autonomously on behalf of the users—these are just becoming possible today. I think it's good to step back a little bit and just look at how fast this has happened.
### 模型压缩飞跃：从云端庞然大物到端侧推理
**演讲者**：我把这个演进趋势绘制成了一张图表。回顾 GPT-3.5 刚出现的时候，那是我第一次真正亲身体验大语言模型，那种震撼感无与伦比。但如果你观察一下这股能力被压缩进参数量越来越小的模型中的速度，就会发现直到今天，相似的能力大致已经可以塞进仅约 10 亿参数（1B）的模型中，并且能够直接在智能手机或者拿在手中的树莓派（Raspberry Pi）上流畅运行。这极其惊人。顺带提一句，我知道各项基准测试（benchmarks）并非完美无缺，像 MMLU 这类基准可能更是如此，但这清晰地揭示了一个持续向前的不可逆趋势。如果我明年再站在这里，这一参数门槛肯定还会进一步降低。而且这不仅体现在通用知识问答上，在复杂的逻辑推理能力上同样如此。
Original English
**Speaker**: I graphed this out. Looking at GPT-3.5, that was the first time I really experienced what an LLM was, and it was a mind-blowing experience. But if you look at how quickly this capability is then being compressed into models with fewer and fewer parameters up until the present day, this can now fit in roundabout a billion parameters and run on a smartphone or on a Raspberry Pi in your hand. This is phenomenal. By the way, I know that benchmarks are not perfect, and MMLU probably less so, but it's indicative of a trend which is going to continue. If I'm here next year, this is going to have gone down even further. And it's not just general knowledge question answering; it's reasoning too.
**演讲者**：在这里我们看到的是当前的前沿探索：利用 40 亿活跃参数（4B active parameters）究竟能做到什么？这就是如今边缘设备上具备的计算能力量级。特别是在混合专家架构（Mixture of Experts, MoE）引入之后，现在的端侧模型在诸如 GPQA Diamond 这类前沿科学基准测试的得分上，甚至已经超越了大约一年半以前 Claude 3.5 Sonnet 首次发布时的水准。这项技术的发展迭代速度简直如同疾风骤雨。
Original English
**Speaker**: Here we're looking at a frontier of what you can do with four billion active parameters—the kind of compute on today's edge devices. Again, especially with Mixture of Experts coming into the picture, this has gone from something modest to now exceeding what Claude 3.5 Sonnet was doing when it was first released with its frontier GPQA Diamond score. I think this was like a year and a half ago. This stuff is just moving phenomenally fast.
### 交互延迟与端侧语音体验：人类对话轮换的人类学视角
**演讲者**：当然，衡量边缘体验的关键不仅仅在于模型能力，当我们谈论人类用户的实时交互时，延迟（latency）更是核心所在。现在已经有许多能够在设备本地极其顺畅运行的模型，它们执行语音转文字（Speech-to-Text）的延迟极低，足以用来构建响应敏捷的语音用户界面（Voice UI），而根本不需要把音频推理请求发送到云端。去思考究竟是什么驱动了这些延迟指标要求，是一件非常耐人寻味的事。归根结底，我们设计这些系统时应该遵循的准则，正是人类原本的心理预期与舒适感习惯。
Original English
**Speaker**: Of course, it's not just about capability; it's about latency when we're talking about human user interaction. There are also models now which can run comfortably on-device which do speech-to-text with latencies that are useful for creating voice UI without any need to send this inference to the cloud. I think it's very interesting to consider what drives these latency requirements. Really, what people expect and what they're comfortable with is the way we should be designing these things.
**演讲者**：我发现有一张图表格外引人入胜：这是一项关于人类日常对话轮换（turn-taking）以及对话者之间应答延迟的人类学研究。非常有趣的是，这种延迟因文化和语言而异。例如，日语在回应时极其迅速高效；而丹麦语的对话节奏则非常悠闲、非常缓慢——顺便说一句，我不知道这是否是因为丹麦语中动词经常放在句末的缘故；英语的应答速度则大致居于两者之间。这项研究为我们提供了一个极佳的参照基准，明确了我们的端侧语音交互延迟究竟需要收敛到什么样的区间之内。
Original English
**Speaker**: I find this graph really fascinating. This is an anthropological study of human conversation turn-taking and the latency between speakers. It's interesting that it's culturally dependent and language dependent. Japanese is very quick and efficient in responding. Danish is very laid-back and very slow—I don't know if that's because the verb is at the end of the sentence, by the way. English is kind of in the middle. That gives you a gauge of where these things need to land.
### 为什么选择边缘端推理？效率、定制与数据隐私
**演讲者**：所以，我希望大家能和我一样，为所有这些推理都能在边缘设备上实现而感到兴奋。不过你可能会问：“既然我们在数据中心里早已能够实现这一切，为什么还要大费周章地把它放到边缘端呢？”这是一个非常好的问题。我认为将推理推向边缘有着几条极其充分的理由。第一是效率：很多人已经开始审视自己的 token 账单与开销，并思考其中有多少工作量其实可以卸载（offload）到本地设备或本地私有部署环境（on-prem）中完成。第二是可以进行深度定制：你可能希望在模型中使用个人隐私数据或组织内部的专有业务数据，并希望将这些数据严密保留在本地内部。第三则是为了实现“意图感知设备”（intention-aware devices）所带来的机遇：这类设备可能需要持续监听你的语音，或者配备摄像头不断观察和感知周围环境，许多用户显然希望能够自主决定这些敏感信息是否要脱离设备上传到云端数据中心。因此，我认为这些都是驱动我们将推理部署在边缘端的核心而强烈的理由。
Original English
**Speaker**: So I hope you're as excited as I am about all of this inference being possible on edge devices. You may say, "Well, we could do this on the data center already. Why put it at the edge?" It's a good question. I think there are some very, very good reasons for this. Number one is efficiency. I think many people are already looking at their token usage and thinking, "How much of this could I offload onto a local device or a device on-prem?" There's the fact that you can customize this: maybe you want to use personal data or proprietary data to your organization, and you would like to keep this in-house. And then there's this opportunity for intention-aware devices, where maybe they're observing your speech or have cameras and are constantly looking and listening. A lot of people may want to choose whether that stuff is shared off the device onto a data center or not. So I think these are some really compelling reasons why we would want to put the inference at the edge.
### 开源硬件与开源权重模型的协同创新
**演讲者**：正如我刚才所说，这里还蕴藏着另一层深远的变革。我认为正是这种触手可及的硬件生态与开源权重模型（open-weight models）的强强联合，带来了令人振奋的全新机遇。我不知道现场有多少人是硬件创客（hardware makers），或者对硬件研发感兴趣？现场有做硬件的朋友吗？能举手让我看一下吗？太好了，很高兴看到这么多同行。大家有目共睹，多年以来，人们一直利用树莓派或 Arduino 等模块化、可组合的硬件组件，最初从最基本的原型起步，进而基于这些构件创造出完全颠覆以往的崭新设计。我相信，这种硬件创新能力与当前开源模型的结合，必将激发出巨大而有趣的潜力。
Original English
**Speaker**: I did say there's something else going on here, and I think it's this combination of accessible hardware and these open-weight models. Putting this together is a fascinating opportunity. I don't know how many people are hardware makers or into hardware here. Are there hardware people here? Can I have a show of hands? Good, okay. I'm glad to see that. It's been notable that for many years now, people have been building pretty amazing hardware innovations using these composable modules from like Raspberry Pi or Arduino, maybe in the first instance, and then creating totally new designs based on this stuff. I think that combination of hardware innovation plus these open models is going to open up some really interesting potential.
### 具身机器人实例：Reachy Mini 与端侧应用生态
**演讲者**：台下眼尖的观众可能已经注意到放在那个角落里的微型小鸭（micro duck）了。关于这种硬件与开源模型结合，我个人最钟爱的一个典范——虽然我认为目前这种实践还没有像它应有的那样普及，但它确实是一个极为出色的早期案例——就是 Hugging Face 与 Pollen Robotics 合作打造的 Reachy Mini 机器人。据我所知 Reachy Mini 来自法国，而我非常荣幸能参与 Reachy Mini 的早期 Beta 测试计划。去年圣诞节期间，我还和我儿子一起亲手组装了一台 Reachy Mini，整个过程令人非常愉悦。
Original English
**Speaker**: I think the observant among you may notice that the micro duck is in the corner there. My favorite example of this combination—because I don't think it's really happening yet as widely as it could do, but a really nice early example of this—is the Hugging Face Pollen Robotics Reachy Mini. I know Reachy Mini is from France. I was very happy to be on the beta tester program for Reachy Mini. Last Christmas, with my son, I built Reachy Mini, and I was really happy to see how it works.
**演讲者**：在这台机器人的内部，它配备了一个斯图尔特平台（Stewart platform），能够让它的头部灵活转动以表达各种情感；它还装有一整套伺服电机（servos）用来驱动头部平台运动，其中内嵌了 ARM Cortex-M0 处理器；天线部位配备了专门的作动器（actuators）；Wi-Fi 版本的核心大脑则是一块树莓派计算模块 4（Raspberry Pi Compute Module 4）。借助这样的硬件配置，你既可以在本地设备上直接运行一些轻量应用，也可以利用它向外部服务发起推理请求。这非常酷炫。但更重要的是，这不仅仅是硬件本身，它还配有一套完全开源的 SDK，以及一个庞大热情的开发者社区在持续为这个平台开发各种趣味横生、实用性极强的应用。
Original English
**Speaker**: Inside of this thing, it has a Stewart platform so it can express emotions with its head, and it's got a bunch of servos that move the head platform. This has an ARM Cortex-M0 in there. They have actuators on the antenna. They have a Raspberry Pi Compute Module 4 for the Wi-Fi version. With that, you can either run some applications on the device itself, or you can use that to then call inference services off-device. That's super cool. But it's not just hardware; it's also an SDK which is open source, and it's a community of developers creating really cool applications for this platform.
**演讲者**：这里我想重点介绍一个非常有意思的应用案例：这是由我们 ARM 团队自己的同事 Marco Domingo 开发的应用。这个应用基本上就是让机器人端坐在你的桌子上，只要你一开始摆弄手机、沉迷于无休止地下刷手机屏幕（doom scrolling）或者注意力涣散分心，机器人就会立刻出言制止你。这个应用甚至还赢得了大奖，为他赢得了今年在圣何塞举行的 Nvidia GTC 大会的 VIP 门票以及一台 DGX Spark。顺便说一句，他当时比赛用的还是我的那台 Reachy Mini——我必须声明我对这事一点都不耿耿于怀，但真的干得漂亮，Marco！这确实是一款非常棒的应用。事实上，如果你现在去应用商店，在 Hugging Face Spaces 上收录 Reachy Mini 应用程序的专区里就能看到它。
Original English
**Speaker**: Here's one application example I would like to highlight. This is by our very own Marco Domingo at ARM, and he created an app which basically sits on your desk, and if you use your phone, start doom-scrolling, or get distracted, it tells you not to. This won a prize: it won him a VIP ticket to Nvidia GTC in San Jose this year and a DGX Spark. And it was my Reachy Mini! I have to say I'm not bitter about that, but well done, Marco. It's a cool app, and actually, if you go on the app store, it is there today on the Hugging Face Space where they keep Reachy Mini apps.
### 端侧多模态视觉应用的分层设计与计算约束
**演讲者**：在设计这类端侧应用程序时，开发者面临着非常严苛的工程挑战，因为硬件资源是受到严格制约的。你必须清醒地考量设备上有限的可用计算资源和内存容量，因而往往必须将整个任务进行解耦和分层划分，以便最高效地利用手头现有的硬件资源。例如对于一个多模态视觉应用来说，业界常见的做法通常是先设置一个运动检测门控（motion gate）；当检测到动态时，再触发轻量级的目标检测算法；如果在画面中识别出感兴趣的目标物体，就截取对应的感兴趣区域（Region of Interest, ROI）；接着可能仅对该局部区域提取图像嵌入向量（image embeddings）；唯有经历前面这些层层过滤之后，系统最终才会去唤醒并调用参数量更庞大的多模态视觉语言模型（VLM）。通过这种流水线式的设计，这些……
Original English
**Speaker**: In the design of these types of on-device applications, there's an engineering challenge because you have constraints. You have to think about the available compute and memory, and oftentimes you'll have to divide up this task to make best use of the available hardware. For a vision application, it's quite common to maybe have like a motion gate; maybe this triggers object detection, and if you classify things you're interested in, you grab a region of interest, maybe you do image embeddings, and maybe only then do you go and fire off a VLM. And so these...
### 端侧处理与具身智能交互体验
**主讲人**：任务可以由设备上不同类型的处理器来执行，而在某些情况下，随后可能会完全分流到其他设备，甚至是发送到数据中心去运行。这里还有另一个例子：这是我在 Reachi mini 的测试（Beta）阶段制作的一个应用程序，它完全运行在一台树莓派 5（Raspberry Pi 5）上。我特别喜欢它的一点在于，当你开口说话时，Reachi 会立刻把头转过来面向你。具身智能（Embodied AI）确实有一种神奇的魅力——智能音箱可能只会发出一声提示音或者点亮 LED 灯，但一个具有实体形象的角色把头转向你、注视着你然后再作回答，这种体验真的非常酷。
Original English
**Presenter**: Tasks may be carried out by different types of processor on device and in some cases may then go off to other devices entirely or even to the data center to be run. And here's another example. So this is the app I made during the beta test of Reachi mini and this is running entirely on a Raspberry Pi 5. What I love about this is you know you say something and Reachi will immediately turn his head and there's something magical about embodied AI in the way that you know a smart speaker may just bong or the LED lights up, but having a character turn its head to face you and then respond is a really cool experience.
**主讲人**：在做这个项目时，我使用的是树莓派，它当然具备运行多种不同大语言模型（LLM）的能力；但仅仅出于延迟方面的限制，我选择使用嵌入向量（Embeddings），随后进行语义匹配并据此调用工具。这样你依然可以询问它天气或时间，它也可以在设备端利用文本转语音（Text-to-Speech）技术生成语音并给出答复。
Original English
**Presenter**: And I think in doing this, you know, I was using a Raspberry Pi which is definitely capable of running a bunch of different LLMs, but just because of the latency constraint, I chose to use embeddings and then do a semantic match and call tools against this. So you could still ask it the weather or the time and it could generate using text to speech on device also and reply.
### Marco 的具身 Agent 现场演示
**主讲人**：接下来是 Marco 的演示。顺便提一句，我刚才忘了说，在本次演讲结束后外面的主展区里，Marco 会现场演示这个 Demo，大家可以向他提问。从根本上说，他这里用的是他的 DGX Spark——凭借他之前获奖获得的设备，他拥有了一台性能更强大的硬件平台——并且他正在运行另一个交互式 Agent Demo。这次他直接在设备端本地运行 Mistral 3 模型，并且实现了语音转文本（Speech-to-Text）。随后，他利用文本转语音合成回答并播放出来。除此之外，他还集成了大量的工具调用能力。这样一来，你问它在说什么，它都能回答你。其实，也许我直接播放视频是更好的展示方式，请允许我快速切一下视频。
Original English
**Presenter**: And here's Marco's demo. And by the way, I forgot to say, so in the main expo outside after this talk, Marco will be showing this demo live and you can ask him questions about it. Fundamentally, what we have here is he has his DGX Spark, so he's got a more capable device from his prize winning, and he's running another interactive agent like demo. This time he's using Mistral 3 on device and he's doing speech to text. And then he's creating responses, synthesizing a response using text to speech and playing it back. In addition to that, he's doing a bunch of tool calling. And so he has this ability to, you know, you can ask it what it's saying. It'll tell you, you know, actually, maybe I'll play the video. That could be a better way of doing this. Let me just do that really quickly.
**Marco**：好的。
Original English
**Marco**: Okay.
**Reachi**：在的。
Original English
**Reachi**: Yes.
**Marco**：你能向全场的观众打个招呼吗？
Original English
**Marco**: Can you greet the room?
**Reachi**：大家好，我是 Reachi。很高兴见到各位！
Original English
**Reachi**: Hi everyone, I'm Reachi. Nice to meet you all.
**Marco**：我这里有一张 AI Engineer 大会的胸卡，来看看 Reachi 能对它做些什么。嘿，Ree。
Original English
**Marco**: I've got a badge here for AI engineer and let's see if Reachi can do anything with it. Hey Ree.
**Reachi**：在的。
Original English
**Reachi**: Yes.
**Marco**：你能见见这位参会者吗？
Original English
**Marco**: Can you meet the attendee?
**Reachi**：请出示你的胸卡。好的，扫描到了，请稍等片刻。嗨，Dominic，很高兴在 AI Engineer 大会上见到你！我看到你来自 ARM 公司。ARM 以设计基于 RISC 的 CPU 架构而闻名，这些架构驱动着从智能手机到超级计算机的各类计算设备。
Original English
**Reachi**: Show me your badge. Yes. Got it. Give me a second. Hi Dominic. Good to see you at the AI engineer conference. I see you're from ARM. They're known for designing risk based CPU architectures powering everything from smartphones to supercomputers.
**Marco**：最后，你能跳支舞吗？
Original English
**Marco**: Now finally, can you dance?
**Reachi**：感觉怎么样？动作太夸张了吗？还想看我再跳一次吗？大家回见啦，Reachi 要去充电了，祝大家今天过得愉快！
Original English
**Reachi**: How was that? Too much? Want me to dance again? See you later, everyone. Reachi's off to recharge. Have a great day.
**Marco**：太棒了。
Original English
**Marco**: All right.
**主讲人**：顺便说一下，我真的很喜欢 Reachi 通过 Mistral 模型所展现出来的角色魅力与个性，特别是那句“太夸张了吗（Too much）”的演示实在太酷了。正如我刚才所说的，如果大家去到展厅外面，可以亲自去看看它能否读取你的胸卡、甚至为你占卜算命等等，实际与它面对面互动真的非常有趣。
Original English
**Presenter**: So, I love the character that and personality that Reachi has through Mistral by the way. "Too much" is a really cool demo. If you go outside, like I said, you can actually go and see if it can read your badge and tell your fortune, etc. It's really cool to actually interact with it in person.
### 意图经济与无感环境智能
**主讲人**：以上是几个具体示例，但我一开始确实谈到了“意图（Intent）”。大家可能听说过“意图经济（Intention Economy）”的概念；与之相对的“注意力经济（Attention Economy）”，则是吸引人们把目光停留在应用程序上、展示广告等等。而意图经济是指去理解人们真正想要什么，并在用户无需把所有细节都逐一明说的情况下就提供解决方案——我认为这就是我对它的定义。
Original English
**Presenter**: So, those are a couple of examples, but I did talk at the beginning about intent. And so, you probably heard about the intention economy, and you know, the attention economy is, you know, you get people looking at apps and there's adverts and stuff like that. The intention economy is understanding what people want and providing solutions to it without them having to spell out everything. I think that's the way I would describe it.
**主讲人**：这里有一个通过这种意图可能实现的交互小例子。比如，你床头有一盏兼具夜灯功能的 Reachi，它同时也接入了你的 Home Assistant 智能家居系统。如果你对它说你今天的工作结束了、准备休息了，普通的系统可能无法理解这句话背后的深层含义。但如果它掌握了你对空调温度或灯光亮度的偏好，它就能主动去执行这些操作。只要它具备环境上下文感知能力，如果 Reachi 能够观察、聆听并记住它所接收到的情境记忆，它也许就能察觉到：“哦，你的伴侣已经睡着了。”因此，Reachi 这时可能什么话都不说，只是轻轻点点头，把灯光调暗，然后自己也进入休眠状态。我认为，这种通过更少的设备交互来提供更优质用户体验（UX）的能力，正是这一领域的核心关键。这仅仅是一个微小的示例，我相信还存在成千上万个类似的场景。
Original English
**Presenter**: And here's a little example of an interaction that could be possible through this intent. So you know if you have for example Reachi as a bed night lamp there that's also connected to your Home Assistant system for example, if you tell it you're done for the day it may not be able to interpret that right if it understands your preferences in terms of AC temperature or lighting it can go and act on it if it's got ambient context and if Reachi is maybe looking or listening and remembering has memory about the context it's receiving. It can maybe see, oh, your partner's asleep. And so maybe Reachi doesn't say anything. It just nods and it kind of dims the lights and it goes to sleep itself. And I think, you know, this ability to deliver a better UX with fewer interactions with a device is really the key to this thing. This is just one small example and I think there are thousands of others.
### 具身智能架构与端云协同调度
**主讲人**：在这一构想中，不可或缺的核心事实是这里运行着一个 Agent。我们之前讨论了视觉流、音频流和传感器数据流，以及可能与 Home Assistant 系统的集成，这些在本质上都是事件，或者说是可以从 Agent 系统中进行查询的状态。当然，对于这个 Agent 而言，它有一个运行在 CPU 上的管控框架（Harness），该 CPU 承担着大量的工作，用于管理上下文和记忆。我认为这里的核心问题在于：有多少推理工作可以在设备本地完成，又有多少需要分流到设备之外运行？我认为这正是当今整个行业都在重点攻关的编排调度（Orchestration）问题。毫无疑问，任务感知路由（Task-aware Routing）以及判断一项任务是否适合在本地设备上运行的能力，是极其关键的一环，我们稍后就会讨论这一点。
Original English
**Presenter**: And inherent in this is the fact that you know there is an agent running here. So we talked about vision and audio and sensor streams and maybe integration with Home Assistant. These are effectively events or maybe state that can be queried from an agentic system. And of course, you know, with the agent, there's a harness which is running on a CPU and that CPU is doing a bunch of work managing that context and the memory. And then I think the key thing here is how much of the inference can be done off or off of the device. And I think this is like an orchestration problem which a lot of the industry are focusing on today. Certainly, task aware routing and being able to judge whether a task is possible to run on a device is a really key one. We'll look at that in a second.
### 嵌入式工业系统中的监督 Agent 与极端容错
**主讲人**：最后，我们之前讨论了很多关于消费级应用以及人机交互的内容。但在现实世界中，还存在着大量的嵌入式系统，它们配备了计算单元，广泛应用于工业生产、智能温室或各类商业楼宇系统中；在这些场景下，设备主要是在监测传感器数据或执行机械控制，而几乎没有直接的人类交互。因此，我非常喜欢去思考这成千上万个散布各处的嵌入式系统：是否有可能构建一种“具身 Agent 嵌入式系统（Agentic Embedded System）”？我并不是说由 Agent 来执行所有的具体控制任务，而是让 Agent 扮演监督者（Supervisor）的角色。当系统出现带外异常时——换句话说，如果发生了一个系统设计者当初未曾预料到的故障，这些 Agent 是否能够自主应变并加以修复？
Original English
**Presenter**: And finally, you know, we've been talking a lot about consumer-based applications and how people interact with devices. There are a ton of embedded systems out there that have computers that actually, you know, maybe they're in industry or smart greenhouses or, you know, all kinds of commercial building systems where they're mainly monitoring sensors or maybe doing actuation but with limited interaction from a person. And so I really like to think about this idea of those thousands of embedded systems that are out there. Is there a potential for, you know, an agentic embedded system? So I'm not saying that the agent carries out all of these tasks, more that the agent takes the role of a supervisor and when things happen out of band, in other words, you know, if there's a fault that the designer of this system hadn't anticipated, can these agents improvise to go and fix them?
**主讲人**：我认为有一个思想实验非常引人深思：我本人也是个太空迷，旅行者 1 号（Voyager 1）是目前距离地球最遥远的人造物体，它已经飞出了我们的太阳系。几年前，该设备上的一个内存芯片发生了故障，NASA 的工程师最终成功编写了一个补丁，通过重新路由内存寻址修复了该错误。但他们面对的是长达约 22.5 小时的双向信号通信延迟，而且当时的通信传输速率只有每秒 100 比特左右。因此，探讨能否使用 Agent 来维护这些深空系统、实时监测并确保它们保持良好健康状态，是一个非常值得深思的问题。我认为这正是混合 AI（Hybrid AI）领域极其迷人的探索方向之一。
Original English
**Presenter**: And the kind of thought experiment I think is really interesting is, you know, I'm also a space geek, right? So Voyager 1 is like, I think it's the furthest man-made object from the earth right now and it's left our solar system. And there was a fault a couple of years ago with one of the memory chips on this device and NASA were able to create a patch to basically reroute this and fix the error, but they had a roundtrip delay of around 22 and a half hours and it was something in the region of 100 bits per second communication to do this. And so, you know, it's interesting to consider: could agents be used to preserve these systems and monitor them and keep them in good health? And I think this is one of the really interesting avenues for exploration on the topic of hybrid AI.
### 混合 AI 路由策略与多步任务评测
**主讲人**：该领域有大量极具启发性的研究论文，同时也有一些开源项目（例如 LiteLLM 或 RouteLLM 等），你可以将它们应用于这一挑战：推理究竟能否在这台资源受限的边缘设备上由合适尺寸的模型运行？还是需要将其发送到外部或内部性能更强悍的硬件平台上去执行？前几天我正在做一些实验，发现这里面有几种不同的路由策略。我认为“嵌入向量核（Embedding Kernel）”是最快的一种策略，因为你根本不需要真正去运行输入的 Prompt。你只需要构建某种校准集，对比那些在本地成功运行的 Prompt 以及运行失败的 Prompt，进而分析哪个与当前用户的提问 Prompt 更相似。
Original English
**Presenter**: So you know there are a ton of interesting research papers in this area and there's some open source projects, you know, LiteLLM or RouteLLM, that you can apply to this challenge of, you know, can inference run on this constrained device in a model that fits there, do I need to send it externally or internally to a more capable piece of hardware. And so I was running some experiments the other day that, you know, there's a couple of different strategies. I think that the embedding kernel is the quickest because you don't actually have to run the prompt. You're really just, you know, you have some kind of calibration set. You look at prompts that successfully ran locally and ones that didn't and you see which one is more similar to your prompt in question.
**主讲人**：市面上还有其他一些解决方案，比如校准不确定度（Calibrated Uncertainty），或者在实际运行推理后进行解码器状态探测（Decoder State Probing）。也就是说，你已经在本地初步起草了对 Prompt 的响应，但如果模型置信度偏低，或者被分类为可能会执行失败的任务，那么此时你可能就需要将该任务转交分流到设备之外去执行。我不得不说，这些都是非常令人兴奋的前沿课题。大家知道，目前的这些基准评测大多还只是针对单轮 Prompt 的评测集；虽然也有诸如 PinchBench、Chlor（是叫 Chloral 吗？）等一系列针对多步任务（Multi-step Tasks）的评测方案，但我个人往往认为，最有效、最真实的评测标准始终还是你自己的实际应用场景……
Original English
**Presenter**: There's other solutions out there like calibrated uncertainty or even decoder state probing which are possible once you've actually run inference. So you've already drafted a response to the prompt and it maybe then you need to kind of defer this off device to be run if the confidence is low or if it classifies as something which would have failed. This is super interesting stuff I have to say. You know these evals are really just looking at single prompt eval sets. There are, you know, there's PinchBench, Chlor—is it Chloral?—there's a bunch of evals looking at multi-step tasks. I tend to think, you know, it's really your own application which is the best...
### 端侧高效模型与 Arm 算力平台的软硬件协同优化
**Dominic**: 在这些应用场景中，这是非常值得深思的一点。前面我已经讲了很多，关于如何将真正高效的模型与易于获取的硬件结合起来，以及这种结合所带来的启发——即创造出那些经过更深思熟虑、更加内敛且不具侵扰性的设备。我认为这种方向对每个人都是大有裨益的。
当然，我也必须聊一聊支撑这一切底层的硬件架构。正如我在开场时提到的，我就职于 Arm 公司，而 Arm 提供的处理器解决方案覆盖面极广：从基于 Cortex-M 和 Ethos NPU、专为可穿戴设备或传感器节点设计的超低功耗芯片，到运行 Linux 系统、搭载 Cortex-A 处理器的边缘平台（比如大家在 Jetson 或树莓派上看到的），一路向上延伸至数据中心基础设施。我们的核心重点在于提供一套通用的软件框架。最近我们刚刚发布了 Arm AI Portal，它不仅汇聚了经过针对性优化的模型，还配备了性能优化工具链。
Original English
**Dominic**: ...thing to consider in these cases. So, I've talked a lot about what we can do with this combination of really efficient models and accessible hardware and just like the inspiration of creating devices that are a little bit more, I think you know, more considered and less obtrusive. I think this is something that would benefit everyone.
Um, but of course, you know, I should also talk about the hardware beneath a lot of this stuff. And I mentioned at the beginning I work for ARM, and ARM has processor solutions that span from these really low-power wearable or sensor-node type devices with our Cortex-M and Ethos, up to Linux-based platforms with Cortex-A which you might find in Jetson or in Raspberry Pi, right the way up to data center infrastructure. And our focus is providing a common software framework. We've recently announced an AI portal which has optimized models and also performance optimization tools.
**Dominic**: 还有一点我非常想鼓励大家去尝试：我会在演讲结尾展示一个二维码，大家可以扫码注册并成为该工具的早期体验用户。我们非常期待收集你们的反馈，以便根据大家实际想开发的各类应用持续改进它。
这里我可以举几个具体的落地例子。例如，我们已经将用于 KleidiAI 的加速算子向上游合并到了 llama.cpp 以及 ONNX Runtime 中。因此，当 Gemma 模型推出时，它便能在 SME2 指令集上以极佳的性能运行。
但这不仅仅局限于最新的前沿模型，我们也充分兼顾了那些已被广泛采用的成熟模型和存量硬件。我个人非常喜欢一个我们刚刚发布（或者说马上就要正式推出）的优化案例：针对树莓派 5（Raspberry Pi 5）的指令集架构专门进行了优化的 Ultralytics YOLO26。经过这项优化后，它取得了 1.87 倍的推理加速。
Original English
**Dominic**: And one thing, you know, I would definitely encourage you—I'll show this QR code at the end. Please register, become an early access customer to this thing. We're interested to get your feedback and improve this for the applications you'd like to develop.
And just a couple of examples. So, you know, we've upstreamed kernels for KleidiAI into llama.cpp into ONNX Runtime. So when Gemma came out, it ran optimally on SME2.
Um, but it's not only the new stuff. There's also consideration for really widely used, kind of established models and hardware. So I really like this example which we have just released or due to release very shortly, which is Ultralytics YOLO26 optimized specifically for the Raspberry Pi 5 ISA. And so they got 1.87 times faster inference.
**Dominic**: 我认为这里最关键的一点在于：通常当你查看这类模型在 CPU 上运行的帧率指标时，它们往往会占满全部四个核心。在实际的工程应用中，这种把资源吃满的状态是非常不理想的——你总希望系统能留出一些算力余量（headroom）来处理其他并发任务。而这项优化的根本意义，就在于能腾出整整两个 CPU 核心，让你可以从容地运行应用主进程，或者对模型的推理输出做后续的算法处理。这确实是非常振奋人心的成果。
如果你想获取更多资源，屏幕上就是 developer.arm.com/ai 的二维码。欢迎大家注册并申请我们优化工具的早期访问权限。此外，我也非常乐意在 LinkedIn 上与各位交流，稍后我也可能会在 Reachy 机器人展台附近停留。顺便再提醒一下，Marco Domingo 大概会在下午 4:00 出现在展区展台，也就是接下来的茶歇时间，他会带着 Reachy Mini 在那里。如果大家想找他交流或者体验互动 Reachy，欢迎前往。非常感谢大家！谢谢！
Original English
**Dominic**: I think the key thing to point out here is, you know, typically when you look at the FPS numbers for some of these models, they're occupying, if they're running on CPU, all four cores. And really in an application that is not ideal. You would like some headroom to do other stuff. And this kind of optimization is the difference between, you know, having two cores free to run your application, run other kinds of algorithm on that output. So this is really interesting stuff, I think.
Okay. So, if you would like to get more resources, there's that QR code again for developer.arm.com/ai. Please sign up. Yeah, there's early access to our optimization tools which I would encourage you to get into. Um, also I'm really happy to connect with you either on LinkedIn or I'll be out by Reachy later on. And so, yeah, just to say again, so Marco Domingo, I think he'll be there at 4:00 PM. So, not directly after this, but at the break, he'll be on the expo stage with Reachy Mini if you'd like to go and chat with him and interact with Reachy. All right. Thank you very much. Cheers.
### 主持人串场：引出 Cognition 与大尺度代码智能体演讲
**Host**: 大家感觉怎么样？都还不错吧？太棒了。你们猜到我接下来要干嘛了对吧？大家状态怎么样？好的，我相信接下来的这场演讲绝对精彩绝伦！
非常感谢 Dominic 带来的精彩分享。接下来我们要迎来下一位演讲嘉宾，他将为我们分享面向大规模代码智能体的智能体式 MapReduce（Agentic MapReduce for Large Scale Coding Agents）。我对这个议题充满了好奇，非常想弄清楚这背后究竟意味着什么。让我们以热烈的掌声欢迎 Cognition 的工程副总裁（VP of Engineering）Yanis Storakis 登台！
Original English
**Host**: How you feeling everybody? Yeah. Good. Yeah. Okay. You know what I'm going to do, right? How you feeling guys? Okay. I think the next one is going to be great. Yeah.
Well, thank you so much Dominic for the presentation. Now we're going to move on to our next speaker who's going to talk to us about agentic MapReduce for large scale coding agents. I'm super curious about this one. I want to see what this means. But please join me in welcoming to the stage VP of Engineering at Cognition, Yanis Storakis.
### 走进 Cognition 与 Devin：从局部代码任务到全代码库级挑战
**Yanis Storakis**: 好的，让我看一下……哎呀，我是不是接错投影仪了？经典现场状况。请稍等我几秒钟调试一下。嗯……好了，投上去了，是个好兆头。现在我得把屏幕设成扩展模式。啊，幻灯片出来了，太棒了！
非常感谢邀请我来到这里。能够站在巴黎 AI Engineering 的讲台上，面对台下如此优秀的同行朋友们分享，我深感荣幸。今天我将要分享的内容令我非常兴奋——那就是一种专门用来解决超大规模、涉及整个代码库（codebase-wide）级别问题的新方法。好的，让我们正式开始。
先简单介绍一下我们公司。我们是 Cognition，一家应用型 AI 研究实验室。我们团队的初始基因深深植根于前沿 AI 研究、ACM 算法竞赛与国际数学奥林匹克竞赛。我们的总部位于旧金山，但现在的团队成员已经遍布全球各地，包括伦敦办公室——这也是我们的欧洲运营大本营，我本人就在这里工作。我们正是 Devin 的创造者。
Original English
**Yanis Storakis**: Okay, let's see. Oops. Am I connecting to the right projector? Classic. Please give me a few seconds to sort it out. Uh-huh. That's good. That's good. It's promising. And I need to see how I extend my screen. Ah, slideshow. Excellent.
Thank you very much for having me here. It's such a great pleasure to be speaking to AI Engineering in Paris, this great audience. Um, very excited about what I have to show you today. Um, an approach on tackling very large codebase-wide problems. And yeah, let's get going.
Um, a few words of intro for us. We are Cognition. We're an applied AI research lab. Our founding DNA is in AI research, competitive programming, competitive maths. We're based in San Francisco, but we are now all around the world including London. This is our European base of operations where I'm based and we are the makers of Devin.
**Yanis Storakis**: 简单介绍一下，Devin 是业内首个部署在云端的自主式 AI 软件工程智能体。如今，它已经成长为一个成熟的工程平台：在形态上，它不仅支持云端运行，还拓展了本地终端与桌面端等多种使用方式；在工程深度上，它覆盖了我们在整个软件开发生命周期（SDLC）中所提供的各类深度集成与智能体角色能力。
我们已经将 Devin 部署到了全球众多最复杂、要求最严苛的组织机构中。这其中包括诸如高盛（Goldman Sachs）、花旗（Citi）、桑坦德（Santander）等具有系统重要性的跨国金融巨头，一路覆盖到美国陆军（US Army）、大型科技公司，以及行动极为敏捷的原生技术初创团队。我们由衷热爱攻克那些极其硬核、充满挑战的复杂编程难题。借此机会，我想结合我们的实战经验，深度剖析一下开发者与智能体在工程中所面对的各种问题类型。
Original English
**Yanis Storakis**: Devin is the first AI coding agent in the cloud, which now has grown as a platform both in terms of different form factors—both local and desktop—and also in depth in terms of the different integrations and agentic personas across the SDLC that we provide.
Um, we deploy Devin across the most sophisticated organizations in the world. They range from systemically critical banks—Goldman, Citi, Santander—all the way to the US Army, big tech, smaller fast-moving tech-native startups. And we love tackling really, really difficult coding tasks. And I want to reflect on our experiences on sort of the classes of problems that one can see.
### 局部任务与全局代码库任务的本质区别
**Yanis Storakis**: 我们可以从两个核心分类开始切入。第一类，我们姑且称之为“局部型智能体工作负载”（localized agentic workloads）。这基本上涵盖了你日常与智能体交互的大多数中位数会话场景：比如，“我想实现某一个具体的业务功能”；或者“针对代码库中的某一个特定模块，我想为其编写单测或提升现有的测试覆盖率”；再或者“针对数据库代理模块（DB proxy）的某一部分切片，我想对其执行 SQL 查询优化”，诸如此类。
这类工作所面对的都是高度有界的问题（bounded problem）。它们通常只涉及代码库中一个高度相关的有限文件集合，这些文件本身就清晰地将整体逻辑模块化了。同样重要的一点是，当我指挥智能体去执行这类任务时，我要么可以直接明确地把目标文件路径传给它；即便我不明确指定，智能体凭借自身能力，也能非常轻而易举地检索并定位到该功能模块所在的邻近代码区域，从而顺利推进工作。
Original English
**Yanis Storakis**: Let us consider, let's start by think two categories, okay? Let's call them kind of like localized agentic workloads, which are basically kind of like take your median session with an agent: "I want to implement a feature," or "Given a particular area of the codebase, I want to kind of like write some tests or augment the existing test coverage," "Given a particular like slice of my DB proxy, I want to optimize queries," and so on and so forth.
This concerns work that's kind of like a bounded problem. It is like a set of relevant files in the codebase, okay, that basically modularize the whole logic. And equally importantly, if I were to tell an agent what to do, I will either point them explicitly to the files or, if I don't do, they can trivially go and find the relevant area neighborhood of the codebase to carry out the task.
**Yanis Storakis**: 现在，让我们来看看另一类截然不同的任务——我将其简称为“全局代码库任务”（codebase-wide tasks）。顺便提一句，这里所说的“全局代码库”，其范畴绝不仅限于单个代码仓库，它完全可以是跨多个代码仓库的庞大工程。
让我们拿一个今天会反复探讨的典型场景为例：安全漏洞扫描（security scanning）。一个应用的整体安全态势（security profile）到底由什么决定？我们知道，代码库里每个独立模块自身都有其局部的安全防护与质量等级。但同样关键、甚至往往更加致命的，是不同模块在真实世界动态交互与组合调用的方式——这些独立的调用路径彼此链接串联之后，就会暴露并衍生出系统级漏洞。
另一个类似的场景是提取公共逻辑。我们合作的许多客户都拥有极其庞大的代码资产，他们希望将跨模块甚至跨系统的数据计算方式统一化——比如金融公式的处理逻辑，或是与底层数据库交互的基础设施封装。
Original English
**Yanis Storakis**: Now let us consider what I would kind of like simply call codebase-wide tasks. And by the way, codebase-wide doesn't only mean like a single repo. You can have multiple repos. Let's take, for example, a case that we're going to revisit again and again today, like security scanning: what is the security profile of my application? We know that there are like the individual security kind of like quality levels of different modules of the codebase. But what is equally important is the different kind of like the ways are interplay and how those things can evolve in the real world, how they can connect in order to reveal vulnerabilities.
Similar, in a similar vein, extracting common logic. We work with clients that have like really, really large codebases. They want to commonize ways they treat arithmetic, financial formulas, or even kind of like infrastructure ways of interacting with the database.
**Yanis Storakis**: 解决这类全局任务，首先它天然是一个大规模的代码搜索与检索难题；但与此同时，它更是一个深度语义理解的挑战——你必须能够精确发掘并理清所有细微的边界特例（nuances），明确抽离出来的公共库到底需要满足哪些约束、遵从什么样的协议与行为规范。
组件之间存在着错综复杂的相互交织影响。还是回到安全扫描的例子：如果单独审视某一个微服务，它的代码安全质量可能看起来是过关的，即使扫出某些瑕疵，也只是属于低优先级的脆弱点，完全可以先挂在研发代办任务列表（backlog）里。然而，一旦把这些独立的微服务链式串联起来，这些看似低风险的缺陷就很可能拼凑出一套致命的高危漏洞利用链路（critical exploit chain）。
正因如此，对于这一类全局性任务而言，只有当你站在整个工程代码库的全局全貌，站在整个部署系统、甚至跨多个应用生态的整体高度去审视时，产出的分析结论才是真正可信的。
当然，有人可能会反驳说：“这不正是我们编写端到端（E2E）测试的初衷吗？这不正是我们在软件工程生命周期中设立多道代码质量卡点（quality gates）的意义所在吗？”
确实如此！我们在工程上的终极目标永远是实现“左移”（shift left）——也就是尽可能在更早的开发阶段就捕获这些问题，对系统全貌保持洞察。
如果我们进一步深入探究智能体在应对这类跨代码库全局任务时所面临的核心挑战：假设我向智能体提出要求：“请帮我找出某个特定数学公式或特定业务逻辑的所有存在实例，它可能以多种不同的形态和变体散落在代码库的各个角落，因为我需要将其重构并提炼成一个统一的公共基础库。”
此时，摆在面前的第一个巨大挑战，就是如何在大海捞针般的代码库中精确定位并检索出所有待处理的工作点……
Original English
**Yanis Storakis**: They're all sort of like, first of all, it is a search problem, but it's also a problem of sort of like understanding and finding all the different nuances, okay, by which a common library needs to be extracted, need to conform to, and so on and so forth. There's a complex interplay between the components. Take the security example: the security quality of my individual, let's say, microservices is okay, could be kind of like, you know, okay, if I were to look at them in isolation, low-priority vulnerabilities, they can be in the backlog. But when chained together, they can give rise to a critical exploit. And then, therefore, like for those tasks, the result is trustworthy when you consider the totality of the codebase or the totality of the deployed application or sets of applications.
All right. And of course, one can say kind of like, yeah, that's why we have end-to-end tests, okay? That's why kind of like in our SDLC, we have the different quality gates, and so on and so forth. Absolutely. Our goal is always to shift left, right, to catch those things and be aware of those things as soon as possible.
Now if I were to double-click a little bit on sort of like what the challenges an agent could face in those codebase-wide tasks: I'm telling, "Can you please find, I want to find instances of a particular formula, of a particular logic that appears in different variations across my codebase because I want to extract to a common library." Okay, there is a challenge of finding the work in...
### 单智能体在全代码库任务中的困境与编排构想
**演讲者**: 首先，这是一个搜索问题。智能体往往需要耗费大量时间进行检索与回溯，从而白白消耗掉海量 Token。其次，是上下文污染的问题。随着搜索过程的推进，会累积起极其庞大的贪婪信息，这些高度多变且异构的上下文将激烈争夺模型的注意力资源。第三点，则是我们实际上究竟该以什么作为终止判断标准？
Original English
**Speaker**: ... the first place. Okay, it is a search problem. Okay, agents will spend a lot of time searching and backtracking like burning tokens. Okay, number two, polluting the context. There is a great greedy amount that's going to build up that is going to be competing for attention. Very very variable and diverse type of context. And number three, what is effectively our stopping criteria? Okay.
**演讲者**: 再次，为了让这个论点尽可能严密强固，大家肯定会说：“是的，显而易见，我们绝不可能在单次会话中去解决那么庞大复杂的全局问题。这里必然需要引入某种智能体编排的概念。”也就是说，我们需要设计出某种智能体外层循环机制，将整个问题分解拆解为不同的子任务，并构建出一个有向无环图（DAG）来定义各项子任务之间应当如何流转完成，接着再在智能体内部建立内层循环。我们必须把这整套系统工程搭建起来，因为很显然，谁也不可能单凭一句‘请帮我修复代码库里的安全漏洞’的单次 Prompt 就能达成目标，这是绝对不切实际的。
Original English
**Speaker**: Again, to make this as strong argument as possible, one will say, "Yeah, of course, we're not going to solve those big problems, okay, in a kind of like in a single session. There will be some notion of agent orchestration. Okay, there will be some notion of okay, we're going to have kind of like an the agent outer loop. We're going to break down the problem into different tasks. Okay, we're going to define a dag how those things need to be completed and then we're going to have some notion of inner loop and the agent. We have to build all that stuff. Okay, because obviously no nobody is going to oneshot prompt hey kind of like please fix security vulnerabilities. Absolutely.
**演讲者**: 我今天想要向大家阐述和展示的核心论点，正是我们所思考并提炼出的这一套全新架构框架。它无论在最终输出效果性能上，还是在运行成本控制上，都能带来最佳的表现，并且我们已经将其作为我们产品中的一等公民核心概念加以实现。我之前已经简要提及过单智能体推理的局限，这里就不再过多赘述搜索耗时、上下文污染等细节了——结论就是，单智能体模式根本无法胜任超大规模、涉及整个代码库级别的复杂任务。那么，我们究竟是如何攻克这一难题的呢？
Original English
**Speaker**: My argument is to today is to present to you a framework okay how we think about the problem that gives the best um both in terms of outcome performance and cost and we make this okay as a first class notion in our product um I touched upon this before kind of like single agent reasoning I'm not going to labor too much on this point okay both in terms of kind of like time spent searching context pollution and so on and so forth single agent problem does not solve very very large codebasewide tasks. How we approach this?
### 借鉴分布式系统：将 MapReduce 引入智能体编程
**演讲者**: 我们提出的方案，是针对源自分布式系统经典的 MapReduce 框架进行的一种变体改良与增强扩展。现场有多少同学曾经使用过 MapReduce 架构？太棒了。让我们极快地回顾一个经典例子：Google 最初提出 MapReduce，是为了在分布式文件系统上所存储的超大规模数据集上执行高效分析处理。举个最典型的教科书级例子：假设我们在分布式文件系统上存有海量的文本语料和各类文件，现在需要进行词法分析，简单来说就是统计每本书里的词频；这就可以拆分为一系列并行运行的 Map 函数，它们都是无状态的纯函数，负责分别统计单本书的词数；最后通过 Reducer 阶段将所有局部的统计结果归纳合并在一起。顺着完全相同的思路，经过针对性改造后，我们希望将这种架构范式完整迁移引入到智能体代码编程（Agentic Coding）的领域中来——在这里，我们所面对的不再是海量的数据集，而是超大规模的代码库。
Original English
**Speaker**: Okay. So um we propose okay a variation or an augmentation of the map reduce framework an idea from um distributed systems. How many people here have worked with map reduce? Awesome. So super super quick example map reduce introduced by Google was a way to solve to perform analysis on like a really really large data sets that they are distributed that they live on a distributed file system. Okay. So you if the classic example that we give hey we have a huge textual corpus on the distributed file system um different files and we need to do some lexographic analysis I'll keep it simple count the number of words on each book okay will be a series of map functions these are simple stateless as in pure functions that they kind of like we count each book they can run in parallel and there's a reducer step that consolidates the results. In a very similar vein with a adaptation, we want to port this into the agentic coding world. Okay, where instead of like huge data sets, consider huge code bases.
### 智能体 MapReduce 的完整生命周期与核心架构
**演讲者**: 这里是整套机制运作流程的高层全景概述。首先，我们在最开始引入了两个前置步骤：由一个主智能体（Main Agent）负责制定整体规划。这个规划会结合你的输入 Prompt，充分吸纳已具备的技能（Skills）以及仓库中现有的领域先验知识。主智能体会综合评估所有这些上下文输入，进而生成一系列精确定位的‘选择器’（Selectors）。所谓的选择器，本质上是一些确定性的检索函数；一旦运行它们，就会对整个代码库生成一份编目清单，明确筛选出有待深入探查的候选代码区域。第三步，针对每一个划分出来的候选区域，我们会并行启动一系列独立的子智能体（Sub-agents）。这些子智能体将专门且仅仅聚焦于分配给自己的特定区域展开详尽分析——这就是 Map 阶段。紧接着进入 Reduce 阶段：主智能体重新接入，统一汇集、考量并处理来自所有不同子智能体回传的精炼结论，开展对账与整合分析。这种归并分析可以包括重复数据的清洗去重、优先级排序，或者基于综合结果开展更深层次、更丰富的全局推理。
Original English
**Speaker**: Okay, an overview of how this thing would work. Okay, firstly, there are two steps that we introduce at the beginning. There is a main agent that creates a plan. The plan is takes your prompt or takes including taking skills, taking existing knowledge from the repo. Okay, whatever you provide and performs an anal and those all those considerations in order to create a series of selectors. Selectors are areas, okay, are deterministic functions, okay, when when you run them, okay, it will create a catalog of your codebase with candidate areas for investigation. Number three, for each of those candidate areas, we spin a series of parallel sub aents where they look these areas and only these areas and they perform and then the analysis. This is the map step and then there is the reduce step. The main agent again considers the results the combined results the distilled results from all the different sub aents. Okay. And performs some reconciliation some consolidation analysis. This this can be duplication prioritization or some more richer analysis on the combined result.
### 深度拆解：以安全漏洞检测与修复为例
**演讲者**: 让我们进一步深入下潜剖析这些核心概念。这里我以‘软件安全’作为一个驱动示例：面对眼前这样一个庞大的代码库，我们希望弄清它的整体安全质量和安全水位究竟如何。为此，我会向系统提供威胁模型（Threat Model），并输入关于该代码库的各项必要补充上下文。接下来，前置准备步骤所做的事情就是：编写出确定性的搜索函数，用于在安全语境下精准定位代码库中值得重点关注的敏感区域。大家完全可以想象到，这些区域包括 API 边界、Token 铸造与流转传递的位置、SQL 注入入口点、各类代码输入边界等等。这一步产出的直接结果，就是刚才提到的确定性选择器。它们在代码库上执行后，会切分出一批批任务切片或代码分片（Shards），使得我们随后能够针对每个分片精准分发运行一个子智能体。如此一来，每个子智能体都拥有极其干净整洁且边界明确的有界上下文（Bounded Context），它们可以在各自受限的范围内专心进行排查、记录与汇报，输出针对特定代码区域的有界结论；随后，所有结论统一回传给主智能体，以便在全局安全上下文中对所有分散发现展开全景式的综合推理。
Original English
**Speaker**: Let's kind of like double click a bit on those concepts. Okay. And I'm going to use I'm going to use security. Okay. As kind of like a motivating example. Okay. I have this large code base. And what is the security kind of like the security quality security level of this codebase. Okay. I will provide the threat model. I will provide any additional context on the codebase. Okay. What the pre-work step is going to do is okay, we're going to write this determin a deterministic search function that will identify areas of the codebase of interest in the context of security. You folks can imagine API boundaries um where tokens are being minted and be passed around um SQL entry uh code entry points and so on and so forth. The outcome of that is to produce basically those deterministic selectors. They're going to run and they're going to produce like a series of batches, okay, or a series of shards of the codebase where we can then run each one of those sub agents. Okay, those habages then they have a very nice clean and bounded context which they can perform the investigations and reports and so on and so forth and produce kind of like a bounded outcome for that particular area of the codebase and then the results go back to the main agent where we do kind of like a holistic reasoning across all the different findings in the again in the context of security to make it like a little bit more concrete.
**演讲者**: 为了让这一过程更加具体可感：第一步，我们可能会对搜集到的潜在漏洞进行去重；第二步，评估哪些漏洞的危害最为严重，并给出严谨的优先级排序；第三点——这在全代码库任务中尤为至关重要——那就是排查是否存在‘链式漏洞’（Chained Vulnerabilities）。请大家记住，单个子智能体可能只会告诉你，在它负责审查的局部狭小代码块中发现了一处低危弱点；然而，Reducer 归约步骤所具备的能力，恰恰在于评估这些离散的低危漏洞在组合串联后，是否会碰撞出极其致命的高危安全漏洞？不仅如此，在配套测试靶场环境（Test Harness）的支持下，Devin 能够切实运行起来，进行端到端的实际测试，并模拟攻击路径来验证漏洞链是否真实成立。
Original English
**Speaker**: probably we're going to duplicate number one. Number two, there is a prioritization okay of which vulnerabilities are the most critical and there is a ranking of them. Number three, but this is crucial again in the context of codebasewide tasks. Um presence of chained vulnerabilities. Remember individual agents may tell you that their individual narrow part of the codebase has some low um um criticality finding. But what the reducer step can do is can assess combinations of those low vulnerabil low criticality findings. Can they produce something much more serious? And again with the right hardness, okay, um Devon can run, okay, can actually test, okay, and try to simulate those attack paths.
### 代码库图谱化、漏洞复现与端到端验证
**演讲者**: 如果从更加直观视觉化的角度来理解：我们倾向于将整个代码库视作一张庞大的拓扑图谱（Graph），各个不同代码模块与组件之间存在着错综复杂的调用和引用关联。选择器会精准搜寻鉴权与授权模块、API 入口、数据与 SQL 交互节点等；在借助这些确定性选择器以极低廉的开销对代码库完成初筛之后，我们便会部署各个独立的 Devin / 智能体实例，针对代码库中这些被精巧限定的狭窄路径展开深度安全分析；随后在 Reduce 阶段，Devin 会生成最终的结构化交付结果——第一进行分类分流（Triage），第二实施全面去重，第三实际执行并测试以尝试完整重现这些漏洞，同时有效剔除误报（False Positives）。
Original English
**Speaker**: And to do it kind of like a little bit more visually, we treat the tend to treat of code bases as basically a graph. Okay, all there's all those references between um the different parts of the codebase. selectors look for off areas, look for API entry points, uh data SQL entry points and then from those selectors that they have run cheaply and deterministic in the codebase, we deploy okay individual devons, individual agents that perform security analysis for this particular narrow bounded path of the codebase and then on the reduce step Devon will produce something like this. Okay, where it will sort of like number one triage, number two the duplicate, number three run and test and try to recreate those vulnerabilities including kind of like false positives as well.
### 成本与性能权衡及全代码库任务的通用化前景
**演讲者**: 是的，我特别强调漏洞复现这一点，是因为在工程落地时，你绝对不希望最终只给业务团队堆砌出一大堆形同虚设的 Ticket。大家都深有体会，每次项目发版时运行现有的第二代安全漏洞扫描工具，往往会刷出成百上千条噪音和报警，让人痛苦不堪。我们真正追求的是针对已确证存在的真实漏洞直接交付货真价实、可以直接合并的合规 PR，因此在提 PR 之前能否确凿复现漏洞便显得无比关键。在此我也想借机探讨一下成本（Cost）与性能表现（Performance）之间的平衡关系。诚然，你完全可以直接调用极其昂贵的大模型智能体去粗暴扫描超大代码库，但正如我前面所强调的，至关重要的一点是必须深思熟虑、采用最精巧的手段来最大程度压降搜索带来的开销浪费。在超大代码库中，盲目搜索、上下文污染以及上下文溢出是极其致命的阻碍。而在智能体 MapReduce 架构中，Devin 与主智能体通过编写确定性选择器，切分出大量受控的有界上下文（Bounded Contexts），不仅大幅降低了运行成本，相比于竞品模型更是展现出了显著更优的召回率与准确度。
Original English
**Speaker**: Um so yeah mention about reproducing because obviously at the end of the day you don't want to ship kind of like tickets because we know whenever we have a release you know we run second vulnerability from tools there are like tons and tons of things to turn through we want to be shipping kind of like legit PRs against a known vulnerability so recreating is extremely extremely important now we're pushing the PRs and just want to comment Okay, a little bit on kind of like the relationship between cost and performance. Okay, of course kind of like you can run very kind of like expensive agents. You can run them against really large code bases. Okay, it's what is extremely extremely important as I mentioned earlier is to think hard and have the best possible ways okay to mitigate the cost of searching. Searching is a huge problem in large code bases and context pollution and context overflow. This is solved by the Devon and the main agent in the agentic map reduce writing those deterministic selectors and creating lots and lots of banded context which end up being much cheaper and end up having a better recall profile rather uh compared to competing models.
**演讲者**: 我们在来自 GitHub 的 50 个真实数据集、50 个经典漏洞案例上进行了严格的实测验证，涵盖了 Golang、Python、Java 等多种主流编程语言。另外还有一点值得大家深入思考：虽然我今天主要是围绕安全领域的漏洞检测与修复来介绍智能体 MapReduce 架构，但这套范式实际上完全可以泛化迁移到各种涉及整个代码库级别的多样化工作负载中。比如我经常提及的死代码清理（Dead Code Removal）、从极其庞大的代码库中把公共通用逻辑提炼抽象到统一的中心化公共库，以及全局死代码检测等场景，都能够凭借这种架构取得极佳的成效。
Original English
**Speaker**: Okay. And we've run this on 50 data sets, 50 vulnerabilities from kind of like a GitHub and they run across like a multitude of languages, Golang, Python, Java and so on and so forth. Um, another thing that we should think about, I talked a lot about security agentic map reduce in the context of vulnerability detection and remediation. Um, think about it about it can be a generalized concept across all different codebasewide workloads. Okay, dead code removal I mention a lot extracting common common logic into some centralized library from a very large code base. Um, dead code detection
### 上半场收尾与茶歇通知
**Janis**：等等诸如此类的内容。好的。这也是我们从自身研究出发，逐步将其归入安全范畴，并在我们的产品中提升为一等特性的一项成果。正如我前面提到的，我们是一家应用型人工智能研究实验室，目前正在攻关多个前沿方向，例如基于 Agent 的 MapReduce、构建我们自研的代码大模型 2.0，以及开展像 DevInfusion 这样的模型运行载体（model harness）与 Agent 基础设施相关的基础研究。欢迎大家关注我们的技术博客。如果各位有任何问题，这是我的电子邮箱，非常期待与大家交流讨论，也希望在接下来的会议中与大家线下相见。非常感谢大家！
Original English
**Janis**: and so on and so forth. Okay. And this is something that again we from our research we have kind of like moved it to security and we elevating it into a first class feature in our product. We are as I mentioned kind of like earlier we are applied AI research lab we doing work on areas like agentic map reduce building our own coding models 2.0 know research on model harnesses like dev infusion agentic infrastructure please check out our research blog and if you have any questions this is my email I'm looking forward to your comments and see you around in the conference thank you very much.
**主持人**：非常感谢！请大家再次用热烈的掌声感谢 Janis！好的，各位，我们现在完全有理由休息一下了。我知道今天对大家来说都是漫长而充实的一天，但在茶歇之后，我们依然有非常精彩的讲者阵容等着大家。我们将在下午 4:30 回到这里，请大家尽情享受茶歇时间。好的，待会儿见！
Original English
**Host**: thank you so much let's give it up for Janice once more please All right. Okay. So, we all deserve a break right now. I know that it's been a long day for everybody, but we have a great line of speakers still. Okay. So, we're going to be back here at 4:30, please. And enjoy your break. All right. See you in a bit.
### 下半场开场：迎接 Matt Pocock
**现场广播**：女士们、先生们，请与我一同热烈欢迎本场主持人的登场——来自 Replit 的开发者关系工程师，AI Engineer Paris 2026 大会主持人 Rahul Chevrey！
Original English
**Announcer**: Ladies and gentlemen, please join me in welcoming to the stage your MC for the AI engineer Paris 2026 developer relations engineer at Replit. Rahul Chevrey.
**Rahul Chevrey**：好的，一切准备就绪！大家感觉怎么样？嘿，这已经是本次大会的最后冲刺阶段了！现在正是需要我们凝聚所有精力、向着 AI Engineer 巴黎峰会最后一程全力进发的时刻。对于接下来要登场的这位演讲嘉宾，我感到无比兴奋。坦白说，对于下一位讲者，根本不需要由我来介绍，他本身就名声在外，无需过多赘述。他曾被称为“TypeScript 奇才”（TypeScript Wizard），在开始探讨各类技能工具（skills）之前，也曾因在 Vercel 的杰出工作而广为人知；而现在事实证明，他对人工智能同样拥有非常深刻且独到的见解。
接下来，他将为大家分享如何彻底破除 PR（拉取请求）瓶颈。天哪，我对这个痛点真的是感同身受！现场有人对此感同身受吗？说真的，现在谁不是被排山倒海般涌来的 PR 淹没，结果根本连一个都来不及审查？好了，闲话少说，让我们以最热烈的掌声欢迎 Matt Pocock 登台！
Original English
**Rahul Chevrey**: Okay. All good. How's it going, guys? Hey, this is the last stretch. Okay. So, this is where we need to gather all the energy and we get going for the last stretch of this AI engineer at Paris. And I'm so excited about our next speaker. So our next speaker actually you don't need me. He doesn't need any introduction. He was known as the Typescript wizard and then you know he was also known for his work at Vercel before he started talking about skills and then it turns out that he has really strong opinions and good opinions about AI. So our next speaker is going to talk to you about how to fix your PR bottlenecks. And my god, I relate to that so much. Anybody relates to that? Seriously, like who's who's having like so many PRs coming their way and then they they can't review any of them. So, well, without further ado, please join me in welcoming to the stage Matt Pocock.
### 破除 PR 瓶颈：AI 时代的软件工厂与代码淤积
**Matt Pocock**：大家好！到目前为止大会还过得愉快吗？看起来大家都很尽兴，太棒了！今天我在这里要探讨的主题是“如何解决 PR 瓶颈”。坦率地说，这是一个颇为宏大的标题，旨在解决我认为绝大多数研发团队都在苦苦挣扎的问题——事实上，早在 AI 出现之前，团队就长期受困于此。大家都很清楚，我们的代码仓库里向来躺着海量的 PR，根本没有人有精力去审查。而现在，随着 AI 的介入，它给我们带来了各种各样奇怪的制约与压力，导致未审 PR 的积压规模呈爆炸式增长。为了解决这个问题，我将采用我的“技能体系”（skills rubric）来展开剖析——你们可能听说过它，甚至可能已经实际使用过了。同时，我今天还会在此宣布几项全新的技能工具，希望它们能够切实改善大家处理 PR 的方式，并大幅提升你们审查和提交 PR 的速度。
Original English
**Matt Pocock**: All right. Hello, folks. Having a good conference so far? Having a good conference so far. Okay, good. So, I'm here to talk about fixing the PR bottleneck. And this is kind of a grand title for trying to fix the thing that most organizations struggle with, I think, and have kind of historically struggled with before AI. You know, we have always had huge numbers of PRs just laying around that no one's bothered to review. And this has now increased massively because of the new strains on us because of AI imposing all these weird constraints. And to do this, I'm going to use the rubric of my skills, which is you've kind of heard about, maybe you've used them. And I have a couple of new skills to announce that are going to hopefully improve the way that you do PRs, improve the way that or improve the speed at which you can review them and do them.
**Matt Pocock**：核心在于速度。现在整个行业都在逼迫我们用更少的资源做更多的事，本质上就是要求我们处理更多的 PR、承担更多的工作、交付更多的产出。这正是人工智能所给出的核心承诺：我们将能够利用这些智能体（agents）成倍扩展自身的能力，从而承担海量的工作。这一趋势直接催生了“软件工厂”（Software Factory）的概念——这大概是当下最火热的高频词汇了，凡是和我交流过的人，言必称软件工厂。
在我看来，所谓的软件工厂，其核心特征就在于：过去所有工作都必须由人类主动发起，而现在，我们将其中一部分发起流程移交出去了，有一部分任务的触发完全交由智能体来代劳。比如，你可以在系统中引入像 Sentry 这样的错误分类器；一旦捕获异常，系统就会判定：“好的，让我们直接把它转化为一个修复补丁（fix），或者将其提炼为一个复现用例（reproduction）”，又或者直接自动 @ 相关研发人员。再比如，你还可以接入 PlanetScale 数据库监控，由它自动输出慢查询分析报告；而这份报告随即在软件工厂中触发另一条自动化流水线。请注意，所有这些流程的起点都不是人类，而是由确定性的代码自动化触发的。
Original English
**Matt Pocock**: Speed. Now we're being pushed to do more with less essentially or more PRs, more work, more stuff. And this is kind of the central promise of AI that we're going to be able to use these agents to scale ourselves up to do more work. And this has resulted in the software factory, probably the biggest buzzword of the day. Everyone's talking about software factory that I chat to. And I think of a software factory as primarily something where instead of the human initiating all of this work, we're going to pass some of that initiation, some of the initiation is going to be done by agents. And I think of their maybe from there you have a classifier like Sentry come in and Okay. Okay. Let's turn that into a fix or turn that into a reproduction or maybe I ping people straight away. And maybe you have other things. Maybe you have planet scale hooked up. So it gives you query reports on the slow queries on your database. Maybe that then triggers a different thing of your software factory. All of this is not humans triggering it. It's deterministic code triggering it. Right?
### 刹车机制：抵御“垃圾代码炮”的侵蚀
**Matt Pocock**：因此，这些机制极大地加速了你的软件工厂运转，将源源不断的代码强行推向流水线。但在这种情况下，你迫切需要“刹车机制”（brakes）。为什么？如果你只有永无止境的单向加速，疯狂地把代码往工厂下游硬推，最终迎来的只会是一门轰击不止的“垃圾代码炮”（slop cannon）！你会陷入极其糟糕的境地：仓库里堆满了数以吨计的劣质 PR，烂到你根本碰都不想碰，无法审查，甚至看都不想看一眼。
所以，你必须拥有刹车。刹车是一种主动降速、提升质量的把控机制，能够确保你的代码库不至于演变成软件熵增的噩梦。这是因为，现有代码正是智能体生存和运作的上下文环境；如果你的代码库里充斥着低质糟糕的代码，那它就只会衍生、滋生出更多糟糕透顶的代码。因此，在今天的演讲中，我将重点阐述三道刹车机制，并探讨我们如何能够反直觉地利用这些“减速刹车”，反而实现更高质量的“整体加速”。
Original English
**Matt Pocock**: And so these accelerate your software factory. They push more code through it. But then you need breaks, right? If you just have permanent acceleration pushing stuff through your factory, you're going to end up with a slop cannon, right? You're just going to end up with a ton of slop crappy PRs that you're not going to be able to touch or review or even freaking look at. So, you need breaks. These are mechanisms that slow down, that increase quality, that make sure that your codebase doesn't turn into a software entropy nightmare because code is the environment your agent operates in. And if you have bad code in your codebase, that is going to beget more bad code. And so I'm going to talk about these three breaks in this talk and talk about how we can use them to counterintuitively go faster.
### 代码审查的三层蛋糕模型
**Matt Pocock**：首先是“自动化检查”（Automated Checks）。这是你代码仓库中确定性的检查机制，我们已经沿用上千年了——或者更准确地说，自 20 世纪 50 年代以来就一直存在。这类确定性检查涵盖了代码风格检查（linting）、自动化测试、类型检查以及代码质量指标分析。所有这些工具紧密协作，每次执行时都保持完全一致的行为准则。
在这之上叠加的第二层是“自动化审查”（Automated Review）。在这一层，我们引入智能体来审视我们的代码，并指出：“好的，这些是自动化测试未能捕获的问题”，或者由智能体去评估整个代码库宏观架构层面的合理性。
最后，在最顶层的第三道防线才是“人工审查”（Human Review）——也就是由人类工程师亲自审视 PR。这三个层次共同构成了我们在最终进入人工审查时所面对的“三层蛋糕”。显然，更高的速度必然带来更多数量的 PR。因此，我们在此的核心目标，就是全力仰仗并榨干前两道防线的防护能力，从而让最终的人工审查变得极其轻量与快速。我们必须坚决遏制垃圾代码——这是整个体系的第一原则：如果你大幅拔高了交付代码的基准质量，你最终需要进行的人工审查工作量就会显著减少，因为产出的成果本身就更加优质可靠，进而你被迫介入纠偏的频次也会大幅降低。
Original English
**Matt Pocock**: So automated checks, these are the deterministic checks in your repo that we've had for thousands of years or you know since the 50s. Deterministic checks where we have linting and tests and type checking and code quality metrics. All of these things going together and they all work the same every time. Layered on top of that we have automated review. So we have agents who look at our code and say okay you know these are for the things that the tests didn't catch or this is looking at the structure of the codebase in general. And then on top of that the third layer the final layer is human review. So people looking at the PR. And these three layers form this kind of cake that we end up with when we get to human review. And so more speed of course means more PRs. And so the goal here is to make human review faster by leaning on those first two phases. And we got to stop the slop. That's the first principle here, which is if you raise the quality of the code that you're shipping, you're going to end up doing less human review because it's just going to be better work. And so you're going to end up needing to make fewer interventions.
### 第一道刹车：自动化检查及其内在谎言
**Matt Pocock**：我们先来深入探讨“自动化检查”。自动化检查最大的优势在于成本极其低廉。它的美妙之处在于，它不像自动化审查那样会持续消耗大模型的 Token，也不需要耗费人类的心力与精力，消耗的仅仅是纯粹的 CPU 时钟周期。举例来说，你在每次代码变更时都会执行测试套件。诚然，如果某个智能体引入了一个 Bug，而测试刚好捕获了它，那么为了让智能体排查修复这个 Bug，你确实可能需要支出一些 Token 开销；但在我看来，这些 Token 绝对花得物超所值。自动化检查成本极其低廉，这意味着你完全可以在代码仓库中层层叠加海量的检查规则。事实上，大家现阶段使用的检查手段恐怕还远远不够充分，在运用策略上也缺乏足够的创造力。
然而，自动化检查是会“撒谎”的。持续集成（CI）全部亮绿，难道就意味着这段代码已经完全具备合并条件了吗？不，绝非如此！正因如此，我们向来都需要在这些确定性检查之上再追加一层保障，以便在真正将代码发布上线之前，彻底查明代码中是否存在任何灾难性的隐患。从这个角度来看，后续的自动化审查与人工审查，本质上都是“测谎仪”（lie detectors）——它们的存在，就是为了精准戳破自动化检查所编织的谎言。
Original English
**Matt Pocock**: So automated checks. Now automated checks are cheap. That's the cool thing about them is that they don't cost tokens like automated review does. They don't cost human effort. They just cost CPU cycles. So these are for instance, you know, you run your tests on every code change. Maybe those tests do incur some tokens because let's say an agent creates a bug and the tests catch it, then you need to spend some tokens to go and fix it, but those are tokens pretty well spent in my opinion. So checks are cheap. That means you can layer on loads and loads and loads of them on your repos. you're probably not using enough of them or not being creative enough with your use. But checks can lie. Does a green CI mean that the code is ready for merge? No, it does not. And so we've always needed on top of these checks some extra layer to figure out if there's anything catastrophically wrong with the code before we ship it. And so all of the other phases, the human review and automated review, these are lie detectors. These are for finding lies in the automated checks.
### 谎言实录：同义反复测试与结构过度耦合
**Matt Pocock**：现在，我想首先给大家展示几个真实发生的“谎言”案例。这有助于我们在深入思考代码质量和自动化检查时，切身体会到情况究竟能败坏到何种荒谬的程度。
第一种典型谎言就是“同义反复测试”（tautological tests）——也就是那种仅仅把底层实现代码原封不动再重新断言一遍的无意义测试。Claude Opus 5 简直对写这类测试上了瘾，我至今完全无法理解为什么会这样。举个例子，业务代码里写着：`x_post_character_limit = 280`（X 帖子字符数限制为 280）。大家能猜到它写了什么测试用例来验证这一行为吗？没错，你们大概已经在日常中见过上千次这种荒唐事了。这全是我在实际智能体运行日志中抓取到的真实代码。它居然写道：`expect(x_post_character_limit).toBe(280)`。
业务代码本身就是这么赋值的，而测试用例本质上只是把底层的静态赋值重新断言了一遍！这就是教科书般的同义反复测试。同义反复测试之所以危害极大，是因为它们对代码的内部实现结构具有极度的脆弱敏感性。它们与系统的内部运作机制过度捆绑，这意味着：只要我修改该常量的数值，测试就会挂掉；更糟糕的是，只要我对该常量进行重命名重构，测试依然会挂掉！你的一举一动都死死受制于系统的静态结构之中。
而我还发现了另一个更加令人瞠目结舌、性质更为恶劣的案例。这是一个堪称匪夷所思的奇葩测试：它在这里所做的事情，本质上是在测试系统中的两个对象……
Original English
**Matt Pocock**: Now, I want to show you some of these lies first of all because this helps when we're thinking about code and thinking about automated checks to see how bad it is and how bad things can get. The first is tortological tests. A test that just reasserts the implementation. Opus 5 got addicted to these. I don't quite understand why it would have for instance x post character limit equals 280. Can anyone guess the test that was written to test this behavior? Right? You've probably seen this a thousand times. This is real code from agents or from stuff that I found my agents doing. It said expect x post character limit to be 280. So the implementation looked like that and the test essentially reasserted the implementation. That is a tortological test. And tological tests are bad because they're extremely structure sensitive. They're very sensitive to the actual internal workings of the system. So it means I cannot change that constant without a test failing. But I cannot rename that constant without a test failing. Like I have to do is so tied into the structure of my system. And I found another one which is even more egregious I would say. This is incredible test. What it's doing here is it's essentially testing whether two things in the
### UI 顺序检查与对结构过敏的虚假测试
**演讲者**：UI 元素按正确的顺序显示。所以它的检查逻辑是：确保宣传详情页面——或者更准确地说是视频区域——出现在内容规划区域之后。但它的实际做法并不是将页面渲染到屏幕上，而只是把源文件对应的模块代码直接读取到自己的内存中，然后在里面搜索匹配的目标。它先找到“content plan”，再找到“videos”，然后断言在源代码文本中“videos”必须排在后面。如果仔细想想，这简直太荒谬了，因为我只要稍微调整一下源代码的排版或组织方式，这个测试就会直接挂掉。它对我的代码库结构过分敏感了。这是自动化检查可能失效——或者更确切地说是自动化检查会说谎——的又一种典型方式。
Original English
**Speaker**: UI appear in the right order. So it's checking that the pitch detail page uh or rather the video section comes after the content plan. What it does is it doesn't render it to a screen. It just reads the actual file the module into its own memory and then it finds the right thing. So, finds content plan, finds videos, and then it expects the videos to be after it in the source material, which is crazy if you think about it because I can just like change the way the source material looks and this test will fail. It's too sensitive to the structure of my codebase. So, that's another way that automated checks can fail. And there are also or sorry, automated checks can lie.
### 滥用 Mock 与永远无法失败的测试
**演讲者**：还有一种测试，是从字面意义上讲根本不可能失败的。如果你过去曾经滥用过 Mock，或者滥用过类似的机制，你对这种情况一定会感到非常熟悉。比如，我们这里有一个 `useAudioBoost` 函数。这个 `useAudioBoost` 函数的内部实现调用了 DOM 中的 Audio Context API。如果你不了解这些细节也没关系，但我们在这里做的事情，其实就是用一些假的假方法直接把它桩化（stub）替代掉了。然而事实证明，Audio Context 本身具有一些复杂的报错模式，如果在某些异常特定条件下使用它，它是会直接崩溃的。由于我们采用了这种粗暴的模拟方式，我们的测试在这些报错模式下完全不可能失败，结果我们在生产环境中就会遭遇测试根本无法捕捉到的离奇错误。
Original English
**Speaker**: There are also tests that literally cannot fail. And this will feel familiar to you if you've abused mocking in the past or abused various things. For instance, here we have a use audio boost function. And this use audio boost function uh its internals use the audio context API in the DOM. Don't worry if you don't know any of this, but what we're doing here is we're just stubbing it out with some fake methods. And it turns out that audio context has some complicated error modes and it will fail if you use it under strange conditions. And so just doing this means our tests cannot fail using those modes and we're going to hit strange errors in production that our tests can't fix.
**演讲者**：所以接下来的核心问题是：既然自动化检查是可以被钻空子作弊的——而且你知道，甚至很多时候是以出于善意的方式被绕过去的。AI 在这里并不是故意想要写出糟糕的测试，它只是完全遵循我们的指令，结果写出了与代码结构绑得太死、而没有真正去执行运行代码的测试。那么，我们究竟该如何让自动化检查变得更难被作弊呢？如果我们能够做到这一点，就能显著提升这些检查的质量，从而真正拉高我们的质量基线。
Original English
**Speaker**: And so the question is then if you can cheat on automated checks if you know and even in good faith ways as well. The AI isn't trying to write bad tests here. It's just taking our instructions and writing tests that are too tied into the structure instead of actually executing code. So how do we make automated checks harder to cheat? If we can do that, then we can increase the quality of those checks, which means which increases our quality bar.
### 代码库设计：用深模块规避不良测试
**演讲者**：针对这个问题，我个人非常推崇的第一种手段就是代码库设计（codebase design）。你完全可以通过良好的架构设计，从根本上避开这些糟糕的检查陷阱。那么，优质的代码库设计究竟长什么样？我在之前几次 AI 工程师演讲中其实就提到过这个概念，那就是“深模块”（deep modules）。所谓深模块，就是将复杂的内部行为隐藏在简洁界面之后的模块。这是 John Ousterhout 在《软件设计的哲学》（A Philosophy of Software Design）一书中提出的思想。如果我们来对比这两种模块：模块 A 拥有非常庞大的具体实现，但这些实现全都隐藏在顶部一个极小的暴露接口之后，对吧？而模块 B 则相反，它的接口非常庞大，暴露了大量可供调用的函数，但这些函数各自本身其实并没做什么复杂事情。大家能理解这种对比吗？是的。
Original English
**Speaker**: And the first thing I really like about this is codebase design. So you can actually design your way out of these bad checks. So what does good codebase design look like? I've talked about this before in previous AI engineer talks I've given, which are deep modules. These are modules that hide complex behavior behind simple interfaces. This is a John Ousterhout idea from A Philosophy of Software Design. If we look at these two modules, we've got A which has a large implementation hiding behind a tiny little interface at the top. Okay? And then B is a large interface, lots of functions you can call and those functions individually don't do very much. Does that make sense? Yeah.
**演讲者**：现在来看，如果你拥有像 A 这样的深模块，你写出的对结构敏感的测试就会少得多，因为你已经把绝大部分的实现细节牢牢隐藏在那个简洁的接口之后了。只要测试聚焦在那个接口上，你就能得到质量高得多的测试用例。因此，大家在这里的工作，就是要强制 Agent 使用那个小巧的公共接口，而不是直接伸手探入内部实现去测试那些稀奇古怪的实现细节。
Original English
**Speaker**: Now, if you have a deep module like A here, you're going to have fewer structure sensitive tests because you're hiding more of the implementation behind that interface. If it's just testing at that interface, you're going to get better tests. And so, your job here is to force the agent to use that little interface instead of reaching into the implementation to test these weird implementation details.
### 模块深化 Skill 与架构统一语言
**演讲者**：为此，我专门开发了一套用于代码库设计的 Skill。哪怕你面对的是最诡异的凭借直觉胡乱敲出来的代码库（vibecoded codebase），是你这辈子亲眼见过的最垃圾的代码库，你只要对它运行这个 Skill，它就能把代码变得更好。它的核心作用，基本上就是帮你识别并提供深化模块的各种重构机会，产出的结果看起来大致就像这样。顺便问一下，用过这个 Skill 的人请举下手？我不太确定在场有多少是我们这个圈子的人。好的，确实有人在用。它真的非常非常好用。它本质上会为你生成一份 HTML 文档——我先往旁边站一点让大家看清楚——里面列出了它所识别到的所有潜在优化机会。大家可以看到，这里有修改前和修改后的对比展示，我们通过它减少了重复代码，构建出一个极其优雅、具备高可测性的深模块，之后你就可以直接动手去实现它。
Original English
**Speaker**: So, I've got a skill for this. You can have like the weirdest vibecoded like codebase, the crappiest codebase that you've ever set your eyes on and you can run this skill on it and it will make it better. What this does is essentially gives you opportunities for deepening modules and kind of looks like this. Raise your hands if you've used this skill. By the way, not sure how many of my folks are in this room. Yeah. Okay. It's really freaking nice. Essentially gives you a HTML document. I'll get out of the way here of all of the different um potential opportunities it sees. So we can see here we have a before and we have an after where we're sort of like reducing duplication. We're creating a nice deep testable module and then you can go ahead and implement that.
**演讲者**：与这套体系相配套的，还有一套我整理出来专门用于描述模块的设计语言。因为如果你平时尝试去阅读关于如何组织代码库的资料，你会发现存在 20 种截然不同的流派，而且它们居然全都被冠以“领域驱动设计”（DDD）的名字。你真正需要的，是一套在团队内部沟通时能够达成共识的统一语言。因此我有一个小巧的 codebase design skill，它明确定义了什么是局部性（locality），什么是杠杆率（leverage），以及什么是接缝（seam）——顺便说一句，早在接缝这个概念流行起来之前我就已经在用了。所谓局部性，是指所有相关代码聚集在同一位置的紧密程度，即你如何能够修改某个模块的一小处代码而不至于引发广泛的外溢连锁反应；而杠杆率则是你在拥有深模块时所能获得的巨大收益，因为调用方——也就是实际调用该模块的人——只需调用一个非常简单的函数，就能获取极其庞大的价值。事实证明，这两者不仅在传统代码库中极具价值，对于 AI Agent 来说也同样大有裨益。
Original English
**Speaker**: And attached to this, there's also this kind of language that I've put together for describing modules. Because like if you ever try and read up about how to structure a codebase, you're going to find 20 different approaches and they're all going to be called DDD. And like what you need is a consistent language that you can use in your team to talk about this stuff. And so I have a little codebase design skill that defines what locality is, defines what leverage is, defines what a seam is. I was using seams before they were cool. And what locality means is kind of how uh well located together all of the code is. How can you change like a small change in one module and have it ripple out? And also leverage is what you get when you have a deep module because the caller, the person who's actually calling that module gets a lot of value of calling a simple function. Both of those are very good in codebases and good for agents too, it turns out.
### 实现端上下文过载：不要在 Implement Agent 中堆砌规范
**演讲者**：虽然我刚才讲了这么多听起来高大上的编码规范，但我们究竟该如何确保 Agent 能够切实把它们执行到位呢？你该如何确保 Agent 真正创建出深模块、写出高质量的测试，而不是写出那些垃圾一样的同义反复测试或结构敏感测试？在这个问题上，我认为绝大多数人的做法都是错的。我的第一条核心建议就是：千万不要把编码规范硬塞给你的实现端 Agent（implement agent）。好的，请容我解释一下背后的原因。
Original English
**Speaker**: But I'm sort of describing all of these highfalutin coding standards. But how do we actually make sure the agent does them right? How do you make sure the agent creates deep modules and creates good tests and doesn't write these crap-tological ones or structure sensitive tests? Well, I do think most people get this wrong. And my first piece of advice is don't put coding standards in your implement agent. Okay, let me explain this.
**演讲者**：如果你去想象一下实现端 Agent 的工作状态，它大概就像这样：在单个上下文窗口中，堆满了 Agent 必须能够完成的全部任务。它首先需要进行代码探索，去检索和查找即将修改的代码；接着它必须去实际修改代码，也就是在绿色部分对文件执行更新写入；然后它还需要留出专门的上下文预算来进行调试，也就是运行那些自动化检查，去真正核对并验证代码是否能正常跑通。事实证明，这本身就已经是非常庞大的工作量了。如果你在这个时候还试图把你的编码规范强加给它，它的表现必然会大打折扣。所以说，“实现端是处于超载状态的（implementation is overloaded）”，这就是我希望大家建立的心智模型。那么，我们是否能够找到一种机制，在不导致过度负荷的前提下贯彻这些编码规范呢？
Original English
**Speaker**: If you imagine the implement agent kind of looks like this where this is all of the things the agent needs to be able to do in its single context window. It needs to be able to explore like to look for the code that it's going to change. It then needs to actually change it. So make the updates to the files in the green and then it needs some budget for actually debugging the thing. So for running those uh automated checks, for actually checking and verifying that it works. Now this is quite a lot of work it turns out. And if you try to impose your coding standards on it as well, it's going to perform worse. So implementation is overloaded. That's the mental model I want you to have. And so can we find a way to impose those coding standards in a way that isn't so overloaded?
### 代码审查 Sub-agent 与两阶段编写流程
**演讲者**：我的解决方案就是 code review skill。它接收一份 diff，并读取代码仓库内部一个名为 `coding-standards` 的文件——这个文件完全由你编写、由你自由定制——然后去逐项检查代码是否遵循了这些规范要求。如果我们看一下这里的审查端 Agent（reviewer agent），关键在于它是在一个独立的 Sub-agent 中运行的。这意味着它拥有自己专属的独立上下文窗口，拥有专属的上下文预算。这个审查 Agent 确实也需要做少量的代码探索，因为它虽然接收到了 diff，精确知晓受影响代码的具体位置，但它大概还需要进行一些探索以便掌握更广泛的上下文背景、真正理解业务代码；但是，它完全不需要去写任何实现代码，也根本不需要去执行任何调试动作。
Original English
**Speaker**: Well, this is my effort. This is my code review skill. And it receives a diff and it reads a file inside the repository called coding standards which you can write, you can customize and then it checks if the code follows those standards. And so if we look at the reviewer agent here, it also crucially runs it in a sub agent. So it's got its own context window to kind of handle here. It's got its own budget. This one it needs to do some exploration, right? Because sure it receives the diff so it knows exactly where it's located where the code is but it should probably do a bit of exploration just so it has the wider context understands the code but it doesn't need to do any implementation doesn't need to do any debugging.
**演讲者**：因此，当实现端处于严重超载（overloaded）状态时，审查端实际上是处于低载（underloaded）状态的，它所承担的职责要少得多。这也就意味着，你可以把大量的编码规范一股脑全部交给它，而它完成任务的质量，会远远优于你试图在实现阶段直接强加规范的效果。所以，我的思考方式是这样的——尽管这可能会让人有些不太习惯，对吧？因为我们大家都希望能一步到位、第一次就直接生成优质代码——但我将其视作产出优质代码的两阶段流程（two-part process）：第一步是实现阶段，目标是“让代码跑通”（make it work）；第二步是代码审查阶段，目标是“让代码变好”（make it good），在此阶段强加并落实你的编码规范。对于我们当中那些偏向复古流派的开发者来说，这本质上就是“红-绿-重构”（Red-Green-Refactor）的方法论。我们先消耗一个上下文窗口去把事情做成，完成红绿测试循环；然后再开启另一个上下文窗口专门去重构优化。至少在我脑海中它是这样运作的，而且这种方式对我来说一直非常成功。
Original English
**Speaker**: So while implementation is overloaded review is actually underloaded. So it doesn't have that many jobs to do. This means you can pile in a bunch of coding standards to it and it will do a much better job than if you try to do it with implement. So, I think of this, and this is uncomfortable, right? Because we we all want to be able to just get good code out the first time, but I think of this as the two-part process for writing good code, which is implement, you make it work, and then code review, you actually make it good. You impose your coding standards. And for, you know, the retro developers among us, this is essentially a red green refactor approach. We use one context window to make it okay. Do the red green and then we do another context window to refactor it. That's at least how it works in my head and it's been very successful for me.
### 规范的文件作用域与第三方审查 Bot 的误区
**演讲者**：这也意味着，当你制定编码规范时，切记不要把它们扔进全局作用域。不要直接把它们统统写在 `AGENTS.md` 里面，因为那样做会干扰并冲垮你的实现端 Agent——它可能会读，也可能根本不读。你应该把它们独立存放在 `coding-standards.md` 中，这样一来就只有专门负责代码审查的 Agent 才会去加载它。另外，我这里还有一个想法：今天我和在场的很多同行交流过，很多人都在谈论审查机制，谈论自动化审查是如何普及的、如何解决 PR 吞吐瓶颈的。许多人都说：“噢对啊，我们直接调用第三方服务就行了，比如用 Cursor BugBot，或者用 CodeRabbit 之类的工具。”但在过去，我确实非常努力地尝试过构建一套通用的代码审查 Skill，试图让它能找出所有的 bug、同时完成安全审查等等。然而事实证明……
Original English
**Speaker**: This means that when you have coding standards, you don't put them in global scope. You don't put them in agents.md because then they sort of drown out your implement. It may read them, it may not. You put them in coding-standards.md and that way just the code review agent does it. Now, I've got another idea here, which is I've been talking to lots of people today. Lots of people saying, you know, I've been talking about review and automated review increased, you know, fixing the PR bottleneck. So many folks say, "Oh, yeah, we just use a third party service. We use a cursor bug bot. We use code rabbit or something like that." I think that I've I've really tried to make a generic code review skill in the past that finds all the bugs and does security review and that kind of thing. It turns out
### 自建自动化审查与编写规范
**Matt**: 这确实非常非常困难，因为你要么把它做得太宽泛，导致它不断给出很多误报，而这些误报对于你的实际应用场景根本不相关；要么你把它做得太具体，比如你说：“好吧，把所有关于 TypeScript 的东西都找出来”，结果写 Rust 的团队成员就完全用不了。所以，我的建议是：不要把自动化审查外包出去，而是要自建你自己的自动化审查体系。随着时间的推移，逐步建立起你自己的编码规范，并在整个团队中共享它们。如果你有机会推行编码规范，比如手头刚好有一些放在角落没人去读的文档，那么这里就是放置并应用它们的最佳场所。
Original English
**Matt**: it's really really hard because you either make it too general and it just gives you false positives that aren't actually relevant to your use case or you make it too specific. You say, "Okay, find all the TypeScript stuff and then Rust people can't use it." So, I would say don't outsource automated review. Build your own. Build up your own coding standards over time. Share them across your team. And if you have an opportunity to impose coding standards, if you've got some docs sitting around that no one reads, this is the place to put them in.
### 代码审查 Agent 的核心应是提交修复而非仅留评论
**Matt**: 此外，当你拥有这种自动化审查器时，很多人的本能倾向通常会是这样：“哦对，我的代码审查 Agent 做的事情就是去阅读代码，然后在 PR 上发表评论。”但是在这种情况下，你的代码审查 Agent 实际上是在给人类审查者增加更多的工作负担。人类审查者不得不去逐个阅读所有这些冗长啰嗦的评论，并费力琢磨：好吧，我们到底要不要实现这一项建议？要不要实现那一项建议？代码审查器应当直接提交代码，它应该主动完成修复。因此，它应该把发现的问题真正修改掉，这样当人类审查者过来查看时，面对的就是一份非常漂亮、整洁的产出物。当然，如果它发现了任何存有疑问的地方，它依然可以留下评论，但默认的行为应当是直接进行提交。别再试图一次性就能生成（oneshot）完美的代码了，也别再把“负责实现的 Agent”当成你唯一关注的环节然后心里想着：“好吧，我要逼着它一步到位达到极致。”从这种思维定势中跳出来确实需要花一点点心思去思考，但一旦你领悟到了这一点，你会发现这种体验简直棒极了。
Original English
**Matt**: And also when you have this automated reviewer, a really natural inclination for lots of people is to say, "Oh yeah, my code review agent, what it does is it reads the code and then it comments on the PR." So what your code review agent is doing in that case is it's providing more work for the human reviewer. The human reviewer then has to read all of these verbose comments and figure out, okay, do we implement this? Do we implement that? The reviewer should commit. It should make fixes. So, it should actually change the things that it finds because then when the human comes around, you're reviewing a really nice artifact. If it finds anything that it has any questions over, then of course it can comment, but the default should be commits. Stop trying to oneshot good code. Stop trying to make the implementer agent the only thing that you do and go, "Okay, I'm going to force it to be amazing." It takes a little bit of, you know, thinking your way out of there, but once you realize it, it is fabulous.
### 构建对人类审查友好的 PR 与 PR Skill
**Matt**: 那么好吧，经历了所有这些流程之后，我们已经运行了自己的自动化检查，接着又运行了自动化审查来确保自动化检查没有在说谎撒谎。接下来，我们在人类审查的维度上，又该如何最大化这个 PR 的质量呢？我们怎样才能让它以最佳的状态运转起来？答案是：我们需要一个对人类极度友好的 PR。这也是即将引入到代码仓库中的一个全新 Skill，目前正在开发进行中，不过我很快就会正式发布它，这就是“PR skill”。这个 Skill 是我已经琢磨、权衡了很长很长时间的一个东西，我之前一直没有完全弄明白现阶段业界最顶尖的做法（state-of-the-art）到底应该是什么样的。后来我意识到，打造一个 PR skill 最好的方式，就是去把每个人所拥有的最棒的想法统统借鉴过来，这效果真的非常棒。
Original English
**Matt**: So, okay, with all of that process, we've run our automated checks. We've now run our automated review to make sure the automated checks aren't lying. How do we then maximize the PR's quality in terms of human review? How do we get it like working the best it can? So, we need a human friendly PR. And this is a new skill uh coming into the repo which is currently in progress, but I'll be releasing it soon, which is the PR skill. This is one I've been mulling over for a long, long time. I haven't quite figured out what the state-of-the-art is. And I realized the best way to make a PR skill is just to steal all the best ideas that everyone's got. And it's very nice.
### 单向门、双向门与合并风险评估
**Matt**: 那么，一个优秀的 PR 描述正文（PR body）究竟长什么样呢？如果你想凭自己的力量重新创建这个 Skill，你该怎么做？首先，第一条原则是：某些审查要比其他审查更为重要，并不是每一项审查都是不可或缺的，对吧？一旦你理解了这一点，你就会明白：好吧，这意味着我可以把我所有的精力集中在那些真正极其重要的审查上。但我们应该如何对其进行分类呢？你必须借助 AWS 的经典术语来思考这个 PR：它究竟是一扇“单向门”（one-way door），还是一扇“双向门”（two-way door）？你遇到的大多数 PR 基本上都是双向门。你可以先把 PR 合并进去，之后随时都可以把它回滚撤销回来。这是作为一名软件工程师相比于土木工程师而言最美妙的优势所在，对吧？大多数情况下，当你是一名土木工程师时，那都是一扇单向门，一旦你搞砸了某些东西，那座大桥就会彻底坍塌。但如果你拥有一扇双向门，这意味着你能够轻而易举地恢复、回滚这一变更。
Original English
**Matt**: Now, what does a good PR body look like? How would you recreate this skill on your own? Well, the first principle is some reviews are more important than others. Not every review is essential, right? And once you understand this, you realize, okay, that means I can focus my energy on the really important reviews. But how do we categorize that? Well, you got to think about the PR as using this AWS terminology, which is is it a one-way door or is it a two-way door? Now, most PRs that you have will be two-way doors. You can merge the PR and then always pull it back later. It's the glorious benefit of being a software engineer as opposed to a civil engineer. Right? Mostly when you're a civil engineer, it's a it's a one-way door, right? If you get something wrong, that bridge is going down. But if you have a two-way door, it means that you can easily revert the change.
**Matt**: 不过实际情况可能比你想象的还要更加微妙复杂一些。很有可能只是一个非常细微简单的改动，却意外地给 60,000 个人群发了电子邮件，遇到这种情形，它就是一扇单向门，你必须非常非常仔细谨慎地审查它；或者涉及到了高昂成本的数据迁移或数据丢失风险，那同样是一扇单向门，你必须竭尽全力、彻彻底底地把那个 PR 审查清楚。但与此紧密相连的另一点是，我们还需要明确理解该 PR 的“爆炸半径”（blast radius）：到底可能会出什么差错？而如果真的出了差错，后果会有多严重？这意味最终会在我所有 PR 的最底部生成一个漂亮小巧的概要信息，也就是“合并危险等级”（merge danger）。我一眼就能看出：这是一个双向门，它的爆炸半径局限在局部。这样我就能立刻明白：太棒了，我不需要对这个 PR 投入过多的精力，我只需要稍微粗略浏览审查一下就行了。这一点至关重要。
Original English
**Matt**: Now, that might be more um bit more nuanced than you might expect. It might be that a very simple change accidentally blasts out an email to 60,000 people or something in which case that is a one-way door. You want to review that very very carefully. Involves expensive migrations or data loss. That is a one-way door. Review the hell out of that PR. But also tied on to that we need to understand the blast radius of this PR. What can go wrong? And if things do go wrong, how bad is it? And this means I end up with a nice little sort of summary right at the bottom of all of my PRs, which is the merge danger. I can see this one is a two-way door. It blast radius is localized. And so I can see, fantastic. I don't need to pay that much attention to this. I'm just going to sort of review it a little. That's really important.
### 用伪代码与图表直观展示变更意图
**Matt**: 接下来，我们必须理解这个 PR 到底在干什么，对吧？关于这一点，我曾经尝试过很多种不同的搞清方式，而我总结出的最佳方案就是使用伪代码。在这里，我要向 Dex Horvath（Dex Holley）的人类层技能库（human layer skills repo）中的 `show me` skill 表达巨大的感激之情。这是一个极其惊艳出色的 Skill，它基本上抛弃了绝大部分冗长的文字，转而通过图片和图表来向你呈现事物。这极大地简化了理解成本，让你能够迅速看懂实际上到底改变了什么，以及为什么要做出这些改变。所以你会得到那种标准配置的 Mermaid 图表和 UML 图，用来展示“这是发生的一系列时序事件”；你也可以得到像这种非常简洁漂亮的图表，举个例子：我们看一眼就能明白，我们正在命令行 CLI 环境下工作，可以看到这里添加了一个新命令，并且我们在顶部有两个新增加的小选项标志（flags）。仅仅是这样的小型摘要，就真的能带来极其巨大的改变，无论怎么强调它都不为过。因此，你所做的就是让理解“为什么修改”的过程变得尽可能迅速高效。
Original English
**Matt**: Next, we need to understand what the PR is even doing, right? And I've tried lots of different ways of figuring this out, and the best thing I've come up with is using pseudo code. Now, a huge um point of gratitude here to the show me skill from the human layer skills repo by Dex Holley. This is a phenomenal skill that just essentially dispenses with most text and shows things to you in images and diagrams instead. This makes it a lot easier to grasp actually what's changing and why it's changing. So you get the kind of standard sort of set of like mermaid diagrams and UML for you know this is a sort of sequence of things that happened. You also just get these lovely simple ones like this for instance. We can look at this and go okay we're working in a CLI. We can see a new command has been added and we've got two new things little flags up the top here. Just little summaries like this. It really does make a massive difference. It's hard to overstate. So you're trying to like make understanding the why as fast as possible.
### 审查产生代码的系统本身与复盘机制
**Matt**: 接下来我认为第三条原则是：每当你在进行人工审查时，都应该时刻牢记的一件事。因为我们现在的研发流程已经变得如此精简高效，而且我们都在围绕着相同的技能文件（skill files）、相同的引导约束文件（steering files）进行协同，我们共同在为我们的各个 Agent 构建可供它们运行协作的环境。因此，你应当把“生产代码的整个流程”视作与“代码本身”同等重要的存在。换句话说，当你进行人类审查时，你审查的绝不仅仅是具体的代码，你更是在审查创造出这些代码的整个系统。这里的核心理论在于：你永远不应该写两次相同的审查评论，对吧？你绝不希望抓到同一个 Agent 在两个不同的 PR 中犯下完全相同的错误。那么，你通过什么机制才能确保你的人工审查真正发挥长远作用呢？
Original English
**Matt**: And I think a third principle here is something you should be thinking about whenever you do human review because because we're not doing like um because our processes now are so sort of streamlined and all we're all collaborating around these same skill files, these same steering files. We're all building an environment for our agents to operate in together. You should think of the process that produces your code as just as important as the code itself. In other words, when you do a human review, you're not just reviewing the code, you're reviewing the system that creates it. And the theory here is that you never want to write the same comment twice, right? You never want to catch the agent doing the same thing over two PRs.
### Retro Skill：沉淀编码规范与实现复利改进
**Matt**: 那么，究竟用什么机制才能让你的人工审查产生长效价值呢？这就是一个全新的技能，名为“retro”。retro 即 retrospective（复盘回顾）。你基本上就是拿一个已经完成的 Session 会话来进行分析——它可以是一个单独的 Agent 会话，也可以是一个 PR 加上其对应的会话；或者你也可以让它统一去审视：“好吧，把我们在过去一周内完成的所有 PR 以及所有的代码审查统统拉取进来，让我们针对它们统一做一次复盘回顾”。随后它就会自动为你推荐相应的自动化检查规则以及编码标准，从而让下一次的开发变得更加完善。这就是一种复利累积效应（compounding effect）：在本质上，通过进行人类审查，你正在不断提升下一次人类审查的质量标准，并在下一次为你自己节省下更多的工作负担。
Original English
**Matt**: And so what's the mechanism by which you can make your human review matter? Well, this is a new skill. This is called retro. Retro for retrospective. You essentially take a session that you've done. It can either be like a single um agent session or it can be a PR plus the session or you can just get it to look at okay look at all of the PRs that we've done over the last week all of the reviews pull them all in let's do a retrospective on them and it will suggest automated checks and coding standards to make the next one better. So this is the compounding effect where you essentially by doing human review you're making the quality of the next human review higher and you're sort of saving less work from yourself next time.
**Matt**: 而且 retro 是一个极其聪明的 Skill，它加入了大量实用的功能。显然，它会提供自动化检查的建议，会向 `coding standards.mmd` 提供规范更新的建议。除此之外，它还会做其他非常精明的分析：比如它会审视导航指针（navigation pointers）——Agent 寻找信息的难易程度如何？我们能否在 `agents.mmd` 内部增加一个导航指针来协助它下一次更快找到信息？它还会审视工具使用经济性（tool economy）——我们在会话中调用的各类工具是否存在哪些可以进一步优化、使其在 Token 消耗上更加高效的改进空间？事实上，它能捕捉发现的事情数量之多令人惊叹，因为那些问题从外部来看往往非常难以调试定位。它同时还会检查膨胀与冗余（bloat）——是否存在过度臃肿的引导文件（steering files）？是否存在过于庞杂臃肿的技能导致了这些不良结果？我们能否把它们整理得更加井井有条？
Original English
**Matt**: And retro is a really smart skill. It adds a bunch of stuff. So obviously it suggests automated checks. It suggests updates to coding standards.mmd. It does other smart stuff too. So it looks at navigation pointers. How easily did the agent find its information? Can we provide a pointer inside agents.mmd to help it out next time? It looks at tool economy. Are there different tools that we're using in the session that can, you know, could be made more token efficient? It's amazing how many things this catches actually because those are often really hard to debug from the outside. It just looks at bloat as well. So, are there bloated steering files? Are there bloated skills that contribute to these bad results? Can we make them more organized?
### 总结与演讲致谢
**Matt**: 所以，我们的终极目标就是让人类审查的速度变得更快。我们实现这一目标的方式就是层层叠加自动化检查，层层叠加自动化审查；并且我们将人类审查打造得尽可能无痛、尽可能简单，甚至在需要时尽可能可选化。你真的不需要对每一个双向门变动都去进行全盘审查，但每一个单向门变动你都必须严格审查。以上就是我的技能集合：`aihero.dev/skills`。我会在本周发布 1.3 版本。能和大家相聚在这里交流真是太美妙了，这是一场非常棒的技术大会。演讲结束后我会在外面的大厅，如果有谁想要聊一聊随时欢迎。非常感谢大家的倾听和邀请，谢谢巴黎！
Original English
**Matt**: So that's the goal is to make human review faster. We do that by layering up automated checks. We're layering up automated review. And we make the human review as painless, as simple, and as kind of optional as we need it to. You really don't need to review every single two-way door. Every single one-way door you do. And so these are my skills. AI her.dev/skills. I'm going to be shipping version 1.3 this week. It has been glorious hanging out with you. It's been a really nice conference. I'm going to be outside in the lobby if anyone wants to have a chat. Uh thank you so much for having me. Thank you, Paris.
**Audience**: 非常感谢！
Original English
**Audience**: Thank you so much.
**Host**: 好的，让我们再次用热烈的掌声感谢 Matt！我太喜欢那个 retro skill 了，我想我很快很快就会在实际中用上它。好的，接下来有请来自 DeepMind 的嘉宾，我们将迎来 Martin，他将向我们分享关于 Gemma 的内容。他不仅会向我们讲解一些架构底层的基本原理，还会介绍到底是什么让 Gemma 如此飞快且独树一帜。那么，让我们用热烈的掌声欢迎 Martin！
Original English
**Host**: All right, let's give it up for Matt once again. I love that retro skill. I think I'm going to use it very very soon. All right. Okay, so up next is somebody from Deep Mind U. So we have Martin who's going to talk to us about Gemma. He's going to talk to us about some architectural fundamentals, but also what makes uh Gemma so fast and unique. So please let's give it up for M for Martin.
### 开源模型的端侧效率挑战
**演讲者**：没有，我什么都没看到。太好了，现在正常了。今天我想和大家聊一聊我认为在发布开源模型过程中非常核心的一个要素，那就是效率。因为一旦你把模型开源发布出来，让大家能够真正上手去使用，从那一刻起，它们就会受到各种各样的资源限制。之所以受限，是因为你知道它们需要在大家的移动设备或笔记本电脑上运行，我大致可以推测，运行这些模型的并不是那些大规模的数据中心。因此，效率就成了一个非常关键的课题。
Original English
**Speaker**: No, I'm not seeing anything. Yay, it works. Today I get to talk about something that I think is a very central component of releasing open models and that's about efficiency. Because the moment you release a model out into the wild and people can actually use them, uh that's the moment that they become constrained. Constrained because you know they're being run on your mobile devices on your laptops and I can kind of assume that there are not data centers that are running these models. So efficiency becomes a very big part of it.
**演讲者**：我希望从几个不同的维度来探讨效率：知识储备、模型尺寸、运行速度、架构复杂度，以及架构本身的演进。我想深入剖析其中的几个关键组成部分，看看我们能从中汲取哪些经验。
Original English
**Speaker**: And I would like to focus on a couple of perspectives on efficiency. knowledge, size, speed, the complexity of the architecture, but also the architecture in itself. And I want to dive deep into some of these components and see what we can learn from it.
### 逐层嵌入与知识解耦（Per-Layer Embeddings）
**演讲者**：其中一个部分是逐层嵌入（per-layer embeddings）。这在 Gemma 系列的小尺寸模型中是一项非常重大的创新。比如有 20 亿（2B）和 40 亿（4B）参数量的模型，它们都在重度使用逐层嵌入技术。现在这基本上也解释了 E2B 和 E4B 中的字母“E”到底代表着什么。在混合专家（MoE）模型中，我们熟知字母“A”代表活跃参数（active）；那么这里的“E”又代表什么呢？让我来给大家展示一下。
Original English
**Speaker**: One part is per layer embeddings. It's a very big thing in the smaller models of Gemma 4. There's these two billion and four billion parameter models and they use per layer embeddings heavily. Now they essentially explain what that E actually means in E2B and E4B. We know the A right in mix of experts models they stand for active. But what does the E then stand for? Well, let me show you.
**演讲者**：这就是 Gemma 通常采用的整体架构，对吧？它包含了局部注意力（local attention）、全局注意力（global attention）、分组查询注意力（grouped-query attention），在此处和彼处还包含了一些归一化层。然后在最底端，对于这些小模型而言，突然引入了逐层嵌入层。顾名思义，这些不仅仅是针对每个 token 的嵌入，更是按网络层细分的逐层嵌入。它们在形式上非常类似大家所熟知的传统 token 嵌入，但不同之处在于，每一层都会被累加引入。这意味着，像“high”这样的 token，在第 1 层拥有的嵌入表示，会与它在第 4 层或第 10 层的嵌入表示截然不同。
Original English
**Speaker**: And this is the architecture that Gemma 4 generally uses, right? It has local tension, has global tension, group query tension, there's some normalizations here and there. And then right at the end with the smaller models there are suddenly per layer embeddings there. And as the name implies these are embeddings per tokens but even per layer. So they're much like the token embeddings that you know we all know so well but this time they're being added with every layer. That means that the token high has a very different embedding on layer 1 than it has for instance on layer four or layer 10.
**演讲者**：这是一种在无需将所有参数常驻内存的前提下，向模型注入大量知识的巧妙方法。在推理处理过程中，模型实际上可以动态权衡如何利用这些 token 表示。对于给定的上下文，模型可能会判定：“好的，比起嵌入的那个维度特征，我更想聚焦在这一维特征上。”采用这种机制最绝妙的一点在于，它本质上是一个关于 token 信息的查找表（lookup table）。当你拥有一个查找表时，你根本不需要把它塞进显存（VRAM），也不需要常驻内存（RAM），你完全可以把它存放在任何现有的存储介质上。这里发生的其实是一种类似检索增强（RAG-like）的行为，因为在推理期间，我们只需要按需加载实际使用到的 token 对应的数据。所以这就解释了为什么“E”实际上代表有效参数（effective）——因为虽然逐层嵌入查找表可能包含数十亿参数，但在推理的单次前向中它们并不会全量参与计算。在某种程度上，我们可以将一部分知识储备与模型的实际推理能力分离开来，而这种“知识与能力解耦”的理念，在当前的开源模型领域正变得越来越明确和普及。
Original English
**Speaker**: It's a way to add knowledge to the model without having to, you know, store them in memory. During this processing, the model can actually weigh how these tokens uh are being used. So for a given context, the model might say, "Okay, I want to focus a little bit more on this aspect of the embedding rather than that aspect." And now the great thing about using something like this is that it's a lookup table of information about the tokens. And when you have a lookup table, you don't need to store it in VRAM. You don't need to store it in RAM. You can just store it on whatever storage you have. It's a rackl like behavior that's happening here because during inference, we only need to load in the tokens that are actually needed. And so that explains that the E actually stands for effective because that per layer embedding lookup tables, billions of parameters, right? But they're not being used during inference. we can separate in a way part of the knowledge from actual model capabilities and this this this idea of separating them is becoming more and more pronounced in the field of open models.
### 模型尺寸优化与 MobileQuant 混合量化方案
**演讲者**：既然我们讨论了知识，也知道了如何高效地处理知识，那么模型尺寸又该如何兼顾呢？例如，你打造了一个非常轻量级的模型，但仍然希望它具备强大的能力。你可以采取的做法是量化感知训练（quantization-aware training），在尽可能保留模型能力的同时降低数值精度。这是一种非常优秀的解决思路。但在 MobileQuant 方案中，我们实际上可以更进一步。MobileQuant 是专门针对这两个小尺寸模型研发的定制化量化模式；因为当你面对小模型并且要进一步将其压缩到更极端的尺寸时，必须确保不能损失过多的模型性能。
Original English
**Speaker**: So we have knowledge you can do knowledge efficiently. What about size for instance uh you can create a very small model but you still want to have a lot of capabilities. So what you can do is you can do quantization aware training. you reduce the precision to something smaller while trying to maintain the capabilities as much as you possibly can. And that's a very nice way to approach this. But we can actually take it a step further with mobile quants. And that's a very specific schema that was developed for these two smaller models because when you have smaller models and you're going to quantize them to even a smaller size, you have to make sure that you don't lose too much performance.
**演讲者**：这是一个高度定制化的方案，结构大致是这样的：首先看 token 嵌入层，你其实可以直接把它一路降到 2-bit 精度，同时依然保留模型的绝大部分能力——至于具体原因我稍后会详细说明。对于全局注意力，采用的是 4-bit 精度；4-bit 是我们普遍使用的方案，它在模型大小和综合能力之间取得了很好的平衡。但对于局部注意力——也就是放大聚焦于我们需要深入探究的信息时——保持略高一点的精度会非常有帮助，特别是在 KV cache 方面。前馈网络（FFN）采用 4-bit；而对于我们刚才谈到的逐层嵌入，则采用 2-bit 精度，因为它们某种程度上也弥补了基础 token 嵌入精度降低所带来的影响。当你把所有这些组合在一起时，就为特定模型量身定制了一套精妙的方案。由此可见，确保高效性往往意味着需要针对模型进行大量的定制化设计，尤其是在 20 亿和 40 亿参数这类尺寸规模下。
Original English
**Speaker**: So a very specific schema and looks a little bit like this. So we have the token embedding layer right you can actually reduce that all the way to two bits and retain a lot of the capabilities of the model still uh and now the reason I will come into a little bit later for global attention four bits 4bit is what we generally use right it's a it's a nice balance between the size of the model and capabilities but for local attention which you know zooms in on the thing that we want to know more about it helps to have a little bit of a higher precision Especially with the KV cache feed forward network is four bits but then the per layer embeddings that we just talked about those are two bits because they kind of compensate for the lower position of the token embeddings as well. And so when you combine all of this you have a nice schema for a specific model. And what you see is that making sure that it's efficient also means doing a lot of customizations to these models especially at sizes like two billion four billion parameters.
### 推测解码与多 Token 预测加速
**演讲者**：有了上述技术，我们在知识管理和尺寸压缩上都实现了高效。那么在推理速度方面又当如何呢？在速度优化上，我们采用了一种如今越来越受欢迎的技术——基于多 Token 预测的推测解码（speculative decoding with multi-token prediction）。其工作原理是利用一个较小的模型来预先提议候选 token，供你实际想要运行的大模型使用；这样一来，大模型只需要对看到的这些候选 token 进行验证，而无需一次性自主生成处理每一个 token。
Original English
**Speaker**: And with all of that you have knowledge, you have efficiency in size. What about speeds? Well, with speeds we use something that's becoming more and more popular fortunately these days speculative decoding with multi-token prediction. And it works by using a rather small model to kind of suggest tokens that the bigger model, the model you want to actually run, can use because that allows the bigger model to then only have to validate all of the tokens that it sees without having to process all of those tokens themselves at once.
**演讲者**：因此，我们有一个目标模型（target model），也就是那个架构更庞大的大模型；同时还有一个草稿模型（draft model），也就是轻量的小模型。在 Gemma 的设计中，我们的目标是实现 KV cache 的跨模型共享。为什么呢？因为这极大地简化了草稿模型或候选模型的处理负担。其工作流程是：目标大模型接收查询并进行单轮前向传播处理，生成其自身的 KV cache，然后直接共享给轻量级草稿模型；草稿模型可以直接利用该输出继续向下处理并快速生成 token。它接收输入后，会连续预测生成 1 个、2 个、3 个甚至多达 8 个候选 token，供大模型随后进行一次性验证。
Original English
**Speaker**: So we have a target model, right? It's a big one. It can be whatever architecture that you have and a draft model, a smaller one. And with Gemma 4, what we aim to do is do some KVK cache sharing. Why? Because that simplifies the process for the smaller model, the candidate model or the draft model much, much more. Because what happens is that the target model takes in and query, does a single round of processing, one single pass, generates its KV cache, and then shares it with the smaller model. And the smaller model can then use the output of that and continue on processing and generating tokens. It will take the input, generate one, two, three, perhaps eight tokens for the larger model then to validate.
**演讲者**：这样一来，大模型只需一次性审视所有这些生成的候选 token 并做出裁定：“好的，这几个 token 我认可，但后面这几个不太合适。”因此，例如大模型可能只接受前两个 token。但这仍然只是一次单轮前向推理，对吧？由于它已经针对整个输入序列完成了前向计算，它只需要顺势再补上一个新 token，从而以几乎等同于生成单个 token 的计算代价，一次性产出了 3 个 token，这大幅提升了生成速度。当然，天下没有免费的午餐，这种方法同样有其权衡与利弊。你终究是额外引入了一个辅助模型，尽管它很小，但开销依然存在。此外，这种机制带来的加速比在不同使用场景下差异巨大：有些 token 极其容易预测，尤其是在代码这类结构化任务中；但在创意类任务中，草稿小模型很难摸准该提议什么样的 token，从而导致更多的候选 token 被大模型拒绝。不过幸运的是，如今现实中代码和各类结构化任务越来越普遍，在这些场景下，多 Token 预测能够展现出异乎寻常的卓越加速效果。
Original English
**Speaker**: And so the larger model only has to look at all of these tokens at once and decide, okay, I like these tokens, but I don't like so much these tokens. So I'm only going to accept these first two, for instance. But it's still a single pass, right? So all it needs to do is then add another token because it can it did the computation for the entire sequence anyway and then it generates three tokens almost at the cost of a single one and that speeds things up tremendously. Now obviously nothing is a free lunch right? So there are pros and cons to a method like this. You add a model it's a very small model but you add it anyway. It also differs greatly on the use case, the speed up that you get from something like this. The thing is some tokens are super easy to predict, especially in structured tasks like code for instance. But in creative tasks, it's very hard for the smaller model to really figure out okay, what kind of tokens am I going to suggest? So more of these tokens will then be rejected. Well fortunately there are more and more coding task more and more structured task that we see out there. So for those use cases multi-token prediction works exceptionally well
### 架构复杂度考量与无编码器（Encoder-Free）模型
**演讲者**：我们在知识、尺寸、速度上都实现了高效，那么复杂度方面呢？哦，我看到这里有个东西，让我把那个服务窗口关掉。接下来就是架构复杂度，这也是一个必须正视的问题，对吧？如果你构建的是一个闭源模型，对于最终用户而言内部架构有多复杂其实并不太重要；但当你把模型开源公开发布、供大家实际下载使用时，我们必须确保它不会太晦涩，易于大家理解、部署和运行。
Original English
**Speaker**: and then you have efficiency on let's see knowledge size speed uh what about complexity? Oh did I see this one here? Let me remove that server. And then we have complexity because that's also a thing, right? If you have a model that's not open, it really doesn't matter that much how complex it is for the end user. But when you release a model out into the wild for folks to use, uh, let's make sure it's not too difficult to actually, you know, understand, use, uh, and process.
**演讲者**：正因如此，社区中出现了一种名为“无编码器模型（encoder-free model）”的发布范式。无编码器模型审视了以往用来处理多模态输入的各种编码器，并思考：维持这些模块开销太大了，如果我们彻底把它们拿掉会怎样？在 Gemma 系列的许多模型中，通常会配备一个音频分词器（audio tokenizer）、一个音频编码器（audio encoder），图像模态也是类似的配置。这种设计固然很好，效果也非常稳定，为不同模态实体配置独立的编码器在业内早已是标准做法；但它们依然占据着不小的体积，拖慢了首 Token 延迟（TTFT），而且在对模型进行微调时也增加了额外的考量成本。而无编码器模型则尝试将这些独立编码器彻底移除，并放入大量……
Original English
**Speaker**: And so, as a consequence of that, there was this model that was released called an encoderree model. And the encoder free model kind of looked at the encoders for you know processing these multimodal inputs and said that's a lot of work. What if we remove them? For many of these models in the Gemma 4 line you have an audio tokenizer an audio encoder and the same for images. And that's great. That works really well. It's a staple in the field right to have these encoders for different multimodal entities but they still take up a reasonable size. It reduces time to first token and it's something else to consider when for example fine-tuning your model with the encoder free model. They try to just remove them entirely and put a lot
### 移除独立编码器：端到端多模态处理与位置嵌入
**Speaker**: ……将处理这些多模态输入的负担直接转移到模型本身，而不是依赖这些独立的编码器。对于音频来说，它的运作方式大致是这样的：你有一个输入，即一串振幅值序列；接下来唯一发生的事情就是，他们直接把它切成碎片，变成一个……嗯，音频 token 序列，然后直接将其投影到模型自身的维度空间中，仅此而已。因此，你所拥有的参数基本就只存在于线性投影层中，除此之外别无他物。随后，之前在音频编码器中发生的所有上下文处理工作，现在都直接由大语言模型（LLM）自身来承担了。这样一来，你实际上就是把所有这些参数从专用编码器转移到了 LLM 当中。
Original English
**Speaker**: ...of the burden of processing these multimodal inputs onto the model rather than these encoders. For audio, it works a little bit like this. You have an input, a sequence of amplitude values and the only thing that happens is they cut it up into pieces, make it a sequence of, well, audio tokens and then project it onto the dimensionality of the model itself and that's it. So the only parameters that you have are kind of in the linear projection, but that's about it. And then all of the contextual stuff that was happening in the audio encoder before is now being handled by the LLM itself. So you're kind of shifting all of these parameters from the encoder to the LLM.
**Speaker**: 同样的处理方式也可以应用在图像编码器上。不过，图像面临的一个独特问题在于三维空间信息。我们不能简单地把图像切成小块碎片，拼成一个序列，然后盲目指望模型能把它处理好——因为模型根本处理不好，这一点我完全可以向大家保证。所以为了解决这个问题，我们需要引入位置信息。因此，我们并没有直接将图像 Patch 投影到模型中，而是获取这些 Patch 并为它们赋予额外的位置信息。这样一来，模型就知道……比如说第 4 个 Patch（那个小图像 token）实际上代表着什么含义。因为你可以想象，如果你面对的是一张非常宽的图片，位置 4 所代表的几何含义，与面对一张非常高的图片时的位置 4 是截然不同的。于是，实际做法就是使用投影后的位置嵌入（positional embeddings），最终你得到的依然是一个被彻底移除的编码器。
Original English
**Speaker**: The same can be done with the image encoder. The one problem though is with images is 3D information. We can't just cut it up into pieces, make a sequence out of it, and just hope the model processes it well because it doesn't. I can promise you that. So what we need for that are positional information. So instead of doing a direct projection from the patches onto the model, we take those patches and provide it with additional positional information. So it knows that, well, let's say patch four, that small image token, has actually some meaning to it because you can imagine if you have a very wide image, position four means something very differently than if you have a very tall image. And so what happens is you just use the positional embeddings projected and what you have again is an encoder that's entirely removed.
**Speaker**: 这种架构设计最棒的地方在于，它大幅缩短了首字延迟（Time to First Token, TTFT），这带来了极为显著的帮助。你把所有的表征与理解能力都集中交给了核心模型，同时削减了很大一部分参数量。尤其是对于较小的模型尺寸而言——比如 3 亿参数（300M）或 5 亿参数（500M），在如今人们动辄讨论千亿级参数模型的背景下，这些数字听起来似乎微不足道，但若放在 20 亿（2B）或 120 亿（12B）参数的体量下，省去独立编码器带来的差异是极其巨大的，特别是当你能把这些参数预算全部投入并释放给主模型自身的能力时。
Original English
**Speaker**: What's so nice about something like this is that it reduces time to first token that very much helps. You put all of the capabilities onto the model and you reduce a big portion of the parameters because especially at smaller sizes, something like 300 million parameters or 500 million parameters, it doesn't sound large when we're talking about hundreds of billions of parameters for some of these models, but at two billion sizes or 12 billion, this makes a big, big difference especially if you can use it for the capabilities of the [model].
### 从自回归到扩散机制：从内存受限走向算力受限
**Speaker**: 至此，我们探讨了知识、尺寸、速度和复杂度——这些全都是聚焦于“效率”的不同切入维度。最后，还有一个我非常想深入聚焦、多解释一下其内涵与运行机制的核心方向，那就是架构革新。常规的模型、常规的 LLM 本质上全都是内存受限型（Memory-bound）的，这种特性既有优势也有劣势，它们表现优异，并在众多不同领域和场景中得到了广泛应用。但如果我们把它们转变为算力受限型（Compute-bound）呢？如果我们把整个叙事逻辑彻底反转 180 度，尝试从一个完全颠覆的全新视角来切入呢？这样一来，你得到的将不再是自回归机制（Auto-regression），而是一种彻底不同的范式：扩散机制（Diffusion）。我们是否可以在大语言模型中引入扩散机制，同时依然保持其实用价值，让广大开发者能够将其运用在截然不同的独特场景中呢？
Original English
**Speaker**: And so we have knowledge, we have size, we have speed, we have complexity, all different ways to focus on efficiency. And there was one last thing that I really wanted to focus on and, you know, explain a little bit more about what it means and how it works and that's architecture. Regular models, regular LLMs are all memory bound and those have advantages and disadvantages and they're great. They're used for many different purposes and in many different fields. What if we make them compute-bound instead? What if we flip the narrative 180 degrees and try to approach it from a very different perspective? And so what you get is not auto-regression, it's something entirely different: Diffusion. Can we use diffusion for LLMs and still have it being meaningful? Have it used in a way that, you know, a lot of people can still use it but for very different use cases?
**Speaker**: 现在让我来为大家详细拆解一下扩散机制在大语言模型中到底意味着什么，尤其是因为我们马上会谈到一个我相信很多人在过去几周都看到过的主题。首先，依然是从输入开始：我们有一个正在被处理的 Query 或输入提示。通常情况下，这会由单一模型来完成，而现在基本上也是由同一个模型在处理。不过在扩散架构中，我们引入了一个被称为“编码器”（encoder）的组件。在 Diffusion Gemma 中，这个所谓的编码器实际上依然是一个带有因果注意力机制（Causal Attention）的解码器，它本质上就是一个经过微调的 Gemma 4 模型，并没有采用什么极其奇怪诡异的架构，它就是大家之前见过的标准模型之一。之所以把它叫做编码器，纯粹是因为它的职责是专门用来处理 Prompt 的——它负责生成 KV 缓存（KV Cache），也就是我们后续构建推理并逐步稳定生成输出所必需的上下文状态。
Original English
**Speaker**: Now let me go through a little bit about what diffusion means in large language models, especially because we will come to a subject that I think a lot of folks will have seen in the last couple of weeks. You start with an input, right? We have a query, an input that's being processed. And normally one model would do that and now kind of the same model does it. But what happens in diffusion, we have something called an encoder. And the encoder in Diffusion Gemma at least is still just a decoder with causal attention. It's a fine-tuned Gemma 4 model. No crazy different architecture or what have you. It's, you know, one of the models you saw before. The reason why it's called an encoder though because it's meant for processing the prompt and so it generates a KV cache, the context that we need to then build upon and slowly and more surely do inference.
### 双向注意力与去噪画布：多步迭代并行生成
**Speaker**: 我们以这个 KV 缓存为基础；但在扩散模型中，为了执行真正的生成处理并实际产出 token，你并不是从零开始逐字生成的。你不会采用“先吐出第一个 token、再吐出第二个、接着第三个”那种自回归模式。相反，你一开始拿到的是一个充满噪声的画布（Noisy Canvas），其大小通常设定在 256 个 token 左右。接下来，“去噪器”（Denoiser）的任务就是对这个画布进行实际处理，并通过多次迭代，把模型具有高度置信度的确切 token 逐步填入其中。值得注意的是，负责去噪的依然是完全相同的模型，它共享完全相同的权重绑定的 Gemma 4 26B 模型。不过这里存在一个关键差异：如果面对的是一个宏大的全局画布，并且你希望在开头和结尾处同时更新 token，你就需要一种完全不同的注意力机制。因此，当模型切换至去噪器模式时，它转而采用双向注意力机制（Bidirectional Attention），以便能够同时纵览前后双向的上下文。
Original English
**Speaker**: So we start with the KV cache, but to do the actual processing, to actually generate tokens in diffusion you don't start from scratch. You don't just, you know, here's one token and here's the second and here's the third. Now you start with a noisy canvas. It's typically sized around 256 tokens and then the job of the denoiser will be to actually process that and iteratively fill it up with tokens that it's very confident about. It is the exact same model though. It uses the exact same tied weights Gemma 4 26B model. There's one difference though. If you have a large canvas and you want to update the tokens simultaneously at the beginning and at the end, you kind of need a different type of attention. So when it's in the denoiser mode, it uses bidirectional attention instead, so it can look both directions.
**Speaker**: 在此之后，去噪器会接收来自编码器的 KV 缓存——再次强调，编码器的全部目的就是预处理初始状态，为去噪器开展工作做好一切准备。随后，去噪器便可以预测或建议哪些 token 应当被填入哪些具体的位置，因为我们手头已经有了这个初始的 token 画布，它所需做的就是在对应的适当位置对其进行更新替换。这样模型就得到了一个预测画布（Predicted Canvas），但它不可能一次性完美搞定所有内容，因此需要经历若干个步骤来迭代完成。很显然，在生成的候选词中，有些 token 是模型极度确信的，而另一些则没有那么高的置信度。于是，你接下来便会得到一个所谓的“接收画布”（Accepted Canvas）：部分置信度达标的 token 被正式采纳接收，而未被接收的 token 则会被重新施加噪声（Renoised）。因为该模型的整个核心宗旨就是执行去噪——消除原始画布中的随机噪声，并挖掘出真正应当存在于该位置的正确 token。
Original English
**Speaker**: What happens after that is it gets the KV cache of the encoder because again it was meant to process the state and get it ready for the denoiser to do its work. And then the denoiser can, well, predict or suggest what tokens should go at which specific places because we have this initial canvas of tokens and all it needs to do is update them at the appropriate places. So there's this predicted canvas that it then has, but you know it doesn't do stuff perfectly at once. So it needs a couple of steps to do that. There are some tokens it's very confident about obviously and some not so much. So what you then get is in so-called accepted canvas: some tokens are accepted and some tokens are not. The tokens that are not accepted though, those are renoised because the entire purpose of this model is to denoise it, to remove the noise from the original canvas and find the actual tokens that should be there.
**Speaker**: 于是我们便拥有了一个重新加噪后的画布，它可以顺利进入第二步迭代。在第二步中，模型再次利用之前的 KV 缓存，借助该缓存重新执行类似的去噪计算。但我们在实践中发现，如果仅仅机械地重复这一过程，其实并不是最完美的方案。因为在最初的第一步中，模型对整体生成方向其实已经形成了一定的初始构想；即便其中某些 token 未被最终采纳，它依然蕴含着模型原本想要迈向的潜在趋势。因此，这里引入了一种类似于残差跳连（Skip Connection）的机制，即自条件机制（Self-Conditioning）——将上一步输出的 Logits 直接提取出来，喂入第二步的输入端。接下来，你可以不断循环执行这一过程，通常大约只需要 8 个迭代步左右来持续更新画布。就在这短短 8 步之内（有时甚至更少，有时略多一点），你就能一次性完整获得 256 个高质量 token！当在高端大显存 GPU 上为单一用户提供推理服务时，这使得模型的推理速度快得惊人。当你对比算力受限与内存受限时，“效率”的定义与内涵便被赋予了完全不同的维度。
Original English
**Speaker**: And so we end up with a renoised canvas and it can go ahead and do its second step. And now in the second step it again uses the KV cache and it can use this KV cache to again do a similar processing. But what we found, if you were to do just this, it's not a perfect way to approach this. Because in the very first step, it had an initial idea of what it wanted to do. Even though some of the tokens weren't accepted, it still had a direction it wanted to go into. And so there's kind of a skip connection there, a self conditioning where you take the logits from the previous one and feed it to the input of the second step. And then you can do this process over and over again, typically eight steps or something where it updates this canvas. And then in those eight steps, sometimes less, sometimes more, you get 256 tokens. And it makes your model go incredibly fast when served for a single user on a larger GPU. Efficiency takes a very different meaning than when you compare compute-bound to memory bound.
### 单步去噪决策：Diffusion Gemma 在 Jeff 基础决策模型中的极速应用
**Speaker**: 尽管如此，这种架构依然拥有许多极具价值的应用场景。其中一个我们在上周左右完全没料到会突然爆火的应用，显然就是 Jeff。因为 Jeff 作为一个基础决策模型（Foundational Decision Model）——本质上是一个被专门用来做动作决策的基础分类器——恰好与扩散机制结合得天衣无缝。它的工作机制大致如下：假设你遇到了一种突发状况，“打印机着火了”，或者任何需要你立即采取行动的紧急场景。你把当前的状态、情境等等详细描述出来，再次使用编码器对其进行预处理，目的同样是为了构建出 KV 缓存。
Original English
**Speaker**: But there's still many use cases for that. One we didn't quite expect would blow up in the last week or so, which is obviously Jeff. Because Jeff being a foundational decision model, you know, essentially a foundational classifier that's being used to make decisions, happens to work quite well with diffusion. And it works a little bit like this. So let's say you have a situation, the printer is on fire, anything that requires you to want to take an action. You describe the state, the situation, what have you, you again take that encoder and you process it for the purpose of creating a KV cache.
**Speaker**: 与常规扩散生成不同的是，此时你面对的并不是一个完全填满随机噪声的空白画布，而是预先填充了部分特定的 token（Prefill），而这些预填的 token 直接对应于你希望模型给出判断与建议的具体候选行动选项。好比说，“打印机着火了，你的备选方案要么是决定立刻撤离，要么是呼叫救援”。这些代表选项的 token 保持固定不变，而画布上的其余位置则依然是噪声状态，等待模型去填补判定。这样做的结果是，模型只需要专门针对这两个关键候选 token 进行预测输出，并结合我们前面所展示的机制，直接计算并输出模型判定是应该采取该动作还是不该采取该动作的置信概率。整个过程仅仅需要一次单一的去噪步骤（Single Denoising Step）！严格从技术上讲，它当然可以执行更多步，但如果你追求的是极度敏捷快速的即时决策，仅需一步去噪就足以构建出……嗯，大家现在显然都叫它 Diffusion Gemma Jeff 了。它的效果极其出色，而且根本没有任何人去专门微调过那个模型，所做的唯一一件事仅仅是从特定维度对其施加了推理约束。但这也极具代表性地展示了这种全新的“效率”哲学：在特定维度以特定方式适度施加约束是完全可行的，因为这恰恰开辟了一条全新的道路，确保它能够被高效应用于……
Original English
**Speaker**: What then happens is instead of having a fully noisy canvas, you kind of prefill some of these tokens and the prefill are then related to the actions that you want the model to make a suggestion on. Okay, the printer is on fire and your options are to either, you know, decide if I should evacuate or call it. Those tokens don't change, but the others are still noisy. That's the one it needs to fill in. And what happens is that when you do that, it can make a prediction for only those two tokens and using the things that we saw before, make a probability of whether it should do something or shouldn't do something. Just one denoising step. Technically it can do more, but if you want to make fast quick decisions, this is all it takes to create, well, apparently it's called Diffusion Gemma Jeff now. But that works extremely well because nobody fine-tuned that model. The only thing that happens is just limit it in a way. But that showcases also this concept of efficiency, because it's okay to limit certain things in certain ways because it opens up the way to make sure it can be used for...
### 开源模型与社区赋能的效率革命
**Martin**：……实现了许多我们此前未曾想象的事情。当你把所有这些因素融合在一起时，促成这一切发生的核心关键就在于：这一切完全根植于开源社区。对吧？我们追求极致效率，并不是为了我们自己，而是为了你们，为了实际在各个场景中使用这些模型的广大开发者和用户。无论是部署在智能手机等移动端、笔记本电脑上，还是运行在大型数据中心的高性能 GPU 上，这些其实都没有本质分别。当你把这些开放模型发布出来时——今天大家已经在好几场演讲中听到过类似的探讨——最核心的诉求就是它必须对你们真正有用。把研究成果回馈给社区，这正是我们最核心的初衷。
Original English
**Martin**: ...things we didn't imagine before. When you combine all of that together, one thing that makes all of this happen is that it's all about the community, right? Efficiency is not for us, it's for you. It's for the folks that use actually these models. And whether that's on mobile, on a laptop, on bigger GPU, that doesn't matter. When you release these open models and you've seen a bunch of talks already today talking about that it needs to be useful for you. That's the main purpose of trying to give back to the community.
**Martin**：与此同时，当你选择回馈社区时，你也会清晰地看到社区正在以惊人的创造力反哺给你。比如基于某一个基础模型打造决策模型，而这个基础模型在设计之初甚至根本不是为此目的而构建的。看到这种现象真的非常酷，它带来了大量来自全新领域的创新——你知道，很多垂直应用场景并不是由我们自己亲自去做的，但只要赋予开发者选择权和探索空间，让他们有机会去摆弄、拆解这些模型，深入内部探究其运行机制，去微调、改造它们，你就会催生出各种充满趣味且极具突破性的酷炫模型。至于未来它们究竟会发展到什么高度，谁也无法断言，我目前也没有确切的答案。现在这无疑是一股巨大的浪潮与狂热（hype）。我认为大家对它的追捧有着非常充分且正当的理由，但狂热终归是狂热，我们仍需拭目以待，看看它究竟能走多远。
Original English
**Martin**: But in the same way when you do that you can see the community giving back to you because a decision model based on a model that wasn't necessarily intended for that. Well, that's very cool to see that gives a lot of innovation from a field that you know we didn't do, but by giving people the option and the opportunity to play around with these models to open them up to see what's happening inside them to tweak it to change it you get these fun cool models that who knows where it's going. I have no clue. It's now a big hype. I think for a good reason, but the thing is with hypes, we'll see how far it gets.
**Martin**：进一步深入探讨效率时，你会发现有各种各样细分的研究领域和切入视角值得我们关注。我可以站在这里连续几个小时滔滔不绝地阐述在性能表现之外，效率到底意味着什么。要想在两者之间取得绝佳平衡绝非易事。当前市面上有众多令人瞩目的优秀开源模型都在全力攻坚这些难题，因为这不仅仅关乎算法性能跑分的高低，更关乎你是否能在真实硬件上真正把它运行起来。无论一个模型理论上有多优秀、多强大，如果你根本无法把它部署运行起来，它就毫无现实意义。因此，来自各方的注意力正在向这里汇聚，大家在 Gemma 端侧设备生态中也看到了源源不断的进展。希望这些分享能够帮助大家理解：聚焦在这些体量更小、更轻量化的模型上究竟意味着什么。这与盲目把参数规模越做越大的发展叙事截然不同。一味做大模型固然很有吸引力，也能带来非常惊艳的能力，但这些开放模型、轻量化模型，尤其是在边缘端、终端设备上运行时，必须兼具足够的能力与极快的推理响应。从多元维度审视，效率都是其中最为关键的核心支柱。非常感谢大家！
Original English
**Martin**: And when you then talk about efficiency, there's all these different types of fields and perspectives that you can focus on. And I could stand here and talk for hours upon hours on what efficiency means alongside performance. And that's not an easy thing to do, right? There's so many of these incredible open models out there that try to focus on these things because it's not just about performance but also on whether you can actually run the model, because it doesn't matter how good a model is if you can't run it it's meaningless. And so a lot of attention from different parties and you see a lot happening in Gemma devices and hopefully this helps you get an idea of what it means to focus on these smaller models. It's a very different narrative than going bigger and bigger and bigger. That's also interesting. Gives a lot of very cool capabilities. But these open models, these small models, especially when they're on device, on edge, your small devices, they need to be capable and they need to be quick. And efficiency from many different perspectives is a very big part of that. Thank you very much.
### 会场互动与现场交流
**Host**：太棒了！哎呀，好的。好的，非常感谢大家，让我们再次把掌声送给 Martin！好的，我们的下一位演讲嘉宾将在几分钟后登台。在此期间，我有好多问题想和台下的各位聊聊。我很想了解一下，在这次大会中，大家最喜欢的环节是什么？有人愿意主动分享一下吗？好吧，可能在这么大的会场里抛出开放式问题不太好互动。那么，不知道大家有没有去逛展区，有没有去领周边礼品？就我个人而言，我最喜欢的其实是在展区看到大家热烈讨论并结识新朋友。我觉得这是显而易见的一点，每次我们去往一个新的地方、参加一场行业活动，最核心的目的就是认识同行。但我真的很享受这里逐渐像一个大家庭一样的氛围。我已经参加过很多场全球各地的 AI Engineer 大会了，比如在旧金山、纽约、伦敦，以及今天在这里。能在这里不断看到很多熟悉的面孔，同时又建立起新的联系，这种感觉非常美妙。所以坦白讲，我对大家的建议是：虽然大会剩下的时间已经不多了，但一定要多交流、多建立合作连接。因为依托这个开发者社区，能够孕育出无限的可能；你永远不知道火花何时会迸发，它不一定发生在当下，更可能在未来结出硕果。
Original English
**Host**: All right. Oops. Okay. All right. Thank you guys. Let's give it up one more time for Martin. All right. So our next speaker is going to come in a few minutes. In the meantime, I have so many questions for you guys. So I just would like to know what was your favorite part of this conference? Like does anybody want to share? Okay, maybe two open-ended questions don't work in a big crowd like this. Okay. So, you guys I don't know if you guys went to the expo, you checked the swag, you checked like... but I just want to know for example for me what I like the most is seeing people at the expo discussing and making new friends. I think it's something that is quite obvious to say right like every time we go to a new place, every time we go to an event is to meet people. But I really like that it really becomes like a family. I've been attending AI Engineer for quite a while now around the world—so in San Francisco, in New York, London, and here. And it's so cool to be able to see like sometimes we see the same faces over and over and then we make new connections. So my advice to be honest with you guys, I know we don't have a lot of time left, but yeah just make sure that you connect, make sure that you collaborate, because I think that through this community a lot can happen and you never know it doesn't have to happen now it can happen in the future.
**Host**：会场里另一个让我感触很深的精彩场景，是看到那么多人排着长队去和我们的演讲嘉宾交流。真的有很多人跑来向我打听并讨论我们邀请到的演讲者。显然，P.A.I. 非常受欢迎，ARM 的分享也人气极高。看到我们的参会者来自如此多元的背景，真的很棒。还是没有人愿意分享自己最喜欢的瞬间吗？是餐饮美食吗？哈哈，果然是这样！再次感谢赞助餐饮的 Sarah，看来我们在伙食安排上确实做对了。好的，那么在所有演讲中，大家最喜欢哪一场呢？台下有人说 Gemma 4 太棒了。哇，太赞了！还有别的吗？好吧，大家都在夸 Gemma 4，看来我们干脆整场大会都安排 Gemma 4 的专题好了。还有人提到了 Kitsa，说那场非常幽默风趣。我也很喜欢 Kitsa，向 Kitsa 致敬！他确实太搞笑了，我完全赞同这个评价。好的，关于我就先聊到这里。接下来将是今天全场大会的最后一场压轴演讲。非常感谢大家一直坚持陪伴到现在。在正式开始前，我想向大家隆重介绍来自 Gradium 的 Olivier Tuboul，他将为我们带来关于对话式 AI 缺失之层的深度分享。让我们热烈欢迎 Olivier！
Original English
**Host**: Another nice moment that I saw is so many people lining up to talk to our speakers like yeah so many people come to me and they ask me also about some of our speakers apparently like yeah P.A.I. was quite popular. ARM also is quite popular. So it's good to see that our audience is quite diverse. Nobody wants to share what their favorite moment was? The food. All right. Yeah, I guess so. Yeah, I guess so. Yeah, thank you Sarah again for sponsoring for the food. So, apparently we nailed that. Okay. Any talk, favorite talk? Gemma 4 was great. Okay. Wow. Nice. What else? Okay, that was Gemma 4 and that's it. Like we should have just have Gemma 4 for the entire conference and that's it. The kids that was hilarious and I love Kitsa. Yeah, shout out to Kitsa. But yeah, he's hilarious. Yeah, I agree with that one. All right. Okay, so enough of me, but this is our last presentation of the day. Okay, so thank you so much for hanging in there. But before we get started I would like to introduce Olivier Tuboul from Gradium who is going to talk about the missing layer of conversational AI. Let's give it up for Olivier. Okay. All right.
### 对话式 AI 的架构演进与演变历程
**Olivier Tuboul**：你太客气了。哦，我不是那个意思，我也不是那个意思。好的，我们准备正式开始。让我确认一下设备是否正常工作——屏幕显示正常，太棒了。好的，我可以开始了对吧？没问题。那我们就正式开始。大家好，我叫 Olivier Tuboul，是 Gradium 的联合创始人兼 CTO。今天我要和大家分享的主题是《对话式 AI 中缺失的层级》。这个标题听起来或许有些神秘，接下来大家就会明白它的具体含义。在展开具体技术细节之前，请允许我先向大家简要介绍一下 Gradium 的背景。我们是一家成立刚满一年的初创企业，专注于全栈语音模型的训练。大家可能经常见到这些技术缩写，如果不熟悉的话：STT 代表语音转文本（Speech-to-Text），TTS 代表文本转语音（Text-to-Speech），而 S2S 则是端到端的语音到语音模型（Speech-to-Speech）。在 Gradium，我们致力于研发、训练并提供服务涵盖所有这些形态的模型体系。
Original English
**Olivier Tuboul**: You're extra. Oh, it's not what I meant. It's not what I meant either. All right. Should get started. Let me see if that is working. That is working. Cool. All right. Should I get started? Yes. All right. So, let's get started. Hi everyone. My name is Olivier Tuboul. I'm a CTO and co-founder at Gradium and today I'm going to talk about the missing layers of conversational AI. It's a mysterious title. So, we'll see what that means. Maybe before we do, let me introduce you to who we are at Gradium. So we are a one-year-old startup focused on training voice models. So those acronyms you might have seen, if you did not: STT means speech to text, TTS text to speech, and S2S speech to speech. So all these kind of flavors of models we train and serve at Gradium.
**Olivier Tuboul**：如果追溯 Gradium 的创立渊源，其实离不开 Kyutai。也许大家听说过这个实验室？不知道在座有没有了解的？好的，看到台下有一些朋友点头，太好了。Kyutai 是一家总部位于巴黎的开源科学实验室，成立于 2023 年，拥有相当充裕的资金支持以及极高的科研自由度。正是依托这种开放自由的探索环境，他们取得了一系列重大的科学突破。其中最为大家熟知的成果大概就是 Moshi，在今天的演讲后面我还会展开介绍它，当然还有其他几项重要突破。但可以说，这些前沿语音模型的成功反响极为热烈，以至于我们意识到：为了让前沿研究成果产生更广泛的实际影响力，我们必须将其真正工程化落地为成熟产品。因此，Gradium 成立的核心目标就是填补从实验室科研到商业化落地产品之间的鸿沟。转眼一年过去，我们今天已经站在这里。
Original English
**Olivier Tuboul**: Maybe before that if I go back to the genesis of Gradium is Kyutai. So maybe you've heard of it. I don't know. Yes, some of you. Great. So Kyutai is a Paris-based open science lab that was founded in 2023 with quite some money and a lot of freedom and out of this freedom they you know made some great scientific breakthrough. Moshi is probably the most famous and I'll go back to that a bit later in the presentation and obviously some others but that I would say the success of those voice models were so big that somehow we thought in order to get the you know bigger reach of our research idea we should turn that into products. So the goal of Gradium was really to bridge this gap from research to product and and here we are one year later.
**Olivier Tuboul**：那么，回到对话式 AI 这个话题。我们在哪里能接触到它？实际上在今天，语音智能体（Voice Agents）几乎已经无处不在。无论是游戏互动、直播交互、个人智能助理，还是具身机器人，只要涉及到人机语音对话的场景，背后都有语音智能体的深度参与。今天我想先带大家回顾一下它的一段发展演进史。如果让我带大家回溯到可以略带挑衅地称之为语音智能体“史前时代”的阶段——也就是大语言模型（LLM）出现之前的时代，大家可能对那段记忆犹新。我想在当时，那大概是大众距离通用人工智能（AGI）最近的一次初窥体验。当时给人的感觉真的就像是 AGI 已经降临，那就是 iPhone 以及它搭载的 Siri。然而当时的 Siri 本质上跟真正的 AGI 毫无关系，它实际上是一个极其精巧的工程产品，由非常多的模块层级堆叠组合而成——准确来说，足足包含了六个层级。
Original English
**Olivier Tuboul**: So, conversational AI, right? Where do we get that? Actually, today pretty much everywhere you've heard about voice agents, whether it's in gaming, live streaming, assistants, robotics, wherever you want to talk to a machine, there is a voice agent involved. And I would like to explain you a bit of the history today, not only but part of it. And if I go back to well what I could call provocatively the prehistoric time of voice agents meaning before LLMs you probably remember that right I think that was the closest um I would say glance at AGI back in the days right it really felt like AGI right that was the iPhone and Siri and Siri was nothing really AGI was A very nice product, engineering product made of different layers, lots of layers, six of them, sorry.
**Olivier Tuboul**：第一层是语音识别，也就是将语音转录为文本的 STT 模块。当用户的音频信号被转换为文本之后，就会流转进入分类器模型，试图解析出用户的意图（intent）是什么、动词对象（object）是什么、主语（subject）和补语（complement）分别是什么等等。在此之后，系统需要依赖对话管理器（dialogue manager），并调用各类手机本地或云端的外部 API，比如查询天气状况、读取并修改手机日历等。接着，可以说是最重度依赖模板规则和正则表达式（regex）的环节：拼接并生成准备让 Siri 说出来的响应文本；最后，再通过文本转语音（TTS）引擎将生成的文本大声朗读播放出来。这就是过去的经典架构。随后大语言模型（LLMs）时代来临，人们纷纷感叹：也许我们根本不再需要这套繁琐的层层级联系统了！传统的自然语言理解（NLU）模块、那些机械的硬编码模板，全都可以统统丢弃。不过，说 LLM 取代了一切其实多少带有一点谎言或夸张成分，在现代语音智能体架构中，对话管理器依然在扮演着不可或缺的角色。而与此同时，这些大语言模型自身的能力也在变得越来越强悍。
Original English
**Olivier Tuboul**: First one is the speech to text the speech recognition. After the audio is turned into text, it was turned into you know classification classifiers trying to say what is the intent, what is the object, what is the subject, what is the complement, stuff like that. A dialog manager, some calls to APIs like, you know, check the weather, check the calendar on the phone, stuff like that. Probably the most, you know, I would imagine template and regex's intensive part was to generate some text for Siri to be spoken out and then the TTS would speak it out loud, right? Then what came after was LLMs and I say maybe we don't need all of that, right? This natural language understanding, those templates all gone. LLM, there's a bit of a lie in here, right? There is still a dialogue manager in modern voice agents. And those LLMs, they become stronger, right?
### 对话式 AI 的架构演进与 Gradium 的业务范畴
**Speaker**: 现在它们已经能够连接到外部世界了。它们可以进行交互，可以与数据库对话，进行网络搜索，调用 MCP（模型上下文协议）、工具调用、信息检索、RAG（检索增强生成），于是它们演化成了语音智能体（voice agents）。你们在这里所看到的，实际上正是目前大约 99% 的语音智能体的工作方式。这被称为级联模型（cascaded model）：ASR（语音识别，或语音转文本 STT）接 LLM（大语言模型），再接 TTS（文本转语音），对吧？它们并不是联合训练出来的。这是三个独立的 AI 模块，我们将它们一个接一个地串联插接在一起。
Original English
**Speaker**: And they are now able to connect to the external world. They can interact. They can talk to a database, a web search, MCPs, tool calls, retrieval, rag and they became voice agents and that's what you're seeing here is actually how I would say 99% of voice agent are working. It's called the cascaded model ASR or TT ST LLM TTS right they are not trained jointly. Those are three pieces of AI modules that we plug one into the other.
**Speaker**: 这种架构还有一种拓展延伸，那就是思考：也许我们并不需要三个独立的模块，也许你只需要一个大模型即可，这就是我们所说的端到端语音到语音（speech-to-speech）。在那种模式下，完全不需要额外去处理轮次转换（turn taking），模型本身就清楚该在什么时候开口说话，什么时候该倾听。这能让大家对什么是对话式 AI（conversational AI）有一个整体的了解。接下来，我也可以向大家概览介绍一下我们在 Gradium 所做的事情，随后我会深入讲解这背后的工程层面和 AI 技术层面的底层内幕。
Original English
**Speaker**: There is a an extension of this which is okay maybe we don't need three pieces. Maybe you can just one big model and that's we call speech to speech and then there is no turn taking to take into account. The model itself knows where to speak and when to listen. So this is to give you an overview of what is convers conversational AI. Uh so I can also give you a glimpse of what we do and then I'll talk about the engineering aspect and the AI aspect of what's under the hood.
**Speaker**: 那么，我们在 Gradium 究竟在做什么呢？基本上所有这些环节我们都在做，对吧？我们做文本转语音（TTS），做语音转文本（STT），我们在端侧设备上做文本转语音，我们做语音到语音的翻译，我们还做声音设计（voice design）——因为当你想要把内容大声朗读出来时，你会希望使用某种特定的声音。比如可以带有法国口音的声音，也可以是女性声音，高音调、低音调等等，各种风格均可。而且，我们马上要推出的正是语音到语音（speech-to-speech）模型。
Original English
**Speaker**: So what do we do at Gradium? We basically do all of this, right? We do the text to speech, the speech to text, we do the text to speech on device. Uh we do some speechtoech translation. We do some voice design because when you want to say something out loud, you want to say it in some voice. Can be a French accent voice for example or it can be a female voice, high pitch, low pitch, whatever. And coming soon is actually the speech to speech.
### 语音 AI 缺失的关键层与真实场景挑战
**Speaker**: 这次演讲的主题叫做“语音 AI 缺失的一层”（The missing layer of voice AI）。那么，这里究竟缺失了什么？表面上看起来它似乎已经很完整了，对吧？有些人会试图向你证明这个技术版图已经完整无缺，但其实并非如此。这些还仅仅是其中的一部分例子，远非全部。不过我们接下来会对此做更深入的探讨。如果你把语音 AI 放到真实的开放环境（in the wild）中，它很可能会彻底失控失序。因为目前绝大多数的应用场景都是在手机端、单一说话人、人机一对一通话这种受限环境下进行的。
Original English
**Speaker**: So this talk is called uh the missing layer of voice AI. So what is actually missing? It feels like it's it's complete, right? And some people will try to convince you this picture is complete but it's not. And those are just some examples. It's not everything. Uh but we're going to talk more about that. So if you put you know voice AI in the wild, well most likely it's going to go wild, right? Because most of the um the applications do there are more on the phone, one speaker uh you know one computer talking to one another.
**Speaker**: 但一旦将其置于现实生活中，你就会遇到多个说话人同时存在的情况，这会让智能体产生混乱；背景噪音也会干扰混乱智能体；而在长时间对话中，你又会丢失上下文；此外还有真正的全球规模化（global scale）问题。我所说的真正全球规模化，并非指任何人都能够访问你的服务这种普通可用性，而是指像当年 Siri 那样具备的能力——让每个人都可以在自己的手机上直接通过说话进行交互，而无需打字输入。
Original English
**Speaker**: But put that in real life you get like many speakers. it confuses the agent. Noises confusing the agent long sessions you leave the lose the context and also the real global scale. Uh and what I mean real global scale is like not the ability for anyone to reach your your service but like the ability as that was Siri was doing like everyone on his phone can talk to it instead of typing.
**Speaker**: 在这些表象问题的底层，存在着若干核心技术挑战，我将挑选其中的两个重点展开剖析，深入解释我们是如何做的以及面临的挑战所在。如果你想打造语音模型，你需要具备若干条件。归结起来，你想要的是具备极低延迟的高速系统，以及能够进行大规模服务化部署的架构。因此，你必须从底层最基础的设计开始，就将“高速运行”和“大规模可扩展”——也就是真正的大规模实时能力——纳入考量。这意味着在模型设计之初就要使其具备扩展能力并能快速提供推理服务，同时还需要对推理过程进行极限优化，并构建能够支撑它的配套服务基础设施。接下来我将重点讲讲第一个方面和最后一个方面。
Original English
**Speaker**: So underlying those issues are some challenges and I'm going to pick two of them to explain to dig into how how we do and what's the challenge. So if you want to do um voice models you need to several things. Uh basically you want something that's fast uh and something you can serve at scale. And so you have to think about this fast and at scale this real time at a scale from from the ground up. So it means like designing a model that will be able to scale and that will be able to be to serve fast optimizing the inference and having a serving infrastructure that will support it. I'm going to talk about the first one and the last one.
### 从 LLM 自回归机制到音频 Token 化
**Speaker**: 那么，这些模型究竟是如何工作的呢？当你想要设计一个新模型时，你基本上不会从零完全重新开始，对吧？你往往会从某个相似的技术出发，并尝试在此基础上进行改造适配。因此，对于语音到语音系统而言，你真正需要的是：一个能够实时接收输入语音的系统，随着对话的持续进行它能够持续接收越来越多的语音流，并且能够尽可能快地开始开口回应。当然，其中一个硬性要求就是“语音进、语音出”（speech in, speech out）。
Original English
**Speaker**: So how do those how does model work? Um, so when you want to design a new model, you don't start from scratch basically, right? You start from something that's similar and try to adapt. So what you really need for and I'm talking about the speechtoech uh system you need something that will take the speech as it comes and that will be able to take more and more speech as the conversation keep going and that will be able to as soon as possible start speaking in return right and of course there is a one of the requirements that it's speech in speech out right so even if we don't those models.
**Speaker**: 尽管传统上并没有天然现成的这类端到端模型，但有一族模型与此极其相似，并且能够满足这三项要求中的前两项，那就是大语言模型（LLM）。对于 LLM，你输入的是文本，输出的也是文本。那么它们是如何运作的呢？你拥有一个上下文环境（context）。一开始这是提示词（prompt），但随后它就成为了对话上下文。这些内容都是词元（words/tokens），而 LLM 经过训练正是为了预测下一个词元是什么。
Original English
**Speaker**: There are a family of models that is really similar to this and that satisfies two out of those three requirements which are LMS. LMS you take a text in and a text out. So how do they work? You have a context. At the beginning it's the prompt but then it's a context. Those are words and the LLM is trained to predict what's next. Okay.
**Speaker**: 在这种情况下，假设有句子“Gradium is an AI model company based...”（Gradium 是一家 AI 模型公司，总部位于……），接下来的词会是什么？是“in”、“of”、“at”，还是“blue bird”？模型会为所有可能的候选词计算一个概率分布。显而易见，“in”看起来是一个非常合理的候选词。如果将其抽象表达：底线代表上下文，顶线代表预测输出。这里的精妙之处在于——这也是为什么它被称为自回归模型（autoregressive models）——当前的下一次预测将会自动合并进下一轮预测的上下文中。比如，你预测出了“AI”，“AI”随即成为上下文的一部分，接着它又能够生成下一个新词、再下一个新词，以此类推，逐步在运行中预测并生成完整的句子。
Original English
**Speaker**: So in that situation gradient is an AI model company based what's next is it in off at blue bird you know it gives a probability to all of them so of course you know in seems to be a good candidate so if you represent it that way at the bottom line this is the context on the top line this is the prediction what's interesting is that and that's why it's called auto reggressive models the next prediction is going to be part of the context at the next round. So you say oop sorry um gradium is unpredicted AI AI becomes part of the context and now it can bring a new one and another one and another one and this way on the go you will uh predict and generate a full sentence.
### 音频连续维度难题与神经编解码器（Neural Codec）
**Speaker**: 那么，这种机制能直接套用到音频上吗？直觉上感觉是完全一样的，对吧？音频本质上也是序列，是序列到序列（sequence-to-sequence）。为什么就不能直接用呢？这里的症结在于维度灾难（dimensionality issue）。音频面临的难题是：像“Gradium is an AI model company based in Paris”这句文本，一共只有 9 个单词——也就是 8 个单词作为上下文，去预测第 9 个单词。当它被大声朗读出来时，时长大约是 3 秒钟。但是 3 秒钟的连续音频，如果在计算机上以普通标准质量（24 kHz 采样率）表示，每秒就有 24,000 个采样点。这 8 个单词直接对应成了多达 72,000 个采样数值。
Original English
**Speaker**: So can it work on audio? Feels like it's the same right? Audio is a sequence sequence to sequence. Why could not it work? Well, there is a dimensionality issue here. Um, the problem with audio is that this text, Grandom is an IMO company based in Paris. It's nine words. So, like eight words for the context one you're going to predict. It's 3 seconds when it's spoken out loud. But 3 seconds of audio, if you represent it on a computer on a normal quality, it will be like 24,000 samples per second. those eight words becomes seven 72,000 sample uh values.
**Speaker**: 如此一来，序列的长度发生了暴增。你需要预测的数值量级足足增加了 10,000 倍。由于 LLM 所依赖的 Transformer 架构采用的是自注意力机制（self-attention），其计算复杂度是随序列长度呈二次方级增长的（quadratic），这意味着你的注意力矩阵规模将会暴涨 1 亿倍（100 million times bigger）。这在计算上是完全不可行的（not tractable）。因此，解决这个难题的第一项关键技术手段，就是把原始音频转换成离散的词元（tokens），这就是神经编解码器（neural codec）。事实证明这是完全可行的：你可以训练一个深度学习模型，将 24 kHz 的高频连续音频大幅压缩为每秒仅 12.5 Hz 的离散音频 token。这样你就将数据维度直接除以了近 2000 倍，从而得到了一个可处理、可计算的离散 token 序列，随后你便可以启动预测生成，因为最终模型再将这些 token 重新解码回可听的音频即可。
Original English
**Speaker**: So it's much longer of a sequence. So you have 10,000 times more sequ uh values to predict and since the transformer architecture which what's the LLM is relying upon is relying on self attention which is quadratic. then your uh attention matrix is 100 million times bigger. So it's not tractable. So the first trick is to turn the audio into tokens. And that's the neural codec. And actually it turns out you can do that. You can train a model to compress the audio in tokens from 24 kHz to 12 hertz 12.5. So you divide by 2,00 then you have a tractable number of tokens that's a sequence and you can start predicting because at the end of the story will decode them into back to audio.
### 多流分层 Transformer 与实时全双工对话
**Speaker**: 如果你想用这种方法来训练一个对话系统，你可以设想：我就是 LLM，我是图里的灰色方块。我们有了一段序列：我说话、我说话、我说话，然后你说话、你说话，接着又是我说话，等等。这是一个时间序列，我们据此预测下一个音频 token，对吧？但是——很抱歉——现实中的真实对话并不是这样运转的。因为在真实对话中，绝不是机械地要么轮到我说、要么轮到你说。我们完全可能在同一时刻同时开口说话。如果不允许这样，那它就变成了如同对讲机（walkie-talkie）一般的轮流对话体验，那种交互方式非常生硬低级。
Original English
**Speaker**: So if you wanted to do that to train a dialogue system you could say me I'm the I'm the LLM okay I'm the gray one. So, we have a sequence of me talking, me talking, me talking, you talking, you talking, me talking, blah, blah, blah. That's a sequence and we're going to predict the next audio tokens, right? But this, sorry, sorry, sorry, sorry. This doesn't really work like this because in a conversation, it's not like I talk or you talk. We might be talking at the same time. Otherwise, it's just like walkie-talkie uh conversation, which which is lame.
**Speaker**: 因此在实际工程实现中，你必须发明一种传统 LLM 中所不存在的全新机制——多流分层 Transformer（multi-stream hierarchical transformers）。这样一来，在上下文的时间轴上不再只有单一的时间序列通道，而是拥有两个并行的通道：一个通道是我在说话，另一个通道是你在说话，这两个流之间进行交叉注意力关联，以此来联合预测接下来的内容。正因如此，这类端到端模型能够做到在说话的同时进行倾听。
Original English
**Speaker**: So in practice you have to invent something new that we don't have in LLMs is multiream hierarchical transformers. So like now you don't have a single line of time for the context. You have two you have me speaking and you speaking and those two things will attend to to be able to predict what's next. And so those models they can listen and talk at the same time.
**Speaker**: 给大家播放一个具体的演示示例。抱歉，请看这里：
Original English
**Speaker**: Give you an example. Sorry, of course.
**Alex**: 那么，目标星球是天狼星 22（Sirius 22）。你能为它规划一条航线轨迹吗？
Original English
**Alex**: So, the planet is Sirius 22. Can you plot a trajectory course to it, please?
**Voice Agent**: 好的，先生。
Original English
**Voice Agent**: >> Yes, sir.
**Alex**: 好的，我们到达那里大约需要多长时间？
Original English
**Alex**: >> Okay. How long is it going to take us to get there?
**Voice Agent**: 我已经绘制好了路线，大约需要 5 个月才能到达。
Original English
**Voice Agent**: >> I've mapped it out. It's approximately 5 months to get there.
**Alex**: 好的，那还不算太久。你觉得飞船上已经备齐了启动任务所需的一切物资了吗？
Original English
**Alex**: Okay, that's that's not too bad. Uh, do you think we have all we need on board the ship to start the mission?
**Voice Agent**: 是的，先生，我们所需的一切都已齐备。
Original English
**Voice Agent**: >> Yes, sir. We have everything we need.
**Alex**: 好的。
Original English
**Alex**: >> Okay.
**Voice Agent**: 今天……
Original English
**Voice Agent**: >> Today,
**Speaker**: 好的，第二个示例我就先跳过了。大家可以看到，这种体验非常自然，模型经过专门训练，能够在说话的同时实时倾听，因此几乎没有任何感知延迟。刚才录音里的是 Alex，他在 2024 年担任我们的首席科学官（Chief Scientific Officer）。不过这里还有一件我刚才没告诉大家的事：早在 2024 年我们在 Kyutai 就已经掌握了这项技术，但当时我们并没有将其商业化出售。
Original English
**Speaker**: >> okay, I'm going to skip the second one. So, you see like it's very natural that the model is trained to listen and talk at the same time. So, there is no latency whatsoever. And that was Alex, our chief scientific officer in 2024. So now you're there is something I didn't tell you is that in 2024 we had this technology at QI but we're not selling it.
### 工具调用：语音到语音模型走向生产的真正缺失层
**Speaker**: 那么，这里缺失的关键层究竟是什么呢？这里存在一个核心症结：这仅仅只是一个单独的单一模型，它却需要同时兼顾听音、发音以及理解处理自然语言。对于一个参数规模十分精简的模型来说，这需要承担的任务实在是太多了。因此当你和它深度交谈到一定程度时，你就会发现它的表现其实差强人意。更严重的局限性在于，这个模型在某种程度上是物理隔离、与世隔绝的（air-gapped），它根本无法与外部真实世界进行通信交互。如果它无法连接外部世界，如果你无法借助它真正执行诸如预约日程或业务操作等实际行动，那么在工业生产环境中，它充其量就只是一个博人眼球的玩具摆件（gadget）。因此，摆在当今端到端语音到语音模型面前最大的核心挑战，就是如何实现与工具调用（tool calling）的无缝深度集成。如果能够做到这一点……
Original English
**Speaker**: So what's the missing layer here? Well, there is something here is that this one model is just one model and it's capable of um listening, speaking and you know processing language. That's a lot of things for a very small model. So when you talk to it at some point you will think okay it's not as good as you know the limitation is even worse is that this model is kind of airgapped it doesn't talk to the external world. So if it doesn't talk to the world if you cannot act on you know booking an appointment or something in production it's it's just a gadget. So the big challenge for speech to speech today is to be able to do seamless integration with tool calling. And if you do that,
### 语音智能体与低延迟架构挑战
**Olivier**: 这样你就能获得相同的体验，同时能够做到语音智能体（Voice Agent）应该做到的事情。好的，上面大致介绍了我们如何构建模型，当然这并不是全部秘方。那么，假设你现在已经拥有了这个模型，你希望把它规模化并推向全球进行服务，你该如何做到这一点？这里我们就遇到了延迟问题。假设有一位用户坐在电脑前——我这里以文本转语音（Text-to-Speech, TTS）为例——他向我们的服务器发送一些文本，而我们必须向他返回相应的音频。如果你想把这个功能接入语音智能体，它就必须足够快，必须是实时的。我们所说的“实时”，意味着延迟要低于 300 毫秒。所以，这就是你面对整个链路所拥有的全部延迟预算。如果做不到这一点，整个对话就会变得非常诡异——你需要等待相当长的时间才能听到答复，用户很快就会感到厌烦。因此，系统必须极其迅速。
Original English
**Olivier**: then you have the same experience while being able to do what a voice agent should do. Okay, so that was about how we build a model roughly, not all the recipe. Um, and now let's say you have the model, you want to scale it and serve it to the world. How do you do that? So here we got the latency problem. So you have one person with his computer. I'm going to take the text to speech example here sending some text to our server and we have to answer some audio back and if you want to plug that into a voice agent it has to be fast has to be real time and by real time we mean less than 300 milliseconds. So that's the latency budget that you have for the whole thing. If you don't, then it it's going to be like a weird conversation where you wait for a while before getting an answer and you're going to be bored. So, it has to be fast.
**Olivier**: 在整个请求往返的旅程中，实际上必须穿透三层结构。第一层是传输层，也就是网络层，对吧？信息必须从 A 点跨越物理网络传输到 B 点。接着是产品业务层，因为有人向我发送了一个请求，我必须对此进行校验：这个请求是否合法？这个人是否有访问权限、账户里是否有足够的额度？他想要使用某个特定的声音，我们系统中是否存在这个声音？它存放在哪里？等等诸如此类。因此，你必须在产品层对请求进行信息补充，或者直接予以拒绝。最后则是计算层。那么，我们怎样才能把这一切都处理得足够快呢？
Original English
**Olivier**: And during that trip, you have actually three layers that you have to go through. One is a transport layer, right? Network layer. The information has to travel from point A to point B. Then you have the product layer because like someone sends me a request. I should check this is that legitimate. Does this person uh does this person have the permission the credits? He wants this voice. Do we have this voice? Where is it? And so on and so forth. So you have to enhance the request or deny the request. And then there is a compute. So how do we do that fast?
**Olivier**: 首先必须解决的是网络传输的延迟问题。比方说，我把服务器部署在欧洲，而我的客户端位于美国。仅仅是在大西洋两端进行一次网络往返，就会浪费掉整整 100 毫秒——直接耗掉了全部延迟预算的三分之一，而这还仅仅是用在纯粹的信息传输上。那该如何解决？办法看似简单：在全球各地都部署服务器，对吧？虽然成本更加昂贵，但能有效缩短客户端到服务器之间的物理距离。然而随之而来的是全新的工程难题：原本是一个没有副本、没有分布式的单体系统，现在变成了一个信息必须时刻保持同步的分布式系统。当然，为了追求速度，你必然会引入缓存。原本有一套数据库，再配上数据库的缓存层，缓存失效（Cache Invalidation）本身就已经是一大挑战了；而现在你拥有大量分布各地的缓存，它们之间还必须全部实时保持同步。否则，用户刚在某个地方创建了一个 API 密钥，随后在加州调用时系统却还不识别该密钥，这就非常诡异了。因此，你实际上是用工程复杂度来置换低延迟。虽然我们有能力应对这种挑战，但保持清醒的认知非常重要。
Original English
**Olivier**: So first problem you have to solve is a latency problem. So let's say I've put my server somewhere in Europe and my client is somewhere in the US. Well, just by going round trip through the Atlantic Ocean, I kind of I'm wasting 100 milliseconds, onethird of my budget for just transporting the information. So how do I fix it? Okay, easy. I'm going to put servers everywhere, right? It's more expensive first, but I'm going to reduce the trip from the client to the server. But now I got new issues, right? Because I had the system with no replication and no distribution. Now I have a distributed systems where the information has to be synchronized. Of course, if you want to go fast, you put caches, right? So I put I have a database. I have cache of my database. It's already an issue per se for you know all the cache invalidation. But now you have many caches that has all to be synchronized alto together. Otherwise, you create an API key and you're going to California. It doesn't know about the API key. That's weird. Okay. So, you're trading latency for complexity, engineering complexity. We can handle it, but still it's it's important to be aware of it.
**Olivier**: 第二个我认为非常新颖的问题是 GPU 算力资源。我们一年前刚起步的时候，大家都清楚：如果突然需要一张新的 GPU 来处理涌入的请求，绝对不可能指望当场向云服务商申请一张 GPU，对方立刻分配到位，然后你拉取模型权重、拉取 Docker 镜像、对 GPU 进行预热——并妄想这一整套流程能在 300 毫秒内完成。这根本不可能，对吧？就算费尽心力通常也得以分钟计。所以动态扩容从一开始就绝不能作为备选方案，你必须提前对负载做出精准预测并随时待命。然而更糟糕的现状在于，目前 GPU 算力的稀缺程度已经严重到即使提前一天申请，也未必能拿得到。你可能急需扩容一张 GPU，但所在的整个可用区根本没有多余卡源，云服务商只能告诉你：“抱歉，所有人都在抢 GPU，我们已经无卡可用了。”面对这种局面该如何处理？同样，你必须通过增加系统的复杂度来对冲这一风险。
Original English
**Olivier**: Then the second thing which I think is very very new. You know when we started a year ago, we were thinking there is no way if I need a new GPU to handle my request. There is no way I can ask my cloud provider for a GPU. He gives me a GPU or she give me a GPU. Uh the GPU is ready and I pull the weights. I pull the Docker image. I warm up the GPU and this going to happen in 300 millconds. Impossible, right? like if you try you you you can maybe make it minutes right today what's happening so we we never put that as a as an option you have to predict and to be ready for the load but now what's even worse is that the GPU scarcity is so important that you may not be able to have it even like in next day so maybe you want one more GPU but you don't have this GPU because there is no GPU available in your region because one of the cloud provider are saying sorry there is everyone wants a GPU we don't have it so how do you handle that so again you're going to trade you're going to mitigate this risk with more complexity and there are two things you can do the first thing is I'm going to go to different providers there is more the odds the odds to find the GPU is is better or I'm going to try to uh have my model run on different architecture on different models on different chip providers, right? Not only Nvidia or not only H100 or stuff like that. So that I increases the odds of finding a GPU. So again to mitigate that risk, I'm increasing the complexity of my system.
**Olivier**: 具体来说，有两条应对路径：第一种是接入多家不同的云服务商，这样能找到可用 GPU 的概率就会显著提升；第二种是优化模型，使其能够跨不同的硬件架构、不同的芯片型号以及不同的硬件供应商运行，而不仅仅局限于英伟达（NVIDIA）或仅绑定 H100 等单一芯片，从而最大化捕获可用算力的几率。为了化解硬件短缺的风险，系统的复杂度再次被推高。那么在遇到流量洪峰（Burst）时又该如何应对？有时所有用户会同时发起请求。针对突发峰值，我们主要采取两种策略：第一种是充分利用全球分布的所有集群，将瞬时过载的流量智能重定向至距离最近且有空闲算力的其他集群；第二种是动态调整模型的批处理大小（Batch Size）——在流量洪峰冲击系统时，适度以微小的延迟让步换取更大的 Batch Size 吞吐量，等洪峰平息后再将 Batch Size 回调。实际工程中，当你把优秀的模型算法、极速的推理引擎以及敏捷的服务架构串联在一起时，就能达成理想的效果。本周我们正在构建一个全新的 TTS 测试版模型，并进行了一场竞速对比：基本上在同一时间触发请求，实测各家服务商与我们返回音频所需的时间。接下来请听实测展示。
Original English
**Olivier**: And how do you handle like burst, right? Because you can uh sometimes everybody wants at the same time uh to be served, right? And there are different strategies you can do and we take two options. First one is to say now let's take advantage of all those clusters and reroute the traffic from one to the other the closest one. Or you can also do change the batch size of your model. Maybe you can trade a little bit of latency with a bigger batch size right for the time where you hit the wave. You increase the batch size and then you decrease it. In practice, if you put all that one after the other, a good model, fast inference, fastly served, that's what you get. So, we did um we're building this week a new beta model for text to speech and we did that race where we click somehow at the same time, but we measure the time for uh how much it takes to get the audio back for us and some providers. And this is what we we got.
### 端侧轻量化模型与声音快速克隆演示
**Gradium 语音合成模型演示**: 大家好，我是 Gradium 的最新模型，也是我们迄今为止构建的速度最快的文本转语音模型。首包音频生成更快，对话更自然，零等待，速度有保障。欢迎立即体验。
Original English
**Gradium Model Demo**: Hi, I'm Gradium's newest model and I'm the fastest text to speech we've ever built. Faster first audio, more natural conversations, no waiting, and guaranteed speed. Try it today.
**Olivier**: 没错。看到我们能够不断突破延迟的极限，我们感到非常振奋。行业内以往最顶尖的模型延迟大约在 150 毫秒左右，而我们将其提速了近三倍。延迟在物理上必然存在极限，永远不可能做到绝对的零毫秒，但能将这些全栈技术整合落地，我们感到非常欣慰。在结束今天的分享前，我想探讨一个更深的问题：我们还能走得更远吗？刚才提到物理延迟不可能做到绝对的零毫秒，这确实是事实。但是，我们能否消除网络传输的耗时，做到 0 毫秒传输？能否消除产品业务层的逻辑耗时，做到 0 毫秒产品层，同时依然保持极高速度并实现更大规模的扩展？这就是将计算直接从云端服务器下沉到客户端本地的核心构想。要实现这一点，就必须研发出能够直接运行在端侧设备上的超微型模型——无论是你的手机、浏览器还是树莓派（Raspberry Pi）等等。为此，我们还发布了 GradPhon 模型，它拥有极低的参数量，仅有 1 亿（100M）参数，但保持了极高的语音准确度。大家同样可以上手试用，如果想体验这些模型欢迎随时与我们联系。接下来我为大家播放几段它的实际生成音频。
Original English
**Olivier**: Yeah. So, we're happy that we're, you know, pushing the boundaries of of latency. you know, the best models were about like 150 millconds and we made it like three times faster. So, at some point there is no there is no limit, right? It's not never going to be, you know, one zero milliseconds, but we're happy that we're able to put all that together uh to make it happen. Now, to close this talk, I'm going to take talk about can we go further? Actually, I'm telling I told you we cannot go zero millisecond and that's true. But can we go zero milliseconds of travel? Can we go zero milliseconds of product and still have something fast that would scale even more? And that's the idea of moving the compute from the server to the client directly. And for this you have to have very tiny models that can run on your device on your phone on your brother on a raspberry whatever. Uh so we also released like grad phon which is the smallest it's not exactly the smallest one of the smallest but with a very low u number of parameter 100 million parameter um only uh and a very good accuracy and this you can also try feel free to reach out if you want to try those models I'm going to give you a little bit of examples of what it sounds
**GradPhon 语音模型演示**: 只有 1 亿参数，我可以使用你给我的任何声音进行说话。
Original English
**GradPhon Demo**: only 100 million parameters and I can speak in any voice you give me.
**GradPhon 语音模型演示**: 这是一个全新声音的示例，仅克隆自一段 10 秒钟的声音样本。
Original English
**GradPhon Demo**: And here's an example with a brand new voice cloned from a 10-second sample.
**Olivier**: 好的，这个片段我们先跳过。
Original English
**Olivier**: Okay, we're going to skip.
**GradPhon 语音模型演示**: 这个声音在一分钟前还根本不存在。我是完全从一段单次的 10 秒录音中生成出来的。
Original English
**GradPhon Demo**: This voice didn't exist a minute ago. I generated it from a single 10-second recording.
**GradPhon 语音模型演示**: 完全相同的模型，截然不同的声音表现。
Original English
**GradPhon Demo**: Same model, completely different voice.
**GradPhon 语音模型演示**: 仅凭借一段简短音频，实时动态克隆。
Original English
**GradPhon Demo**: Cloned on the fly from a short clip.
**Olivier**: 好的，以上就是我今天想向大家分享的全部内容：带大家深入了解语音 AI 的底层架构、我们如何构建对话智能体、如何训练这些模型、如何高效部署服务，以及如果我们把前沿研究与精湛工程结合在一起，我们能够如何真正拓展对话式 AI 的前沿边界。非常感谢大家！
Original English
**Olivier**: All right, so that's that's pretty much all I wanted to uh tell you about. you know what's below the hood in in the voice uh AI how we build conversational agents how we train those model how we serve those models and how if we you know put together nice research and nice engineering we can really uh push the boundaries of of conversational AI thank you
### 下半场议程过渡与 Cassandra Chin 登场介绍
**主持人**: 非常感谢 Olivier！让我们用热烈的掌声送给 Olivier 和 Gradium！太棒了。好的。
Original English
**Host**: thank you so much Olivier let's give it up for Olivia for and Gradium Nice. Oh, all right.
**主持人**: 好的。如果大家扫描刚才分享给各位的二维码，就可以领取体验 Gradium 的额度。好的，谢谢大家。接下来我们还为各位准备了精彩内容，请大家先不要离开。实际上，我在人工智能领域工作已经有相当长的一段时间了。虽然不敢说是一段非常漫长的岁月，但体感上确实历经了很久。而我被问到最多的问题之一就是：你认为下一代人将会面临怎样的未来？这是一个非常棘手的问题，对吧？因为技术的演进实在太快了，即使作为成年人的我们，要完全理解这些技术变革带来的深远影响也已经十分不易。不过值得庆幸的是，已经有先锋者正在为此付诸实际行动。接下来，我想隆重邀请 Cassandra Chin 上台，她将为大家深入分享这方面的内容。请大家用热烈的掌声欢迎 Cassandra！
Original English
**Host**: Okay. So, if you follow the QR code that was uh that was shared with you guys, then you can have uh credits, right, to to try out Gradium. All right. Thank you. Okay. So, uh we still have a few more things for you guys. So, don't don't go don't leave just yet. Um actually I've been working in AI for uh quite some time now. I can't say that it was a long time but it feels like a long time. And one of the questions that I get the most is what do you think is going to happen to the next generation? And it's a tough question right because things are moving so fast and it's already quite hard for us as adults to understand the impact of all this technological change. Um, but luckily for us, there are people that are doing something about it. And uh, I I would like to welcome Cassandra Chin, who's going to tell you more about it. So, please let's give it up for Cassandra.
**Cassandra Chin**: 大家好。我叫 Cassandra Chin，我是……
Original English
**Cassandra Chin**: Hi. So, my name is Cassandra Chin and I am one of
### 启蒙下一代：AI Engineer 巴黎青少年工坊
**Cassandra**：……我是 AI Engineer 巴黎大会青少年工坊的讲师。这个周末，我们举办了一场面向孩子们的活动，向他们普及科技知识，帮助他们学习技术，以便更好地为这个人工智能时代做好准备。不过在深入展开之前，我想先聊聊我自己，以及为什么我对向孩子们传授科技抱有如此巨大的热情。
Original English
**Cassandra**: ... the kids workshop instructors for AI Engineer Paris. This weekend we ran a kids event where we educated kids and helped teach them to learn about technology to prepare them for this AI world. But before I get too much into it, I want to talk a little bit about myself and why I'm so passionate about teaching kids technology.
**Cassandra**：其实，我自己还是个孩子的时候就开始教别的小朋友了。那时我只有 13 岁，会在周末给孩子们开办工作坊——主要是树莓派（Raspberry Pi）工坊，孩子们可以在那里动手摆弄、探索各种各样的技术。
Original English
**Cassandra**: I actually started teaching kids when I was a kid myself. I was 13 years old at the time and I would teach kids workshops over the weekend, Raspberry Pi workshops where kids would get to tinker with different technologies.
**Cassandra**：甚至在我开始教学之前，我们家里就充斥着各种科技设备。我们有 3D 打印机，还有很多形形色色的数码硬件。我记得很小的时候就自己动手组装电脑。我始终坚信，激发孩子们热爱科技，远比强推他们去学更有用。因为如果你能向孩子们展示科技是一件非常有趣好玩的事物，而不仅仅是爸爸妈妈整天坐在电脑前办公——也就是真正向孩子们展现科技到底能创造出什么——他们就会爱上它。这也正是我们在这些青少年工坊中竭力追求的目标。
Original English
**Cassandra**: But even before I started teaching in my household, there was lots of technology. We had 3D printers, a lot of different technology. I remember building my own computers. I really believe in inspiring kids to enjoy technology rather than pushing them. Because if you show kids that technology is this very fun thing, not just dad or mom at the computer, like show the kids what technology can actually do. That's what we really tried to do in these kids workshops.
**Cassandra**：在这次周末的工坊中，我负责教授一门指导孩子们如何使用 AI 进行编程的课程。孩子们使用的是 Mistral Vibe 来构建属于他们自己的游戏。孩子们展现出来的能力真的让我大吃一惊。成年人往往对“如何编程”、什么能做什么不能做有着根深蒂固的既定成见，但孩子们没有，他们脑子里全是天马行空的想法。
Original English
**Cassandra**: And for the kids workshops this weekend, I taught a workshop teaching kids how to code with AI. The kids used Mistral Vibe to build their own games. And I was really surprised at what the kids could do because we have these preconceived notions about how to code, what you can or can't do, but the kids, they just have ideas.
**Cassandra**：他们提出的一些构想非常精彩，比如构建角色扮演游戏（RPG）、平台跳跃游戏，甚至还有香蕉主题的游戏。孩子们迸发出无数极具创意的点子，而他们只需要输入提示词，AI 就能替他们把游戏搭建出来。
Original English
**Cassandra**: Some ideas they have are like building an RPG game, a platformer, a banana game. A lot of creative ideas from kids, but they just prompt and the AI could build it for them.
**Cassandra**：另外，我们合作了一个名为 AI Tinkers 的组织，他们为工坊带来了很多弱势群体的儿童。这些孩子学习得非常快、非常好，无论他们有着怎样的背景。而且我认为非常棒的一点是，他们还吸引了许多女孩子参与。科技领域的另一个严峻问题在于存在着显著的性别偏见。随着孩子们逐渐长大，他们会开始认为“科技是男孩子的事情”，因为行业里的大多数现实给他们留下了这种印象。但如果你在他们年纪还很小的时候就介入引导，他们脑海中根本不会有“科技是男生专属”的偏见，很多女孩也能够像我一样受到启发并投身科技。因此，我认为这种青少年工坊对孩子们非常有益。
Original English
**Cassandra**: And we had an organization called AI Tinkers, they brought in a lot of underprivileged kids to the workshop. And these kids, they could learn really well, like it didn't matter what background they have. And I think it was really great that they also brought in a lot of girls. Another issue with technology is there's a lot of gender biases. As kids get older and older, they start thinking that technology is a guy's thing because they see more of the industry. But if you catch kids when they're very young, they don't have this idea that technology is a guy's thing. And a lot of girls can be inspired to join technology like I have. So I think these kids workshops are really great for kids.
**Cassandra**：在我带的工坊里，我们让孩子们制作游戏，并且实际上将整个环境连接到了一个 GitHub 仓库，其中预置了大量的艺术素材以及一个 `agents.md` 规则配置文件。这样便为孩子们使用 AI 构建了一个更加安全可控的环境。因为大家通常最大的顾虑之一就是：放手让小孩子使用 AI 真的安全吗？通过这个 `agents.md` 文件，我们能够明确指示 AI 表现得更加亲切友善，少用深奥的技术术语，更具协助性，主动执行命令而不是要求孩子们自己去跑复杂的终端命令。通过大量这样的指令设定，整个工坊环境变得更加适合儿童使用。我们专门为这次青少年工坊制作了一段回顾视频，请大家观看一下，内容非常精彩。
Original English
**Cassandra**: The workshop which I taught, we had the kids build the games and we actually connected it to a GitHub repo which had a lot of art assets and a agents.md file. So this created a more kids safe way with AI because a huge concern is how is it safe to let kids use AI? But with this agents.md file, we were able to instruct the AI to act more friendly, use less tech jargon, and be more helpful, run commands rather than asking you to run commands. A lot of instructions which make the workshops more kids-friendly. We have this video which we produced about the kids workshop. So if you would please watch it, I think it's really nice.
### 工坊现场纪实：释放孩子们的创造潜能
**Neo4j 讲师（现场视频）**：我们现在身处巴黎的 AI Engineer 青少年日现场，正在向当地的孩子们传授 AI 技术知识，带他们了解模型与神经网络。这是以一种充满趣味且平易近人的方式让孩子们接触人工智能的绝佳途径，也是我们 Neo4j 非常热爱践行的事情，因为我们正在帮助培养下一代 AI 工程师。
Original English
**Neo4j Instructor (Video)**: We're here at AI Engineer Kids Day in Paris teaching the local kids about AI technology, about models, about neural networks. This is a great way to expose kids to AI in a fun way, in an approachable way, and something we love doing at Neo4j because we're helping train the next generation of AI engineers.
**Neo4j 讲师（现场视频）**：现场聚集了来自各种不同背景的孩子。他们当中有的才刚满 8 岁就已经是熟练的代码好手，有的则是十几岁的少年、需要多一点指导和协助。但最令我感到兴奋的，是我们把他们聚在同一个空间里。这里形成了点对点的互助学习：大家相互传授、彼此支持，这样当他们长大并进入这个行业工作时，就能真正掌控自己的生活、职业与思想。
Original English
**Neo4j Instructor (Video)**: There's a whole mix of kids here from a bunch of different backgrounds. Some of them are already expert coders at 8 years old and some of them are in their teens and they need a little bit extra help. But that's kind of what excites me is we're putting them all in one room. There's peer-to-peer learning. They're teaching each other and they're helping each other so that when they grow up and they start working in the industry, they'll have a handle on their own lives, their own careers, their own ideas.
**Cassandra（现场视频）**：在我的工坊中，孩子们使用的是 Mistral Vibe 编码智能体，它能让孩子们创建出专属的游戏。孩子们所做出的游戏种类五花八门，十分有趣。有的孩子选择制作 RPG 游戏，有的做横版闯关游戏，还有个孩子做了一款扎气球的小游戏。格外值得一提的是，这里完全没有语言隔阂。尽管孩子们主要说法语，他们直接用法语向智能体输入要求，智能体完全能理解法语，并顺畅地为他们编写出完整的游戏代码。因此我认为这对孩子们是一次非常宝贵的体验，因为他们亲手打造出了独一无二的作品。
Original English
**Cassandra (Video)**: For my workshop, the kids use Mistral Vibe and it is a coding agent which lets them create their own games. It's really interesting what kind of games the kids build. Some of them choose to build an RPG game. Some do platformers. Another kid did a balloon popping game. And what's interesting is there's no language barrier. Even though the kids mainly speak French, they can type French to the agent and it fully understands French and still codes the games for them. So I think this was a really good experience for the kids because they got to build their own unique games.
**AI Tinkers 组织者（现场视频）**：我们 AI Tinkers 组织筹办过很多场活动，而举办一场能把孩子们和他们的家庭成员聚在一起的活动——我们日常社区成员在这里可能扮演父母、哥哥或姐姐的角色——听起来格外令人振奋。大家能够聚在一起共同动手创造，我认为这就是社区建设不可或缺的核心部分。
Original English
**AI Tinkers Organizer (Video)**: So my organization AI Tinkers organizes a lot of events, and having an event that will bring together the kids and their family members — so our usual community members that could be the parent or the big brother or big sister — that sounded really exciting so that they can build together, and I believe that's part of community building.
**Cassandra**：这正是我们对此项活动充满热忱的原因所在。我们将这些青少年工坊作为 AI Engineer 系列大会的一部分持续开展。接下来，我们将在 AI Engineer 纽约站以及旧金山的 AI Engineer Code 大会上继续举办青少年工坊，并在明年带到伦敦站。如果各位届时会参加这些活动，请一定把你们的孩子带过来，我们非常欢迎小朋友的加入，并将悉心指导他们。
Original English
**Cassandra**: And there's a reason why we were really excited about doing this. We are running these kids workshops as a part of the AI Engineer conference series. So we will be running the next workshop at AI Engineer New York and also AI Engineer Code in San Francisco, and next year we'll be running it at London. So if you're at those events, please bring your kids. We'll welcome your kids and teach them.
### 主持人致谢与赞助商致意
**主持人**：谢谢！非常感谢 Cassandra，让我们再次为她热烈鼓掌！太精彩了。是的，感谢 Cassandra，也感谢 Neo4j 团队付出如此出色的工作。我自己也是一名父亲，对这些深有感触，这项工作真的极其重要。再次向你们表示感谢。
Original English
**Host**: Thank you. Yeah. Thank you, Cassandra. Let's give it up one more time. That was awesome. Yeah. Thank you, Cassandra. Thank you, Neo4j, for doing such a great job. I'm a father myself and I can relate. This is super important work. Um, yeah, so thank you one more time.
**主持人**：现在，也让我们把掌声献给台下的各位！谢谢大家，你们一路坚持到了最后，对吧？我知道早些时候有人跟我抱怨说：“嘿，你带头鼓掌太多次了，我手拍得都疼了。”我就说：“老兄……”我们得找个——噢，你就在那儿坐着呢，自己对号入座了是吧，太逗了。
Original English
**Host**: And let's hear it from you guys as well. Thank you. You made it. Hey, you made it all the way here, right? I know somebody told me earlier like, "Hey, you clap too much. My hands are hurting." I'm like, "Hey, dude." Like we gotta find a... Oh, you're there. You recognize yourself. Nice. Yeah.
**主持人**：非常感谢大家与我们共度这段时光。我还要由衷感谢协助我们组织这次盛会的所有工作人员，对吧？我还要特别鸣谢我们的赞助商：首先是主办并促成本次大会的 Mistral，以及作为我们白金赞助商的英伟达（NVIDIA），还有我们所有的金牌、银牌和铜牌赞助商。请大家为赞助商们热烈鼓掌！
Original English
**Host**: But thank you so much for being with us and I would like to thank everybody that helped us organize this event, right? I would like to thank our sponsors, Mistral first for organizing the event and then Nvidia as our platinum sponsor, but also all our gold and bronze and silver sponsors. So, let's give it up for them, please.
**主持人**：同时，我也认为大家应该为自己感到自豪。我听说你们当中有许多人不远万里从世界各地赶来参会。我们听说有来自法属波利尼西亚的朋友，还有专程从澳大利亚飞来的，以及来自欧洲各地的朋友们。看到大家从全球各个角落汇聚在这里与我们相伴，真的太酷了，我们深表感激。来吧，把掌声送给你们自己！你们表现得太棒了。
Original English
**Host**: And also I think you should be proud of yourselves. So I heard that many of you have traveled all the way here from different places. We heard somebody from French Polynesia and then Australia to live and different parts of Europe as well. I think this is so cool that people come from different parts of the world to come and be with us here and we really appreciate it. So yeah. So let's hear it from you. Let's give it up for you. I think you did a good job.
**主持人**：好了，在正式闭幕之前，首先我由衷希望明年还能见到大家。不过在散场前，我们还有最后一项环节，请出来自赞助商 Backblaze 的 Patrick 上台发言。掌声有请 Patrick！
Original English
**Host**: Okay, so before we close it out, first of all, I hope I see you next year, but we have one more thing from Patrick and our sponsor, Backblaze. So, yeah, let's hear it for Patrick. All right.
### Backblaze：专为 AI 数据打造的高容量存储
**Patrick**：好的，各位。首先我向大家保证，我的发言绝对控制在两分钟以内。另外，让我们为主持人刚才拍了那么多次手鼓掌吧——因为你们还得再多拍一次。能做到吗？好的！
Original English
**Patrick**: All right, guys. The first thing I'm going to promise you is this will be exactly two minutes. And let's give it up for how much he claps, because you got to clap one more time. Can you do it? All right.
**Patrick**：大家好，我是来自 Backblaze 的 Patrick，很高兴与大家在此相聚。今晚由我请大家喝酒！因此按照合同约定，我必须向各位讲清楚 Backblaze 究竟是做什么的。我会尽量用最简明扼要的语言来说明。
Original English
**Patrick**: So, I'm Patrick from Backblaze. Great to meet you all. I am buying you drinks tonight. So I am contractually obligated to explain to you what is Backblaze. I'm going to do it as simply as I can.
**Patrick**：我留一句话给各位记住就好：AI 运行在 GPU 之上，而 GPU 极度渴求数据；Backblaze 就是最适合 AI 数据的高性价比海量存储层。如果各位今天能记住什么，记住这句话就足够了。
Original English
**Patrick**: I'll give you one sentence to remember. AI runs on GPUs. GPUs need data. Backblaze is the best capacity storage tier for AI data. So, if you remember anything, remember that.
**Patrick**：这里我给大家举两个具体的客户案例。第一个是世界模型构建商 Descartes。在他们刚刚起步拓展时，Dean 给我打来电话说：“嘿，我有 15 PB 的数据，我需要在几天之内完成迁移，而且至今还没有把你们的系统打挂，你们太神了！”我当时心想：“太牛了，不过你们是谁？具体在做什么？我其实一无所知。”然而他们之所以坚定选择 Backblaze，是因为我们不收取任何出网流量费（Egress Fees），并且拥有令人惊叹的高吞吐性能。这使得他们能够在不同的 GPU 集群之间快速流转数据，进而实现极致高效的 AI 训练运行。
Original English
**Patrick**: Two examples. Descartes, the world model builder. When they were getting their start, Dean called me up and he's like, "Hey, I've got 15 petabytes and I'm moving it in days and I haven't broken you guys yet. You're awesome." And I was like, "Awesome. Who are you? What are you doing? I have no idea." But they chose Backblaze because we don't charge egress and we have really incredible throughput. So they were able to race different GPU clusters and create incredibly efficient AI training runs.
**Patrick**：另一个客户是 CoreWeave，我们是他们底层采用的对象存储层。他们之所以选择我们，是因为他们做了一轮非常详尽严苛的概念验证（PoC）。他们发现，从 EB 级往上扩展，我们的经济效益（成本曲线）与性能表现始终保持稳定优异、毫不衰减。因此他们最终选择了我们，因为他们坚信我们能够伴随他们的业务规模无上限地持续扩展。所以，无论你正在处理的数据体量是 TB 级还是 EB 级，我们都是极其出色的存储解决方案，能够确保你的数据存放在一个完全受你掌控、任由你按需使用的地方。
Original English
**Patrick**: Another customer, CoreWeave, we are their object storage tier. They chose us because they ran a very exhaustive PoC and what they found was that from exabytes upward, our economics and our performance did not change and so they selected us because they believed we could scale with them forever. So whether you're working with terabytes or exabytes, we are a great solution to ensure that your data is someplace where you control it, you can use it how you want.
### S3 兼容存储与数据自由的使命
**演讲者**：你可以根据自己的意愿与任何对象配合使用。作为一个与 S3 完全兼容的存储层，我们的费用只有 S3 的四分之一，并且免收数据出流量费（free egress），因此你在存储方面的支出将会大幅降低。正如我刚才所说，我们的使命是让你可以随心所欲、在任何地点、与任何人一起使用你的数据——我们目前正在为众多生成式 AI 企业、新兴云服务商（neo-clouds）、数据抓取团队以及物理 AI 公司践行这一使命。我们将一如既往地专注于存储本身，并确保在你需要时，每一个字节都能以所需的最快速度提供到位。我们的核心目标是让你在存储上花费尽可能少的资金，因为在过去几天听了现场的所有演讲之后，我深感各位正在做的事情极其引人入胜，存储理应是你最不需要操心的事情。
Original English
**Speaker**: ...to you can use it with who you want to. And as an S3-compatible storage layer that is a quarter the cost of S3 and has free egress, you're going to pay a lot less to do it. Like I said, our mission—which we execute for lots of genai companies and neo clouds and data scrapers and physical AI companies—is use your data however you want, wherever you want, with whoever you want. We are only going to focus on storage and we're going to make sure every byte is there as fast as you need it. And our primary goal is that you spend as little money as possible on storage, because after listening to all the talks here for the last couple of days, what you guys are doing is so interesting and storage should be the last of your concerns.
### 展区交流、特色特调与初创扶持计划
**演讲者**：所以请务必关注并了解我们。稍后我会在楼上。我们准备了一款招牌特调鸡尾酒，名叫“Lebe Blaze”——非常原创！大家不妨去尝一杯，顺便和我聊聊。另外，如果你目前正处于早期创业阶段，我们还设立了初创企业计划，提供高达 100,000 美元的存储抵扣额度。如果你会算这笔账的话就会明白，以我们仅为 S3 四分之一的成本，这笔额度根据你们业务的增长速度，足以支撑相当长的一段时间。
Original English
**Speaker**: So definitely check us out. I will be upstairs. We have a signature cocktail, Lebe Blaze. Very original. Try one, talk to me. And if you are early stage, we also have a startup program where we offer up to $100,000 in storage credits, which if you can do the math, we're a quarter of the cost of S3 is going to last a good amount of time depending on how fast you're growing.
### 致谢与大会总结
**演讲者**：非常感谢大家。再次感谢 AI Engineer 大会、Mistral 以及所有的赞助商，尤其要由衷感谢在座的各位。我两天前的晚上专程从明尼苏达飞抵这里，明天一早就要乘机返程。我现在非常疲惫，还得回去照顾我的孩子们。我深知创业与技术探索有多么艰难，但我认为这一趟绝对完全值得。非常感谢各位，我们稍后楼上见！
Original English
**Speaker**: So, thank you everyone. Thank you again to AI Engineer and Mistral and all of the sponsors and especially thanks to you guys. I flew here from Minnesota two nights ago. I'm flying back tomorrow morning. I'm exhausted. I have to go take care of my kids. I know how hard this is, but I think it was totally worth it. So, thank you, and we'll see you upstairs.