为什么不能在没有评估的情况下交付 Agent 技能？
在生产环境中，智能体技能（Agent Skills）的检查失败是极其常见的现象。根据对 SkillBench（一个索引了来自 GitHub 等渠道超过 50,000 个技能的流行评估基准）的分析发现，几乎没有任何技能配有对应的评估。大多数技能都是由 AI 自动编写的，且未经过真实测试。由于智能体本身具有非确定性，如果任务失败，你很难界定这究竟是因为技能本身编写得很差，还是因为任务对模型而言过于挑战。
因此，在深入讨论之前，我们需要厘清“我们使用的智能体”与“我们构建的智能体”之间的本质区别。像 Antigravity、Cursor 和 Claude Code 这样供我们日常使用的开发智能体，使用者本身就是工程师。当智能体第一次未能成功调用某项技能时，工程师能迅速察觉并终止任务，进行重新提示或使用斜杠命令手动触发。然而，当我们为最终消费者或客户构建智能体时，他们对“技能”毫无概念。用户不会在提示词开头写上“请使用退款技能来帮我解决问题”，他们只会直接描述诉求。这种使用场景的巨大差异，使得评估机制在构建面向用户的智能体时变得不可或缺。
Original English Source
Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents. And we are going to talk about why you should not ship skills without eval. And maybe before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code. So yeah, hopefully every every hand goes up, right? And do you use skills with it? Okay. Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today. And like very important is like why checks fail in productions. And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up and like try to look into him and almost none of those skills had evals. Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic. So you might not know if your uh task fails because your skill is bad or if your task fails because it's way too challenging for the model. So um very important um before we go into it there's I want to like really make sure that we know the difference between the agents we use and the agents we build. Um most of us use agents for writing code, doing productivity work. That's the agents we use. It's like anti-gravity, cursor, Claude code. And there are you are the engineer and you have context about skills, right? If you write some prompt to I don't know, like help me build a new Gemini API feature. And if your agent does not invoke the skill on the first time, you will notice it very quickly. You stop your your task and reprompt it or like use slash commands for for triggering those skills. When you build an agent inside your application for consumer or customers, they have no idea about what a skill is. They don't start their prompt with use customer support skill to like help me refund or use refund skill to help me solve my problem. So, there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there.技能的分类：能力型技能与偏好型技能
智能体技能（Agent Skill）本质上是一个包含 skills.md 文件的文件夹，辅以让该技能正常运行所需的其他资产。技能的设计遵循渐进式披露（Progressive Disclosure）原则：通常以极小的体积开始，包含标题和描述（Description），其中描述会被注入模型的上下文，让模型知道何时使用该技能；第二层是技能主体（Skills Body），包含更详细的指令和对外部文件的引用，模型可以通过探索这些外部文件获取解决任务所需的完整上下文。
在实际应用中，我们可以将技能分为两类：
- 能力型技能（Capability Skills）：用于教导模型目前无法稳定完成的事情（例如分析特定日志、创建新的 React 应用）。这类技能通常是临时的，随着基座模型能力的提升，我们可以逐渐废弃它们。评估机制会清晰地告诉我们何时可以安全地退役这些技能。
- 偏好型技能（Preference Skills）：更加持久，主要用于编码特定引用和规范。例如，团队特定的工作流、公司的代码风格指南或其他特定领域的偏好。由于通用基座模型很难融入这些非常特定领域的知识，因此偏好型技能极具价值，必须通过评估来防止智能体在迭代时发生性能退化。根据 SkillBench 1.1 的数据，合理使用技能平均能将模型的任务表现提升约 15%。
同时，评估研究也表明：人类编写的技能文件效果最好，而完全由 AI 生成的技能有时反而会对性能产生负面影响。此外，skills.md 文件的长度应当控制在 500字/行 以内。如果你的技能文件过于臃肿，它在被加载到上下文时会消耗大量的 Token 成本，影响模型的推理效率。
Original English Source
And what is a skill? I mean, every one of us knows hopefully in by now what a skill is. It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work. And the big difference with skills is that they work on progressive disclosure. So, most of the skills start very small. So, you have the title and a description. The description is normally part of the model's context. So, the model knows when to use the skill. Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files. And then you can really go deep in those reference files where there's all of the context the model needs to discover to to solve the task. And I like to differentiate between two kinds of skills. So, they are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment. Maybe it's like, I don't know, like tracing some logs, creating a new React app. And those capability skills are temporary. So, the better our model gets, the more likely it is that we can remove those skills. And Evals will tell us when we can retire skill and when not. And then we have preference skills. Those are more durable, mostly encode some references. So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, Um will have or create preference skills and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain. And preference skills are very valuable, so we really want to make sure that those are working and we don't like update our agents to uh degrade performance. So, do skills work? Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses showing that skills on average improve the performance by roughly 15%. Skills bench covers around 100 different tasks uh based on like coding and also productivity across different languages. It's uh openly available and they have a very nice website, a very nice leaderboard, are also very open for uh community contributions. And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a skill.md file. We maybe look at it very closely. It roughly covers what we want to do and then we just accept it and start using it. And what I found out is that human-written skills are the best we can provide. Uh AI-generated skills can uh impact performance negatively. And that skills or skills.md files should be below 500 lines of words. So, if you have your laptop open and have a skill available, if you open that and if it's above 500 lines, you should definitely look at the skill after our session. And um the last topic about what is a skill and how a skill works, uh we have different ways of triggering our skill, right? We can have a model-triggered skill meaning uh based on the context and the description, the model decides to use or read a skill to, uh, get more context to solve a task. And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um, and they most of the time just accept, uh, the overhead by pro- like adding it into the context. I have like many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all like of the very uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill. And when you build agents for customer, you don't have those user-invoked skills. We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second.高质量技能编写的黄金法则
在编写模型触发的技能时，技能的描述（Description）至关重要。因为这通常是系统指令中仅有的两句话，直接决定了模型何时触发或不触发该技能。如果描述写得太宽泛或太弱，模型可能会过度触发，或者在真正需要时却没有触发。
以下是编写高质量技能的核心建议：
- 使用祈使指令而非陈述句：不要写“Interactions API 被推荐用于处理多会话聊天状态”，而应直接命令模型：“如果正在构建聊天应用，请使用 Interactions API”。给模型非常清晰、明确的动作指示。
- 精简化与分层设计：技能描述是每次模型调用时都会消耗的固定成本（约 100-200 个 Token）。因此，描述应保持精简，而将具体的实现细节、部署指南（如 AWS 与 GCP 的不同部署步骤）提取到外部参考文件中，让模型在需要时自行读取。
- 显式定义负面用例（Negative Cases）：我们常常只告诉模型何时使用该技能，却忽略了何时不应该使用。例如，若某技能只适用于 React 组件，就必须显式指出“不要在 Angular 组件中使用”，以防模型在无关的 Web 开发环境中过度触发。
- 为脚本定义目标与约束，而非死板步骤：不要在技能中写“第一步去这里，第二步做这个”。如果步骤是完全固定的，你应该写一个脚本，并指引模型直接执行该脚本。技能应该用来定义模型需要达到的目标和需要遵守的约束，把具体的执行路径留给模型自行发挥。
- 消灭无意义操作（No-ops）：AI 生成的技能往往包含很多无意义的废话指令（例如“在实现前确保代码易读”、“编写高质量的代码”）。这些指令并不能真正改变智能体的行为，却在不断消耗 Token 成本。应该及时清理这些 No-ops。
Original English Source
So, writing skills, um, is an important topic. Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills. And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not. And it's bad if your description is too weak because then it might trigger too often or the it might not be triggered if you need it. So, very important is the the why and the how for the model. So, why it should use that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example. And then, of course, the when. And we should write directives instead of essays. So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles like session state and it's like where you should be way more directive like use the Interactions API if you're working on like a chat application. So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill. And similar to what we have seen in the skills bench results, we should keep the skill lean and layer information. So, the description is the cost you always pay on every model invocation. So, on every model call, the description is part of the model context. So, you always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that. When you have a very long skill MD file, it will be always read into context when the model decides to read the the skill or to use the skill. Uh which can be expensive as well. That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task. And then of course, the layer three is like we can have those reference files where the model needs to like um go really deep into a very specific task. And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud. Those should not be part of your skill MD file. Those should be references. That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the model can basically explore based on the context where it should go to get all of that information. Then we should set the right level of freedom. Um I see many people very clearly describing the exact workflow in a skill. Step one, go there. Step two, do this. Step three, do this. If you have those type of use cases, you should not use skills. You maybe you should write a script because if the the process or the workflow is always the same, you don't need to waste models and tokens for that exercise. You can create a script. You can tell the model use that script to run a specific workflow. So, rather define goals and constraints. So, if you need to like deploy to your update or stage your documentation, describe how the model can do that. Or like for your database updating a config, you should not say like read the config, update the port, and then like deploy again. The model knows what to do. Just like hey, if we need to change the config, here's the file, make the change. Then uh don't skip negative cases. So, we always look at the when we want to use this skill, but most of the time we don't look at when we don't want to use this skill. So, if we have a description for our skill which says use it for web development tasks, it might over trigger. Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that skill for React components or for Tailwind CSS, then the model knows hey, that's very specific for one to use. And with Evals, we can also identify those. Um And then test early. So, that's what we are going to look at. We should really try to test when you create a new skill. Always try to create 10 of 20 prompts. I like to create five for like the happy path. So, when do I want to use that skill? Five when I don't want to use that skill just to make sure the model is not over triggering the skill and confusing itself. And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data. And then tip seven which is quite new and I have to give all credits to Matt. So, if you don't know Matt, he's a great AI educator and you should definitely follow him. He published a tweet and also a skill on like killing all of the no-ops. And what he found is that AI generated skills tend to include a lot of no-ops. And no-ops basically is an instruction which does nothing to change the agent's behavior. It's like before making an implementation easy to read. Like the model knows how it when it should make something easy to read or write clear high-quality code. I mean, like that's what we expect from the model to do without telling it really. So, um definitely look at those no-ops. He have has published a very good skill in in his like skills repository. Uh and then last but not least, uh know when you should retire skill. Um skills are not there to live forever. Models get better, behaviors change, expectation change, um the environment changes. So, um always try to run evals with and without the skill enabled. And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost uh for your tokens, and then also um don't keep like it redundant. So, save cost at the end and maintenance also as well.DeepMind 的实践：如何构建智能体技能评估系统
在今年早些时候，DeepMind 团队需要为新发布的 Gemini Interactions API（用于与 Gemini 模型及智能体交互的新接口）创建一个新技能。由于该 API 是在模型训练截止日期之后发布的，因此 Gemini 3.0、3.1 甚至 3.5 对其完全没有任何原生概念。
为了帮助模型利用 Interactions API 生成有效的代码，团队构建了一套评估流程：
- 测试用例（Test Cases）设计：收集并设计了 117 个测试用例。这些数据来自于尝试使用 Interactions API 的真实用户数据、合成生成的测试用例，以及用户反馈（例如模型在已有新版本时仍倾向于使用旧版 API 模式）。
- 测试集资产结构：主要由两部分组成：
- 一个存储测试用例的 JSON/YAML 文件：包含输入提示词（Prompt）、所用语言（TypeScript/Python）、一个布尔值
should_trigger（指示智能体是否应当读取并触发该技能），以及预期检查项（Expected Checks）。 - 一个运行智能体（此案例中为 Gemini CLI）的 Python 脚本：负责执行任务、捕获输出，并比对检查项。
- 一个存储测试用例的 JSON/YAML 文件：包含输入提示词（Prompt）、所用语言（TypeScript/Python）、一个布尔值
- 基于正则的高效断言：大多数技能评估可以直接通过正则表达式（Regex）来完成。团队利用智能体编写了高质量的正则断言，用于检查：是否导入了正确的 SDK、是否使用了正确的模型 ID、是否调用了正确的方法，以及是否避开了已被废弃的旧代码模式。这种方式运行成本极低，可以频繁重复运行。
- 大模型裁判（LLM-as-a-Judge）：对于涉及复杂执行轨迹或多步操作的技能，如果无法简单通过正则断言，则使用大模型作为裁判，基于设定的评分细则（Rubric）对智能体的输出进行 Pass/Fail 的判定。
在 Google DeepMind 内部，所有技能文件旁都会附带对应的评估用例。每当技能文件（如提示词修改）发生代码变更（Diff）时，CI 流程都会自动触发评估。如果变更没有提升测试用例的通过率，或者导致了回归，该 PR 将不被允许合并。
最后，Philipp 建议大家在下周一回到工作岗位时，挑选自己最常使用的一个技能，为其编写 5 到 10 个测试提示词（5 个正向，5 个负向），运行一次消融测试（Ablation Test，即对比加载技能与不加载技能时的表现）。如果模型不加载技能也能达到同等甚至更好的效果，请果断退役该技能，以节省维护精力和 Token 成本。