评测的两个认知误区
很多人在开发AI系统时常常感到孤立无援，但我今天的核心主张是：人们对评测（Evals）的理解往往是错误的。目前在行业内，关于评测存在两个极端的认知阵营：
第一个是客观指标派（Objective Metrics Camp）。这一派倾向于“刷榜（Benchmark Maxing）”，他们看着仪表盘上的分数，就天真地认为具有相似高分的模型（例如某些号称媲美前沿模型的测试版本）在能力上是等同的。但一旦将这些模型置于真实的实际测试中，你就会发现它们并不一样，表面的高分往往经不起现实世界证据的检验。
第二个是纯凭感觉派（Taste Camp）。这一派走向了另一个极端，他们认为一切都只关乎“感觉（Vibes）”，甚至会像评价歌剧一样去拟人化地评价大语言模型（例如说“我喜欢和她聊天”）。
事实上，真相介于两者之间。评测既不是衡量一切的终极标准，也并非毫无用处。我们需要掌握正确的方法去使用它们。在深入构建自定义评测之前，我们首先需要建立解读评测的直觉。
Original English Source
All right. All right. First of all, thank you so much for for coming. I'm actually rather surprised. Um A lot of times like you're you're working on the stuff and you're like cooked up in a room and you're like no one cares. And then it's like so many people showed up. So I just thought that someone cares. Um so anyway, so the the title of my talk today is evals are broken and you should use them anyway. And a lot of this talk is just like a straight-up critique of like the way we do evals these days. And I kind of want to help you out. I kind of want to give you a way out of this. It's like you have this like interesting technology and you can use it, but there's like so many ways to like mess it up. So I want to help you out. So my first claim is that people are wrong about evals. Actually, let me let me correct myself. Most people are wrong about evals. And I want you to be right about evals. I want you to I want you to use them. I want you to like I want you to be able to build with them, interpret them, use evals in your own agentic flows, um leverage them in any way sense that you can. Um so that's that's that's basically the point of the conversation. So to be right to be right about something that has like a lot of nuances that can go in like many different directions, the final question is like how are people wrong about that thing, right? So there's basically two camps of people who are wrong about things, right? So there's two camps of wrong on evals. The first camp is a camp of objective metrics. So the objective metrics camp is this. Like there are people who would like look at a list this dashboard, right? And they will interpret this as something akin to like like it means something as in like GPT 5.4 is effectively the same as like Gemini 3.1 Pro preview. Believe me, they're not the same. Uh there's a lot of these models which feel like show up like with similar numbers and these numbers at a certain point it's like this is the whole thing is a hoax. Like you you won't believe it at all. So, there was this tweet this came out like just this morning and it was a critique of meta where meta came out and it was just like classic benchmark maxing. Just like we're doing best on the benchmark. Everything's great. And I I assure you if you try a lot of these models like it just just won't won't hold the hold the test of like actual real-world evidence. The other one the other camp the other way where people are wrong is that they could too far the other way. So, this is basically like the taste camp. Um So, people in the taste camp are kind of like this. They just like they're they're an archetype. They're they just they think that it's like it's everything's about like vibes and everything's about like you know, like it's like if you ask them like why do you like operas and they'll say things like I like talking to her. Like they'll they'll like anthropomorphize it. And um and this is also not the right either, right? So, I think the truth is somewhere in the middle that the emails are not the end all be all. They're also not completely useless. There are right ways to use them. There are wrong ways to use them.解读评测的三大启发式原则
要想有效利用评测，我们可以将其分为三个阶段：借用他人的评测、利用评测改进自己的智能体（Agent），以及针对特定场景构建自定义评测。为了帮助大家建立正确的判断框架，我总结了三个启发式原则：
- 对分数保持警惕：当模型厂商发布带有高分数据的应用时，不要轻易相信。很多AI研究人员和工程师已经习惯性地忽略这些数字，因为它们通常只是一种近似值。
- 避免盲目追新：保持对前沿技术的关注，但不要总是做最早的采用者（Earliest Adopter）。观察 Epoch Index 就会发现，前沿模型每隔几个月甚至更快就会更替一次。更好的策略是让新模型先沉淀几周，如果它经历了时间的考验，再去进行模型切换，而不是疲于奔命地追赶测试。
- 寻找最新且精准的评测：许多标准化的评测实际上已经过时了。OpenAI 曾在博客中明确指出，类似 SWE-bench 这样的旧指标已经无法衡量前沿模型的编码能力。因为它们还在测试如何解答斐波那契数列或进行矩阵乘法，而这根本无法代表真实的软件工程问题。你需要的是具备极高分辨力的新型评测体系。
Original English Source
Um so, in order to do that I'll give you like three stages that will help you like uh use them really well. So, the first stage is like you can like leverage emails from other people. The second stage is to like use emails to improve your own agents. And then the third stage is like build actually build your own emails for specific use cases. In the interest of time, I could personally talk about emails for hours, but like I can only talk about level one and two and I think those would be most helpful for most people in the audience. Um so, I'm going to give you like a few heuristics uh to interpret email. So, the first heuristic is that whenever a model apps comes out with like a number, just don't believe them. Just don't. Just like these are approximations. Coming back to the tweet like just like just don't believe the model app eval numbers. Like there's somewhat of approximation. Sometimes they're good, sometimes they're not. Um there's this tweet that's like pretty cool one where Nikunj said that like a lot of like AI researchers and engineers routinely dismiss evals. They don't really They don't really think of it as like something that's like the numbers to be taken that seriously. And it to some extent it's a matter of like actual trying and preferences. And I think I think that is like somewhat more accurate. Um Wait a second. Yeah. So the second heuristic is that like you want to stay current, but you don't want to be the earliest adopter. And why am I saying this? So if this is like Epochs Index is basically like the aggregate score of like different models on like evals. And if you notice like in the last like 2 years, every single couple months like the frontier lab the frontier model is changing. And it's changing so fast. Like it's just like it's so hard to keep up with this stuff. And I work for I work on this. I've been doing this for a living for years at this point. And I I even even me I have like preferences changing so fast. So like I think that like when when you're when you're working through these things like the way I would recommend is that like let the thing come out first. Let things settle on fire for like a couple weeks. And then if the if the thing still stands the test of time, I think at that point you should like do your model switch and like try something rather than like always trying to be on the cutting edge. Like the the people who have to always try the new model and try the new things like they'll be me. But like I do this for a living and you don't have to. Um the third the third heuristic for evals is that like you need to look for very very new and very precise evals. And the reason this is necessary is that like a lot of evals are like at this point they've become like standardized. They're actually kind of old. Like they're they're not useful for you. So like this is a this is a blog post from OpenAI where they straight up said as we bench verified no longer measures frontier coding capabilities. Um, and I think like if to a lot of people in the AI research community, that was like very obvious. It was very obvious that as of the leave edge doesn't measure frontier coding capabilities because it had like it would have problems like solve the Fibonacci sequence. It would have problems like, you know, do matrix multiplication or something. And it's just like it doesn't apply to like real world software engineering. So, you want to have something like like very new, but also like actually legit. Uh, and then it takes some discernment to figure that out.构建真实的工程环境评测
当我们利用评测来改进智能体时，必须将其视为一个工程与哲学的双重难题。在工程层面上，环境隔离极其困难；在哲学层面上，你需要构建一种能够近似反映真实世界无限问题空间的测试框架。
在 Cline（一家开源的编程智能体公司），我们发现过去市面上的单轮评测（Single-turn Eval）过于初级，就像在问大语言模型“猫有几个脚趾”并期待一个二元答案。但当你评估一个复杂的智能体时，任务往往是“帮我修复这个未知 MCP Server 的错误”。智能体会去阅读文件、搜索文档、配置环境、运行 Python 脚本并执行测试，这一连串动作可能需要长达数十分钟。单纯看最终结果，很难衡量它是否在此过程中破坏了其他代码。
为了解决这个问题，类似 Terminal Bench 这样的工具应运而生。它包含了89个真实的编程任务（如竞态条件、数据库问题、基础设施问题）。为了安全有效地运行它，你必须为每一个测试任务提供一个完全隔离的虚拟机环境，在其中加载整个代码库并运行你的智能体（如 Cline、Claude Code 或 Codex）。为了不被最慢的任务拖垮进度，你可以利用像 Harbor 这样的开源软件配置标准化的底层基础设施，并借由 Modal 这样的算力平台，将这89个真实环境隔离的复杂任务并行跑完。
Original English Source
So, that's like the first part. But the second part is like, okay, now that we know that like, okay, there we have a few heuristics of like how to like use evals. Like, how do you use evals to improve your agent upon them? And I think this is a part where I kind of lean into like the the core philosophy of this conversation where like you want to think of evals as like an engineering problem, but also as a philosophy problem, right? So, the engineering problem is obviously hard, but the philosophy problem is also very hard. Uh, the philosophy problem is that you you want to you want to you have a problem and you can't exactly approximate the the search space of like where where the problem could go where the problems could fail. Um, it's it's sort of somewhat easier-ish to do it for coding problems. But even then, coding problems have like an infinite search space. They can go in any direction. So, you want to you want to build evals that like are somewhat more approximate representation of the actual thing that you're dealing with. And for us, like to give some context in Cointreau, so I work at Client. Client is an open source coding is a company. We have a we we have a very interesting product. I encourage you to try it out. So, in Client's journey, one of the things that we dealt with is like in the last year, one of the things we found is that like there were like a few evals available. At the time, it was like we were very rudimentary. Every every other company was very rudimentary, as well. And our thinking was like, okay, like if if there's like if there's like so few standardized emails available and also they're not effective like they really are not measuring what it is that you're trying to do in your day-to-day programming job, like what do you do? So, our stance was and this was the stance of the Codex team and a lot of other teams that we've talked to. Uh why this why this was our stance just like this this email just completely ignore them. They're completely unnecessary. Should probably probably wasting your time and it's just like I don't know who who will be appeased by them. And then last year we we uh we came this idea that like okay like listen I think I think I think we we got to up the ante and we got to like we got to have some measure. We got to try emails. And if no one else is doing it, we'll do it ourselves. We'll build actual emails from scratch um that would like actually test real-world programming problems uh of users. So, we got like we went through a lot of like a massive data sets of like people who had opted in to share their coding usage of client with us and we offered them money and we we got a lot of this data set of like okay this is what the problems that people actually doing then spent a lot of time parsing through that figured out like an actual data set like these are the problems that people are solving and then just like completely cleaning it all up like doing a lot of like really hard manual labor trying to make like very decent problems that can be solved with say uh client or any other coding agent. Um the hardest part for us when we were building emails is that like if you're building emails for anything that's like rudimentary like if you're if you're building email for say an LM model, you have a very simple like one-shot use case of like how many toes does a cat have and then the LM can just be like I don't know 11 or whatever. I don't I don't know how many toes a cat has. But um a single turn email is very easy to do because it has like a binary answer. It has just like a very limited search space of what the answer could be. But when you're working with an agent, that can't be the case. You're working with an agent, you can give an agent a problem like, "Hey, uh I have this new MCP server. It's probably not working. Like, how do you how do you like make it work for me?" And that's usually how a lot of you guys talk to uh Cloud Code or whatever agent you're using, and myself as well. So, in this like it's very hard to gauge because like the agent like reads through files, searches through docs, uh installs the environment, sets things up, runs Python scripts, does all of that, and then in the end like runs some tests, and then maybe the whole thing works. Like, it's so we're we're trying to grade the second thing. We're trying to get grade like all these things that will take a lot of time, and then figure out like, "Oh, did it actually work or did it not? Did it work, but like broke other things?" Like, you So, that's why it was like harder. So, in the same time, uh some very very awesome, smart, bright people from Stanford University came up with Terminal Bench, which does the same thing, where they came up with like 89 coding problems, which are which are just like very approximate, decent representation of like real world programming problems. So, these could be things like, you know, um race conditions, uh database issues, um like um other stuff. Like, it's like figure out this infra issue. And I think that that was built The Terminal Bench was built to like use with any coding agent CLI, so you can like you can actually test and run things um like really fast with like uh CLI, and then it will take like a couple minutes to run. So, some of these tasks would take up to like 30 to 40 minutes. Uh and and that's how you know they're legit because like you the agent does a lot of things and just like runs into our code, and sometimes just goes crazy, and and yeah. Um so, uh so, we started using that. And the way to use that, like the way to use Terminal Bench, is that like you think of an eval problem as like an evaluation suit, which has a set of problems. And this one has 89 tasks. Some others would have more. And what you want to do is like you want to give it an environment. You want to give it an isolated environment, where you just like you Let's say say have a task like, "Hey, figure out this race condition for me in this repo. And the race condition is that like this thing is not working. So, you want to be able to give that the eval run an isolated environment like a virtual machine. In that virtual machine, it has the whole setup, it has the repo, it has everything. And then you install whatever agent you have, in our case client, cloud code, codex, whatever you want to use, you can do that. Uh to do that, it's it's not that it's like hard, but it is also not trivial. And Harbor is another software that came from Loda Institute um where they made this thing where let's say you have 89 tasks. One way to do evals is that learn like each of these tasks in sequence and then like do all the setups. Another way to do is like have a very standardized configuration defined in infrastructure where each of these 89 tasks have like the proper Linux machine, uh the proper RAM CPU usage, um and then like being able to like isolate those environments and then run those 89 tasks in parallel on infrastructure. So, you could use a couple different things for the infrastructure here. Uh you could use Daytona, you could run it on your Docker machine if you have like very powerful machines. I'm sure if you can handle those that much compute, sure, but I wouldn't. Uh we use Model. Uh Model, we're very thankful to Model, they've helped us a lot. Um so, shout out to them. And uh yeah, so in this case like Harbor basically lets you split up like the 89 tasks and then they all run in parallel. So, that way your limit the limiting factor is basically the slowest task.基于评测的渐进式攀登与优化系统
建立起并发的评估流水线后，真正的挑战是如何提升智能体的分数（Hill Climbing）。通过分析每次测试产生的大量大语言模型调用记录（Trace），你需要定位失败的具体原因，并精准规划优化策略。本质上，评估过程是在交叉验证以下三个维度：
- 底座模型：模型自身的推理水平是否足够强大。
- 工程线束（Harness）：你构建的代理框架是否能充分释放特定模型家族的潜力（比如 Claude Code 就将 Anthropic 的模型能力发挥到了极致）。
- 问题的真实性：你是否在解决对现实有真正意义的复杂难题。
针对这些发现，我们可以将优化工作划分为三个层次：
- 第一重优化（Obvious Flaw）：修复明显的系统级物理故障。例如运行环境配置错误、API 频率限制（Rate Limiting），或者是智能体的重试机制代码本身存在导致框架崩溃的 Bug。
- 第二重优化（Nuance Improvements）：这是决定系统上限的核心区域。你需要针对特定的模型系列进行深入的提示词工程（Prompt Engineering）。例如，给模型强制设定过长的思考空间有时反而会导致它陷入死循环（比如输出几千个毫无意义的自我复读 Token）。通过调整和剪裁提示词尺寸，挖掘各个模型的独特能力优势。
- 第三重优化（The Danger Zone）：也就是系统性的过拟合（Overfitting）。纯粹为了拿高分而作弊，虽然能让你发一条充满噱头的推文，但毫无真实的工程意义，必须坚决避免。
最终，对待任何复杂的系统研发任务，正确的方法论应该是：找到一个能反映真实情况的评测基准，构建严密的自动化执行管道，然后不断攀登与优化。你需要让系统不仅在客观分数上不断进步，同时在实际使用体验（Vibe Check）上也能给用户提供真正流畅可靠的反馈。
Original English Source
Um yeah. So, um yeah, so the slowest task is the limiting factor. So, the process is this. You you you get a score, you first do a run uh on the 89 task, you get a score, you evaluate all the failures. So, let's say you get like say 50 failures, right? Out of the 89 task. What you want to be able to do is you want to portfully allocate those failures. You want to say uh you want to run like uh another agent which goes through the traces of all the failures. So, the trace would be like this massive file which has like every single LM call that the that the that the agent did and then be like, okay, this one this specific problem failed because it didn't run past. This failed because the retry tool was broken. And once you properly allocate those failures, you figure out, okay, these are the small levers that I can pull. If I pull those levers, like I can make like massive improvements to my AI agent. So, with your testing is like you're basically testing like um three things. You are testing the model itself. Um like if you have a very decent model, like somehow like you could have a horrible harness, you could have a horrible agent, but like the model just like overshoots so hard that just like you you you know, you you get a great score. You're testing the harness. You're testing your coding harness. So, like you could you're testing say if you're using cloud code codex. So, sometimes you'll find I'm sure I guarantee you some of you have noticed that like let's say Anthropic's models could potentially work with the cursor, could potentially work with droid with with with other coding agents, but for some reason it just seems to work so much better with the cloud code, right? And I think that that that is like the testing the harness that like is the harness actually really leveraging the best best of the model. And the third problem another problem is saying if you're solving stupid problems, it doesn't matter if you score 100% all the time. So, you really got to make sure that like um um the the problems are saying which um the Luddite Institute has done a pretty great job of. So, for us it was the case like this like um we we basically like um had like this original score which was like much lower like 43%. Um we made changes to like CPU, we made changes to memory in front of the containers. Um we raised timeouts, we improved the thinking behavior. Sometimes we'd ask the model to think more. Sometimes asking model to think more actually interferes with the quality of the response because it goes in like it gets like a stroke and it just like goes in like circles and it's like it'll just like I am a model. I am a model. It'll just like keep doing it for like uh like uh 2,000 tokens. So, yeah, so like you you you you got to think through all of that. And yeah, so for us like we have like a huge like internal benchmark for all kinds of models, open source models. So, like we just like keep like a list of like trying different versions and stuff. We encourage other people to try that as well if you pretty helpful. So, whatever whenever you get like zones improvements, you get like basically three zones of improvements when you get an original score. The first one is the obvious flaw. It's like sometimes your harness really has like very obvious flaws of like there's this bug that can straight up crashes the harness. And those obvious bugs you got to fix, right? Sometimes you are not like you're getting rate limited or whatever. Fix those, that's fine. I think the zone two is the most critical one where you actually do nuance improvements. And these nuance improvements are things like there are certain prompt engineering techniques that apply to entropic model families that just straight up would not apply to Codex model family. That would be very different from Gemini model family. And those are the nuance of like why is it that this is a model that's so good that so many people are saying it's so good, but for some reason it just isn't working for me. I think those that's that is the essence of like working with agents and hill climbing. That like you figure out those nuance improvements of like tweaking your prompt, making it larger, making it smaller. And then zone three is the danger zone where it's like you're straight up overfitting. So, you're overfitting in the sense of like you're just straight up cheating to get the highest score and then you can like make a tweet about it. Don't don't a lot of people don't it don't do it. Like I wouldn't do it. I mean, never mind. Anyway, so yeah, so anyway, so this was like this is like basically the rough outline. So, the final the final wording for me would be like basically regardless of the kind of problem that you have, I want you to like find a benchmark and like like build the eval and just like hill climb on it. So, hill climbing means that like you get a score and then you improve the score off your harness on the eval. And you have to do both. Like you can't just like have like a good number and be happy with it. Like, you you you you you you got to both pass the vibe check. Like, does it actually feel good to use this product in this model? And at the same time, you also have like a very very very decent uh score, hopefully. Um if a new thing comes out, you you you do your absolute best to like give it the right judgment. Uh for us, like, one of the things that we learned was that like we were very decent on Anthropic model families, not so much on, say, Gemini model family, not so much on, say, uh Kimini model family, which again are very decent models. Um so, when we when we started hill climbing, we learned that like, "Oh, if we support these models, we have this like entire swaths of people who love these models, and they can start using us." And I think that in some attraction of that would also apply with you. Um so, my final my final note to you guys is that like, you know, I've I've done some hot takes or whatever, and if you work for for some of the companies that I've said not so nice things about, I still love you and everything, and uh it was it was um I work at Client. So, if you find these problems fascinating, and if you want to learn more about these, like, uh this my Twitter, so like you can feel free to reach out to me, DM me about like if you want to work on problems like these, like, by all means, like, I can put a word for you. If you want to learn more about evals, if you want to learn like how do you like I have a problem that's like completely orthogonal to everything you're defining for coding agents, like, how do you work on that? Uh so, feel free to reach out to me, and I'll I'll respond to you. And um once again, it's very very kind of you to give me your time. Thank you so much.📌 文中提及的人物和组织
人物: Ara Khan
公司/组织: Cline, OpenAI, Anthropic
产品/模型: Terminal Bench, SWE-bench, Harbor, Modal, Claude Code