Universal Verifier：如何构建超越人类一致性的网页智能体评估系统

Miguel: Good day everyone. Thank you for coming. I hope you have a great time at the exhibition. We're already at the finish line, but I know it was a great, just a great exhibition, at least for me. So I hope you learned a lot, and we can teach you a little more.

So, I'm Miguel. I am the technical lead for our agent platform at BrowserBase. For those who don't know, BrowserBase is an infrastructure company for deploying agents on the web. It effectively opens up the entire internet, not just what is readily available, by managing browsers in the cloud and the entire runtime environment needed for scalable agent-based web automation.

Corby: So, but I am Corby. I'm a researcher at Microsoft, and I'm working with BrowserBase to help build really high-quality verifiers. Today we're here to talk about a study we published about a month ago. It concerns evaluation. Namely, reinforcement learning (RL) environments and the use of language models as judges for verification.

First, let me say that I've been working on this open source framework called Stat 10 for about a year and a half now. From the beginning, like any good agent product, we created a measurement system to ensure that the changes we make to the framework and model capabilities continue to improve and help drive these metrics. Initially, all of these estimates were deterministic environments. It was relatively simple. The capabilities of the models were somewhat limited. So we were able to build a lot of deterministic environments. As the capabilities of models continue to grow, and we continue to improve the toolkit of agents, this approach has ceased to be scalable.

Corby: We were constantly fixing or creating static sites, inventing longer trajectories with many checkpoints to make sure that the evaluation results could be deterministically verified. There are many different challenges with automating the web at scale and verifying that what you have in mind has actually worked. The fact is that the web is a very open system. There is no single path to correctness. There is no absolute truth. Sometimes the web changes and the product you were checking out before no longer exists. This destroys all determinism of your environment. There are also obstacles. There are many environmental errors that prevent a verified deterministic signal from being obtained. So we resorted to what most people do: we tried to use an LLM judge to improve and scale this operation.

Corby: But we quickly found out that many of the leading benchmarks—some of you may know about OSWorld for computer use, but there are similar benchmarks for the web, like Online Minor Web and Web Voyager. They are supplied with LLM-based judges as verifiers. And we noticed when we used our human experts to test the verifier that in many cases they were very confidently wrong. And this leads to results that cannot be trusted. So we realized very quickly that even if you use this as an RL reward or a signal to improve persistence in other studies, you're not training a better agent. You are simply training a more confident liar.

So, in a real-world case at Microsoft, we trained the Farfetch'd 7B model, which is a web browser agent. And we evaluated the same model on the same benchmark against the official Web Voyager judge, i.e. GPT-4o, and it showed a success rate of 74%. But there is a huge gap between the real truth, which we obtained with our new universal verifier, which has a high level of agreement with human estimates, and that number quickly drops to 38%. So there is a huge gap between what existing verifiers say and what actually exists. And we needed to find a way to bridge that gap.

Corby: Existing verifiers have many weaknesses. They use smaller and much simpler models as LLM judges. They use GPT-4o mini or GPT-4o. They don't even use rubrics. Rubrics are the most important thing to have because you need to distribute credit where it is deserved. They don't look at the relevant screenshots or try to view all the screenshots at once, quickly getting lost and cluttering the LLM judge's context window. And sometimes they don't even look at the final answer or the model's action history.

So our verifier takes all these aspects into account. And here's how it works at a very high level. You can find more details in the article, but for a task like “book the cheapest flight from Seattle to Boston,” the first thing we do is create a very high-quality rubric, and I’ll give you a few examples. The rubric has, say, n different criteria for what success looks like. And then, based on these criteria in the rubric, we analyze the trajectory of the agent's actions. All screenshots in this case are evidence of the true state of the system. We rank the most relevant screenshots in the trajectory for each criterion. We then use this group of K best evidence to determine whether this criterion has been met. And are there any discrepancies or contradictions between what the agent said he did and what actually reflected the state of the environment.

With that in mind, we issue a graded rubric, which we call a process score because it gives partial scores in some cases, and we also issue a Boolean result value to essentially say whether the agent performed the task according to the expectations of a reasonable user. Uh, and this is an example of what the output of the rubric is. This is essentially a list of criteria, plus a result check, which is a logical true/false flag with an explanation of why a reasonable user would expect this trajectory to be successful or not.

Corby: When creating this universal verifier, we were guided by four main principles. The first concerns the creation of the rubric: we wanted to evaluate only what was asked for, not any extraneous criteria. We also didn't want errors to cascade from one rubric criterion to another. We wanted them to be isolated, and I'll give an example of that. You also wanted to look at screenshots confirming the true state. This is a mandatory condition. Agents often claim too confidently that they did something when in fact they did not. So you have to look at snapshots of the true state of the environment. And we also needed to separate what the agent could control and what the agent could not control. The agent runs in a web browser. A web browser is an environment that an agent cannot always control, and I will also show some examples of this.

Corby: So, this is an example of what a good and bad rubric looks like. The task here is to find a cheap hotel in Jakarta for these dates, and then use the hotel address to search for the nearest coffee shop and output the name and address of that coffee shop. But a bad rubric, which we actually saw, would have given a criterion like: "Please tell me the total price for staying at the hotel." This is actually an extraneous criterion that was not asked for in the task, but we have seen many naively generated rubrics do something similar, and they artificially lower scores because the agent did not do something that was not asked.

Um, regarding avoiding cascading errors—there are very subtle errors because many tasks build on each other if they are multi-step processes. So, in this example, the task was to determine the net worth of the person with the longest last name among the members of the bands N Sync and Backstreet Boys, okay? The agent in this case mistakenly decided that Timberlake had the longest last name, consisting of only 10 letters, when in fact it was Kirkpatrick's. So, the model made one arithmetic error in one of the rubricator criteria, and this should not apply to the next criterion—reporting the net worth of the selected person, right? So if you can accurately report Timberlake's net worth, you shouldn't be penalized for this criterion. You should be punished for just that one point.

Corby: Hallucinations are the most important aspect that we really wanted to focus on. Here is an example of a very subtle hallucination in a task where you had to find information about a model for describing images, and the agent stated that this model has a plus of 6.2% according to some Cider estimate. But in fact, the annotation to the relevant article indicated only 2.8%, according to Cider's estimate. It was a very subtle mistake that even a human wouldn't notice until LLM pointed it out. Hmm, that gives some idea of how process assessments differ from outcome assessments.

Um, we mentioned that one of the principles is to separate managed and unmanaged failures. If the task is to buy, say, this plush toy on Amazon, and the agent definitely found it but discovered it was out of stock, he was unable to proceed. Therefore, we believe that the agent receives full credit for the effort made to achieve the goal. But the result was still not achieved, as he was unable to buy the desired item due to it being out of stock.

Um, we actually listed a bunch of scenarios of what could happen and how we assign grades. If the agent could find a similar alternative plush toy in another way, um, he would still succeed in both cases. And he will receive a penalty if he makes a controlled error. Um, he'll be fined if he hallucinates, and so on. So, we actually made a diagram of all the possible types of failures and how we would award points for it. And the universal verifier adheres to these schemes.

Corby: So, to evaluate the verifier and really understand whether we were able to improve its accuracy, we started designing a bunch of experiments involving human experts and created a whole platform for collecting data that we called reference labels, which we also released as Cool Verify Bench for others to learn from.

The process we followed was as follows. Initially, we showed the full trajectory of evidence to human verifiers. Annotators made judgments based on the evidence and results of all the actions the agent performed. After that, they were asked, upon seeing the universal verifier's assessment, to confirm whether they agreed with it or not. We found this to be a very reliable signal to check if we were moving in the right direction, because I'll get straight to the point: the verifier started correcting people. He discovered things that people missed.

Ultimately, the result of the universal verifier working with these reference labels was that we were able to reduce the number of false positives from almost half to zero. Cohen's Kappa is a metric for measuring the consistency between our annotators. And what we see here is that the universal verifier agrees with people as often as people agree with each other. So we see a Cohen's kappa of 0.58 compared to two annotators on the task, where we also calculated Cohen's kappa to see how well they agree with each other.

Corby: Another way to check the quality of our verifier is to train it on data filtered by it. So we conducted an experiment in which we left the number of training examples unchanged. In this case, it was 3 thousand or 9 thousand training trajectories, but we filtered these trajectories depending on whether they passed our verifier. When you do SFT, which is what we did here, you only want to learn from examples that were successful trajectories.

So if you train on 3000 trajectories filtered by the score of our universal verifier, you will get a much better quality model than if you trained on a worse, baseline verifier. This statement turned out to be true on a larger scale as well. So this experiment showed us that if you have a better verifier, you can filter out better data, and training on that data leads to a better model. So, this was like the final experimental proof in addition to the human validation that this model, this verifier works and is reliable in a real production scenario.

Corby: The last thing we did as a side project was try to determine whether artificial intelligence in self-study could create the same verifier as we did. So, basically, Miguel and I sat down for 3 weeks to build a universal verifier, tweak the prompts, and write code for it, and we ran about 30 different experiments over the course of 3 weeks to determine whether the verifier we were building matched people's estimates. Okay, and that's what Cohen's Kappa measures here on the Y-axis, it measures whether a particular verification system agrees with people's estimates.

So the blue line here is the model that Miguel and I were building, this auto-research system, this universal verifier. And then we wanted to see, if we took away everything we had done, could the AI in the auto-learning loop create the same verifier that we had created, with the same level of quality and accuracy? And that's what the red and green lines here represent. The red line shows whether the AI could reproduce this if we removed all our code and clues. The conclusion we drew from this is that while it took us about 3 weeks to build this verifier, an auto-research cycle could do it in about 1 day. He conducted the same number of experiments in about 1 day. However, it only achieved about 70% of the matches that our verifier was able to achieve. So, the gap still exists. The conclusion I would draw is that you can use auto-research to create metrics and verifiers, at least to help you speed up your experiments, but you probably still need some level of human intervention. On the other hand, these results were obtained on Opus 4.6. I haven't tried this with Fable yet. Maybe he'll do better. But it's a pretty strong base, and it's come pretty far. So, auto-research can help you not only build a model, but also, in this case, build a verifier, which we think is a pretty cool thing. And creating verifiers is just as important as creating the models themselves; This is one of the main conclusions I want to leave you with.

Miguel: Yes, and to further emphasize: the green indicator is a verifier of automatic research, prepared based on the results of our three-week experiments. Thanks to this optimization, we were able to achieve higher performance than ever before. Therefore, combining human intuition with cycles of automatic research has proven to be the most effective recipe for many scientific tasks.

Corby: This entire project is open for use. The article is published as a preprint on the arXiv website, and the Microsoft Auto repository contains reference marks for the verification bench. It also contains code for conducting experiments. But beyond that, the study allowed us to create a completely new benchmark. We have already briefly mentioned Online Minor Web and Web Voyager. I spend a lot of time working with labs, helping them get quantitative signals to improve their models. Many such open source benchmarks exhaust their resource very quickly, and it becomes difficult to get any useful signal, since the test data is already effectively included in the training distribution. Therefore, this new benchmark has proven to be the most promising for assessing success, and I use it every day to understand the strengths of the models, their shortcomings, and ways to bridge the gap. That's it. Did you have any more slides?

Miguel: No, that's all. We can answer a few questions, time permitting. Thank you.

Audience: Yes. Yes.

Corby: So, the question is whether we used the universal verifier for tasks other than web tasks. At Microsoft, we are currently working on adapting a version of this verifier for desktop tasks, i.e. what you do on your laptop, such as corporate workflows. We hope to publish another benchmark not only with web data, but also with corporate desktop data. We will create the same type of verifier for this as well. It will take into account more information, as there is a terminal on the desktop, more telemetry and logs than just in the browser. Therefore, the verifier will also take this data into account. Yes. Any more questions?

Audience: Yes. Yes. Yes. Yes.

Corby: So, the question of how to ensure that the verifier does not overtrain on human estimates is a real problem. Miguel and I spent a lot of time working on this issue, so we actually put aside two sets of labels. We had a set of about 150 trajectories. I marked about 50 of these trajectories myself and used them for optimization in those 30 experiments, but we didn't touch the other 100 trajectories. They were processed by people we paid, with double overlap. So when we finished developing our verifier, we handed it over to these mappers, and each of them mapped each trajectory. Sorry, two people marked each trajectory. And that's exactly what we published in the Co-op verifier test. So these are the numbers we are presenting here. We are pretty confident that we did not allow overtraining because we left at least 2/3 of the tags in reserve. We didn't train or iterate on them. Yes. This is very important, you don't want to learn on your own...

Yes. Yes, the gold standard is to hire people, train them to perform verification tasks, and then check if your system output agrees with that. Any more questions? Anyone want to help us create more benchmarks? Yes. Oh, yes. OK. Good. Yes, yes, yes. Perfectly. Well, thank you very much. I hope you enjoy the rest of the exhibition. It was nice. Thank you very much. Thank you very much. Yes, yes. Mm.