Host: Okay, hi. We're here in the OPI studio with Mia and Olivia from the Frontier Evals team. Maybe you want to introduce your name, what you do at OpenAI, and we can get started.

Olivia: Hi, I'm Olivia. I'm on the Frontier Evals team.

Mia: Hi, I'm Mia. I'm a VP of Research at OpenAI. My team are the Codex team, the human data team, and the alignment team, and we work a lot with Olivia's team on Frontier Evals.

Host: Yeah, very exciting. And as by my understanding, you were part of the original team that worked on SWE-Bench Verified as well.

Mia: Yeah. Olivia's team, the Frontier Evals team, and the human data team collaborated on creating SWE-Bench Verified. So you've seen the evolution of coding benchmarks over time, and I think it was round about mid to late 2024 when we first released SWE-Bench Verified. Things have evolved a lot since then. What's the blog post that you've worked on that we're releasing today? What is the main thesis that you're pushing out?

Mia: So the main thesis is that SWE-Bench Verified has been one of the north star coding benchmarks that the field has looked at to measure coding progress. But recently we've seen that progress has kind of stalled, and we realize that this is because the eval is effectively saturated and also highly contaminated. So at this point, we think that it's not really measuring coding performance improvements well anymore, and we think that the field should move away from this towards other benchmarks.

Host: Like SWE-Bench Pro?

Mia: Yeah, like SWE-Bench Pro.

Host: Amazing, yeah. I always have a joke like there's a group chat with all the labs and everyone just takes turns to increment like 0.1% on benchmarks, and then it's like okay well you have the best coding model I guess because you're 0.1% higher, but it's not super convincing at this point at all.

Host: So cool. I think the, let's sort of reset on like what was the original work that you guys did for SWE-Bench Verified, which I think was pretty substantial. Like it was a very significant investment from OpenAI, which like people still don't appreciate. And then what were the satisfactions that we found over time? So like what was SWE-Bench Verified? What should people know about it?

Mia: SWE-Bench Verified was a kind of a cleanup of the original SWE-Bench academic benchmark from a lab at Princeton called SWE-Bench. The agent is basically given a codebase and a task that was sourced from a real-world repository and GitHub issue, and was asked to solve the task and is graded on whether some tests pass. And at the time, this quickly became a popular benchmark because at the time, the field didn't really have good real-world coding benchmarks. But then when OpenAI took a look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb. So we did a pretty extensive human data campaign, hiring like almost a hundred real-world software engineers to go through the problems and figure out like are the tasks well specified? Are the tests actually fair? And kind of created a curated set of like 500 tasks that we thought were much better.

Host: It's just, maybe it's hard to overstate like the amount of effort that it took to like create that benchmark. There's literally like many expert software engineers reviewing the problems like sequentially multiple times, and basically like three different experts independently decided it.

Mia: Yeah, you didn't have to do that, you just tripled your cost. But I mean, we had to do it actually because it's quite a hard task to like look at something like a problem and the patch, and then like it's not just the problem and the patch, you have to like understand it in the context of the codebase that the human or the model is in to solve the task. So it's a very complex problem and it was definitely needed to have three reviews, and I think like maybe we should have done more, but it was definitely a lot of effort to get there.

Host: Yeah. And there's more, but people can read the blog post for that. I will note that you guys had a trend in verifying benchmarks because I just recently saw I think Quinn had a HumanEval Verified for humanities, so now everyone's verifying everything, which is nice and good and like extra quality there. Um, okay. So, but I think that the meat of it is that this was a lot of like, well, here's the issue or problem statement and then here's the diffs, here's the golden tests and here's some regression tests, right? That's like the rough setup of these 500 problems. And there's some contamination always happens because all the measure was fully open. I think you did have canaries, but like you know stuff leaks. There's like multiple avenues.

Olivia: So it's not just like when we usually publish evaluations, we publish evaluations and then we add canary strings to ensure that you know they are easily filled out at training time. Obviously if you use sort of like data from like open source, GitHub, you don't have actually like a canary string.

Host: Yeah. You, uh, just before recording you were telling me that you found this in your own chain of thought with the GPT-5.2, also seeing that like they had extra knowledge or something.

Olivia: Uh, yes. So this was an example where the task asked the agent to implement something, but it wasn't told that there was this specific argument that the test was going to be looking for it using. But in the GPT-5.2 chain of thought, we actually saw instances of the model reasoning like, "hey, I think that at some later version of this repository that implemented this particular argument, maybe I should add it in." So this is an example of a test that like would be pretty impossible to pass without this contamination knowledge.

Host: Yeah. And I think you found that, sort of forced, right, and it triggered like a whole investigation both like in our own models and also in other frontier models like in the market, and like understanding how contaminated the benchmark is like across the industry.

Olivia: So we did some analysis on first of all, are the tests actually fair? And so this happened by first taking all the problems that GPT-4o couldn't solve reliably, and then again getting a lot of humans to do basically another pass of kind of digging into what's wrong.

Olivia: Oh yes. Like in over half of the problems that were investigated in that deep dive, there was one problem or the other. I think the most common problem are like overly narrow tests, where there's some particular implementation detail that the tests were looking for, but wasn't specified in the problem description. So it wasn't fair to expect that model to make that particular design choice. Like one pretty blatant example are cases where the task asks you to implement some feature, and then the tests are looking for you naming that argument or that function with a particular name. But if you chose another reasonable name, the test would fail. And another set of types of bad tests are tests that are just looking for additional features that were never mentioned in the problem description.

Host: But that is significant, is like that means that if you pass a test, actually like you probably did like a really good job. But just because you didn't pass a test doesn't mean that your implementation wasn't like a good one, right? So we only accept like very narrow versions of solutions, and like not the whole space of viable and sort of like good solutions to the problem.

Host: Yeah, I think it's important that you're doing this because it in some way it is you in 2026 going back in time and correcting your own work, right? Because you could have caught all this in the original Verified work.

Mia: I think so. It's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best effort solution and trying to compare it. But I think also, like at the time when we published SWE-Bench Verified, I think it was like a very strong benchmark. It's not like we're like oh this is not a strong benchmark at the time. I think this is something that a lot of benchmarks go through, like as an evolution, right? Like when they start to become like popular and viable, it's because they measure something like important, and models maybe do like 20% correct on them, sometimes even less. And people have something to hold on to and improve models on these benchmarks. And by the time you hit like very high performance on the benchmarks, like additional like 0.1% improvements become sort of meaningless. And so like at the time, I think you know that benchmark was like super valuable, and it taught like us and the industry a lot. It's just like now at the point that we are now, where models are as strong as they are, we're kind of starting to measure not necessarily like what we want to measure, which is like coding capability, but like the agent's ability to correctly guess how to name a specific function.

Host: Yeah. We're going to stop reporting SWE-Bench Verified, right? And then SWE-Bench Pro will be sort of the next one, which is an effort from Scale. What's your sort of comparison analysis? What attracts you to SWE-Bench Pro?

Olivia: Uh, the first one, I think, is just that it's harder. For SWE-Bench Verified, I think something like 90% of the problems are things that were estimated to take like an expert software engineer like less than an hour. They're very well specified, very self-contained. And the SWE-Bench Pro problems are just bigger and harder, and there's much more headroom on the eval because it's not saturated. Like categories of like one to four hours and more. And it's more diverse. Lots of repositories, multiple languages, qualitatively more different types of problems. So all that's great.

On the contamination side, we also think it's better there. So the way we were measuring for contamination in SWE-Bench Verified was with this contamination auditor agent, which is given the description of the task and the patch and the task ID, and told to go and try to find questions that will manage to reveal what contamination might be lurking in that model. And in SWE-Bench Verified, we found many instances of contamination across OpenAI models, across Claude, OpenAI's GPT-4.5, Gemini Flash, and in all of these we saw things like regurgitating the ground truth solutions, things like in some cases giving like the task IDs and other things that are a pretty clear evidence of at minimum familiarity with the repositories. On the other hand, SWE-Bench Pro, we don't see this. The auditor agent found some like very light evidence that maybe a couple models might be very lightly familiar with like one or two of the source repositories, but it's very different than SWE-Bench Verified. Contamination is good. I think there also, we should expect that at some point, that's not going to be like the right benchmark anymore. And like as a field, we kind of have to continue to move on and find harder and more representative problems that we can test our capabilities on.

Host: Awesome. So let's go into that. I think that there are a lot of, I think we also pre-hear that people feel qualitative difference when they're using 5.1 to 5.2, 5.2 to 5.3, and it's not super expressed in these benchmarks because the numbers are already on high. What capabilities do you really want to benchmark in an ideal coding benchmark, or agent coding benchmark?

Olivia: I mean, one thing is uh kind of open-ended design decisions, places where the problem maybe is a little bit underspecified, and seeing if the model can make reasonable design decisions.

Host: What's a reasonable prompt for that? Like code me a B2B SAS to make no mistakes, or you know, like that's the meme, but okay, what's like an actual usable open-ended problem?

Olivia: Oh sure, I mean, maybe an example could be finding a way to speed up a particular part of a codebase, but there might be multiple different ways to do it.

Mia: I think there's just many things that people like value about working with software engineering agents. SWE-Bench Verified obviously measured like some, it measures some important capability, which is like given a description of a GitHub issue, can you produce like a patch that solves that issue? And like obviously there's like some issues with the benchmark that means that now that we're at like 80%, we don't really trust like further improvements on it. But like it does measure something that is like a real capability of models. But I think as a field, we're like moving beyond sort of you know, can my coding agent like solve a small GitHub issue for me? And so we're starting to look at like much more longer-term tasks, right? Like that don't take like 15 minutes, but maybe like an hour, sometimes days. And then beyond sort of like what kind of tasks can my agent solve, like there might be things that are kind of a bit harder to grasp, right? Like Olivia talked about sort of like does it have like design taste, right? Like does it solve the problem the way that you know my team likes to solve problems? Is the code nice, right? Like is it well written? Like is it sort of like clean code, right? Like people care about these. Is it maintainable in the future? People care about a lot of these maybe less tangible, less tangible, and like harder to measure, frankly, things that that are still like super meaningful for people that are working with coding agents.

Host: Yeah. So, these are all qualities that are obviously no longer the low-hanging fruit. Like we have no idea how to eval these. There's sort of two forks in the road. One is the sort of very human-intensive, money-intensive path, which is hire a bunch of contractors and try to annotate this. The other is use an LLM to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? What you want to do both?

Olivia: I think like maybe you should talk about GDP as like an example.

Mia: Um, sure. So GDP is an evaluation that was again produced by a collaboration between human data team and the Frontier Evals team, and it's trying to measure whether agents can do kind of a variety of like real-world white-collar work. That was an evaluation where grading is very hard, requires kind of a lot of domain knowledge on exactly what you are looking for in each different context. Across like 15-16 white-collar professions, which is basically the evaluation for AGI. But because it was so hard, it required so much domain knowledge, the human data team hired like a lot of people from these professions to be very involved in creating tasks and creating the gold solutions and trying to help create rubrics, and so forth, so we can create it reliably. So basically, take the GDP evaluation, which is a generalist thing, take that same approach to apply it to code, and you roughly have like a rough roadmap.

Host: I think it's an interesting solution. I think what you're pointing out as an important problem, which is sort of this, like how realistic is it, and like you know what we want to do is like coding agents should write code that you know we think is good. And so it's like asking human, that's actually like a good way to ensure that. It's also kind of a slower, like complex way to do that. And so part of why I think you know SWE-Bench Verified ended up being super popular, and where we are seeing like other benchmarks like this being super popular, it's like it's very easy. It could even be easier, but like validating that a solution passes all the tests, it's like pretty trivial once you can like run the tests on your computer or wherever you're running them, and you can kind of like okay is it correct or is it not correct, and you can kind of aggregate that. And that it's super simple, but it doesn't tell you if the agent solved the problem like, would have an open-source maintainer of that project have like merged that PR? But there is a lot of value in having benchmarks that are both like easy to compare across the industry, and also that can be sort of run really fast without human involvement.

Host: Yeah. Amazing. Your teams also put out other kinds of evals that are related, like I think there's an RL Paper Bench. And then sort of like the more sort of recursive self-improvement type evals. How much should that figure into mainstream coding evals? Like is there a way in which those things join together?

Olivia: Um, I think I just think like those are some of the most advanced evals that we have and we're not using them in the normal path, and it's just it's an interesting split between well here's evals for coding normal things, and then here's the one for machine learning that is like completely different. That's mostly a safety argument, I guess, but also like it's actually really useful for people to understand if the model is really good at like AI code basically.

Mia: Yeah. Oh yeah. Like my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of like what datasets are easy to gather, because a lot of the state-of-the-art AI codebases are proprietary. So if we make evals for that, we're probably not going to release them, and it's harder for people in the field to make that kind of measure, like is this a realistic research coding workflow? I do think that it's good for the field to try to measure these skills in a public way. I think it's just harder to make it realistic.

And then one more thing that a lot of people are trying to do, which is like sort of well, instead of like a percentage of 0 to 100, maybe we redenominate in dollars? So you had freelancer and all that, other people are doing like Upwork Bench, whatever. Any alpha in those, or are they, are they you still want like a traditional academic benchmark?

Olivia: I think in a way like there's like different ways to measure the same thing, right? If we're like, "Oh, this is like how much money it produces," it's a fairly similar thing to saying like, "Oh, this problem would take like a human, you know, two hours to solve or something like that." Usually, they're like fairly correlated, right? But like, however much it would take like a human to solve that problem kind of determines the value that we ascribe to a solution. And so I do think that's an important thing is like how complex and how sort of long-running are the tasks that we are able to entrust our agents with. And so I think that's like an important piece. But I think here, sort of monetary value, time, or complexity, they all kind of try to capture similar things.

Host: Yeah. Okay. So they're they're all proxies for some amount of increasing capacity that we want to measure. I think that's a good thing. I think the only other sort of major player in this field is Aider, which has done the long-running autonomy test eval. Congrats, you guys have completely destroyed the curve for that. Any takes on that? Obviously you've come up really well. So like it looks good, but I don't know if like that approach is something that you want to incorporate in your own work making evals.

Olivia: Yeah. I know we're from here and we work with Aider on these evaluations. So like we do appreciate them. I think then they're using time, right? They're not using money. So I think that was your question. I think complexity, however we can sort of quantify it, is really important to understand like where our models are getting to.

Host: Okay. Complexity is the abstract thing and then it projects down to time, projects down to story points, whatever, dollars. Great. Um, one last question on just like the overall preparedness framework, you know, I was actually kind of looking at people mention the preparedness framework a lot. I don't think it's well explained to a lot of people. And you actually have a nice website where it's like test and inform and teach something. And I feel like you you actually do a lot of work there. Do you want to talk about how the preparedness framework applies?

Mia: So the preparedness framework is an open kind of like public framework for how we track frontier risk. So these are kind of capabilities that are typically dual use. Like you can use them for good things or bad things, but we want to at least keep an eye out for the bad things to make sure that we have, both we as a company and like the broader society, are kind of prepared to handle the potential downsides.

And so at the moment, we kind of track three different categories. One is kind of uh biorisk, another is cyber security, and a third is kind of uh research automation and model autonomy. And that's kind of what ties most into the bench, where coding is not all of automating research, but it is one very important key component.

And so we initially created SWE-Bench Verified as part of building out evals for that model autonomy workstream. And now, we're like we have to move beyond that towards looking more at like can models actually start to automate research workflows.

Host: Yeah. Amazing. Great. Is there anything else to add on just the general what people should know about preparedness and how eval and human data alignment all work together?

Mia: I think maybe the thing that I would say is that we really appreciate, we work really hard to build these evals, and so we, that's where we published SWE-Bench Verified and that's where we're like sharing GDP and these sorts of things. We also deeply appreciate like other people and the entire field to kind of build evals and share them and reuse them, like SWE-Bench Pro, like yes, that's a better eval, now we should use them. So would really encourage people to find more ways to create and share evals that we can and the entire field can use to measure progress on a variety of capabilities, including encoding, because it's important to understand where we are.

Host: Mia had to leave, but we're just kind of talking a little bit about like the future directions that we want evals to go.

Olivia: Um and I I think here we can dive in on like give us work, good work on these things, we'll talk to you know, here's your platform to make a call for what you're looking for. I think a few things that would be useful. I'd say first of all, really really hard tasks, like the kinds of things that would take top-notch engineers months or teams weeks. Um, would be quite good, um, especially if grading is reliable and grading has like rubrics that have been sourced and validated by many people in the field. I think that would be quite valuable.

I think also benchmarks on kind of creating products end-to-end. I think as people are deploying more, that would be quite useful.

I think a third thing I'd say that is maybe not quite an eval, but I think is still relevant to the kind of overall mission of like we as a field and as a world should be tracking like where are these capabilities going. I'd like to see more metrics tracking like real-world usage. Like how much is AI actually being used in the field? How much is it replacing people's jobs? How much is it augmenting people, speeding people up? Just like real-world metrics.

Host: Yeah, yeah, the replacement thing is always like a sensitive on the PR side of things, but you know we create new jobs that manage the old jobs, and that's how it is. In terms of the frontier evals that OpenAI is really going to be excited to push, like you put out really good work every single time. What should people expect from OpenAI itself?

Olivia: I'm not sure I could say what we're going to do.

Host: General directions.

Olivia: I mean, a general directions, I think looking at real-world impact, like real-world, um, real-world you know, whatever.

Host: Yeah. Amazing. Um, okay. Well, I'm excited for real-world impact. I think you guys have, you know, really made a lot of progress, and I think taken a lot of industry leadership for SWE-Bench Verified and now moving on to SWE-Bench Pro. So, thank you for doing this. Thank you for being so transparent. And I think people will respond in kind.

Olivia: Yeah.

Host: Thanks for your time.

Olivia: Thank you.