Hey guys, thank you so much for having me. My name is Joel Becker. I work as a researcher or member of technical staff at METR, which stands for model evaluation and threat research. As we'll see in a second, I'm going to be talking about AI capabilities. How do we know how performant AIs are today? How how performant they might be in the near future from these two different sources of evidence that seem to give somewhat conflicting answers. You know, I I could have done this whole talk without reference to meter papers in particular, but we'll look at two papers I've been um involved with as as examples of benchmark style evidence and then more economic style evidence. On the benchmark side, measuring AI ability to complete long tasks. This is the paper um that comes with the the charts that many of you would have seen on on Twitter and so on that meter is well known for. And then the second this um RCT measuring how allowing AI affects developer productivity. And then we'll be talking about how to reconcile uh the the gap that's implied between these two different kinds of measurements.

As I mentioned, META stands for model evaluation and threat research. We are an independent research nonprofit that seeks to inform the the public, policy makers, labs about the degree to which AIs might pose catastrophic risks to society. The model evaluation part uh means that we seek to understand AI capabilities and propensities. And the threat research part means we try to connect those capabilities and propensities to potential catastrophic risks.

Okay. The first paper we're going to talk about associated with this chart that that many of you I think might have seen. Um take taking a step back first before we dive into the paper. You know how how usually do we think about measuring AI capabilities using benchmarks on a SWE bench or a GPQA so on and so forth. There's some notion of 0% performance um or or random performance. So for GPQA that's that's 25% which corresponds to this flaw that the worst you can possibly do. Perhaps there's a um human baseline that's below 100% for GPQA. I think this is something like 75% that represents maybe expert human performance. And then of course you can go all the way up to 100% potentially on on these kinds of benchmarks. But but what does it mean? you know, if I'm getting 50% on GPQA, if I'm like half the way from the um from the floor to the to the expert baseline, what you know, what does that really mean about how performant the AIS are? If I meet the human baseline, does that mean that the AIS are now as performant or even more performant than than expert humans in in a relevant sense that I that I care about? It's hard to interpret. You know, another thing that you see from this graph is that um benchmarks seem to have less and less time between coming online sort of giving any signal at all and being fully saturated. It's harder and harder to create benchmarks that have uh plenty of signal that you know might might be informative to us about how capable models are for for an extended period of time. So, we're we're going to go about this a different way.

First, we're going to gather human baseline data for diverse tasks spanning a range of difficulties. You should think of these humans as, you know, experienced experts, but on their first day or or or first week on the job. These are not people with context on the tasks in particular. It's not exactly the kind of thing that's come up in their work before, but if it's a software engineering task, you know, there are relevantly skilled general software engineer. Same for the machine learning tasks and the cyber security tasks here that we'll talk about. the the [snorts] type of tasks come from these three um buckets or task distributions. Hcast which is a collection of um softwarebased tasks seemingly requiring autonomy you know interacting with tools um uh interacting with the environments thinking thinking through the problem not not just this kind of Q&A style um style data set um the SWAR suite which are these atomic problems these are problems that you know maybe GBT2 can do maybe maybe it can't problems like um here are four files one of them is called passwords.txt txt which file contains the passwords and then on the other end of difficulty we have rebench which are challenging novel open-ended um machine learning research engineering challenges which are are very difficult even for top human experts

in addition to gathering the the human baseline data we'll also under as close to identical conditions as possible measure AI performance for the AIs that we're that we're interested in on the same set of tasks and then we're going to convert the time it takes for humans to complete these tasks into an estimate of AI autonomous capabilities as I'll I'll show you in a second. Here's an illustrative diagram in this case for claw 3.7 Sonet which was the the frontier model at the time that this paper came out. You can see that you know for the for the very short tasks something like 4 minutes or below Sonet is getting the answers correct you know essentially 100% of the time or or maybe even here literally 100% of the time. for the very hardest tasks it's struggling and then and then there's some range where we're kind of in the middle you know we're somewhere between 10 and 10 and 90%. I'll say that this empirical pattern where models are less performant at tasks that take humans longer is you know it's not a fact of nature but it's it's something that we see pretty pretty commonly pretty pretty robustly across models at least on this task distribution and I'd conjecture for for other task distributions as well. So we try and fit this dark purple line to to something like this data on on how long it took humans to complete the relevant tasks that the models are uh um are attempting. And then we call the point on the x-axis this horizontal axis this human time to complete axis at which we predict the models will succeed 50% of the time the time horizon of those models that there's much to debate in the 50% number. I can I can talk later about the reasons why we chose that and and then we'll do the same exercise for the other models. So here I have uh claw 3 opus has a time horizon of something like 4 minutes. That's where we're predicting that it has a success probability on this task distribution of 50%. For 01 preview I'm seeing something like 15 minutes so on and so forth. And then of course all these models you know they they come out over um calendar time. So if we plot the time horizon, the x-coordinate on uh on on this set of plots against um against calendar's time, we find something like this. It looks, you know, kind of like um kind of like an exponential trend that's that's going up at some constant rate. In fact, it doesn't just look like an exponential trend. If we had a perfectly straight line here, it would indicate um a perfectly exponential trend. um we we see something really remarkably steady actually much more steady than we were anticipating when we uh went about doing this research project and that's continued to be the case. So many of you will have seen updates that we've made of of this graph on on on Twitter. This is going all the way up to GPT 5.1 CEX max. So extremely recent um the predictions from this you know shockingly straight line have have held up very well I think.

Taking a quick step back, what are benchmarks telling us or or here kind of benchmark like evidence? Well, one thing is that AIs can succeed at what for humans would be exceedingly difficult tasks. The tasks in our ebench are, you know, really far beyond my capabilities uh personally and and you know the AI is having a good crack at them some some decent percentage of the time. And the second's you know kind of obvious is that progress is rapid. >> [snorts] >> On the other hand, um you know, how much how much stock should we put in the um the evidence suggested by benchmarks? Um what limitations might they have? Lots, but here are here are three that I'll note. One is, as I mentioned, these are humans who are, you know, expert in some relevant sense, but they're low context. It's something like their their first week on the job. They haven't seen tasks exactly like this previously. They just have some relevant experience. presumably people who were more sort of you know not not just having the relevant experience but also highly familiar with um uh with the with the set of tasks would perform the tasks even sooner and then we think relative to those people the AIs were more performant. The second is that benchmarks can be low ceiling. Even you know GPQA or use that example again um where we're beginning to get to the point where where that benchmark is um is totally saturated not providing um additional information for marginal models whereas time horizon is providing this nice way to sort of chain benchmarks together in in in some sense over time. Um but you know nonetheless it's still very hard to um uh to create these ever harder tasks when the um when the time horizon of models is doubling every something like six to seven months. So even time horizon might be might be saturated in not too long or the benchmarks underlying time horizon. And the next one is you know not not a concern that's limited to the to the meter task to the task behind time horizon. It's also true for sweet bench. which is also true for for many of your um favorite agentic benchmarks that the problems aren't very messy in some sense. They don't require a ton of coordination with humans. They're often in relatively small contained environments where where not much can go wrong. You know, not these sort of massive open source code bases or or um other ways in which the the problems can involve more interaction with the real world or or or be messy in in some sense.

Um so we did this we did this project and then um early this year we were you know we were trying to think about um uh how can we attack some of these limitations? What what's a different source of evidence that um might have its own own pros and cons but you know importantly be more externally valid in in the scientific jargon. Perhaps field experiments are the answer. [snorts] So more economic style evidence. So here we might be interested in very high context developers who are expert on the kind of tasks they're already doing speed up or some notion of productivity boost. You know it seems to have more signal through even some um superhuman according to benchmarks range. You know perhaps GPQA is fully saturated and you're getting a 1.5x 2x speed up something like that but you can still achieve a 3x 4x 5x speed up even even after that we we maintain more signal. And the last is that you know that the tasks are messier. They are tasks that are coming up in people's real work. They're not um synthetic. They're not small and contained. Um this is a real deployment scenario. Here's what we're going to do for this paper. We're going to gather 16 experienced developers on large mature open source projects that we'll go through in a second. Each of these developers will on average complete about 16 tasks from their real work. These are these are issues on the on the relevant GitHub repositories. The kind of thing that they might otherwise have completed with the with the caveat that the very longest issues we're not going to include. The tasks will be randomly assigned to AI disallowed or AI allowed. AI disallowed, you know, it means it means what you think it means. It means software development in 2019. It means no AI powered tab autocomplete. It means no cursor agentic coding tools. It means no LLMs via the web UI. or they can be randomly assigned to AI allowed in which case everything's on the table. You know, any of the AI tools I just mentioned or not using the AI tools. If you're in the AI allowed condition, you're not compelled to use AI. You just have the option. And we buy these developers Cursor Pro. So, um for the for the most part, that's the tool that they're using with typically 3.6 or 3.7s on it at the time, uh which was the Frontier model when we conducted this work. And then we're going to record the time it takes for the developers to complete each task and see the degree to which they might save time when AI is allowed versus when it's not.

These are some of the repositories. Many of you will be familiar with them. We've got the Haskell compiler represented. We have scikitlearn. We have hugging face transformers. These are on average a million lines of code plus. They've been around for 10 plus years. The developers who are going to be working on these repositories as part of this study are on average the third top contributor out of hundreds or or even in some cases thousands of contributors to these repositories. They personally have been contributing to the repository for something like 5 years on average. These are top experts.

Some of you might have seen this graph too. And and so the punch line's been spoiled for for the rest of you. Um we asked uh economics experts, machine learning experts, you know, these are people at major AI companies and labs, um uh top academics, um some graduate students, so on and so forth, you know, how much they expect developers to save time when they're using AI. They say something like 40% or a little bit less. We ask the developers themselves, the study participants, how much they expect to be sped up ahead of time, and they say something like 24 25%. Then we ask the developers after the study has been completed how much they think they were sped up in the past by AI being allowed on the issues they completed as part of this study and they say that it will have sped them up by something like 20%. And the punch line is that we find that developers are slowed down by 19%. They take 19% more time when AI is allowed relative to when AI is not allowed.

You know, when I first saw the data coming in, saw sort of early versions of this plot, um, I thought presumably the same thing that many of you might be thinking right now, that we've messed something up. Um, that that, you know, something's gone wrong. There's some there's some issue in in how we've set up the experiments. How could it possibly be the case? You know, at least these um, uh, these developers have access to the zero points because they cannot use AI at at any time. Um, so we poured over, you know, many, many, many, many, many hours of screen recordings from these developers working on issues as part of the study. We look to dive into um, a bunch of hypotheses that might explain what's going on and try to categorize, you know, the things that that we think are going on versus not. Um, many of this is is listed in the paper. I I'll just quickly go through some of the things that we think are contributing. First, overoptimism about AI usefulness. that that seems like an obvious one. You know, the developers even after the study is completed, they think that um uh that AI is going to be helpful to their work. It's it makes sense they might overuse AI um on that basis. Um two more implicit repository context and high developer familiarity. You know, these developers are coming to these problems already knowing the solution to the problem. They don't they don't um they're so expert in this work. you know, I I I imagine them as as not trying to spend a bunch of time thinking through the solution that the the AI can can work through. Instead, they're just limited by how fast they can type. Um, which which means that, you know, using AI, instructing AIS to do it, um, comes with some significant time cost versus how they might otherwise have spent their time. I think many of us have the sense that AIS might be less performant on on large and complex repositories, which is a different from this difference from this benchmark style evidence or or from or from some previous work. And then low AI reliability. You know, um maybe the AIs are very performant on these kinds of tasks, but you know, they're only performant um 50% of the time or 80% of the time, 20% of the time. And so, at the very least, you need to check their work afterwards. And perhaps even you need to spend time correcting their work afterwards, which is which is something we see quite a lot on these issues.

One thing from the factors with an unclear effect that I that I'll mention briefly I have to talk to people about later is below average use of AI tools which came up in the public discussion. This this is in the unclear column because it's sort of evidence evidence for and against. Um that that's true for for many of the things here. We don't have anything so conclusive to say we're still working on on this line of work. Here are some here are some caveats. All important. Um first you know obviously we do not provide evidence for all software developers or tasks. These are extremely experienced developers working on extremely complex longived open source repositories. I in my own work you know not um as expert in the relevant sense as as these people are. I'm working on much smaller repositories. Um I I feel more comfortable saying that even at this time I was sped up by AI tools even if even if the developers weren't. This setting is weird. It's weird for the same reasons that it's that it's interesting this this unusual developer population. Second, the experiment is concentrated in March 2025. As I mentioned, uh we know that AI progress is rapid. Um perhaps this this result will have already changed by the by the time I'm giving you this talk. So there's a kind of puzzle suggested right that the benchmark style evidence is giving um a very impressive sense of what benchmark of what AI capabilities look like today whereas the more economic style you know I include labor market impacts um uh uh working here too in addition to our in addition to our field experiments look somewhat more bearish or or unimpressive. You know why why is the former not not translating to the latter at least naively there seems to be a clash. How might we go about resolving this puzzle?

So one possibility is that in fact we we messed something up. This is this is still live and on the table. Uh you know maybe the developers really are um uh not very capable at using AI and if we continue to run this experiment as as in fact we are they'll you know learn more familiarity with the tools and and so get productivity benefits that they they weren't getting at the time. I'm a little skeptical of that story but but but that's one possibility. Another that economists like to bring up is that we're not incentivizing these developers to finish quickly. we're paying them per the hour, um, which we do for external validity reasons. Um, you know, looking through their videos, I I really, uh, do not think that they're developing differently in accordance with these incentives, but but that certainly is one possibility that's on the table. You know, another um, more statistical in nature possibility is, you know, this is a small study. You shouldn't you shouldn't over update so much from small studies. We we are doing um, bigger things that I'm excited to release at some point. Okay, but let's let's assume we haven't messed something up and this is uh this this is a result um uh that that we think that we think does hold up. How could we resolve the puzzle? [snorts and sighs] So, one possibility, you know, as I as I alluded to briefly is that reliability needs to be very high to save time. That you need to be getting um the the answers these problems that developers are putting in correct. you know, something like 95 99% of the time in order for developers to tab tab tab through and you know, not not um not spend lots of time verifying the AI's work, which which of course um is pretty costly from a time perspective. Another possibility is bbenchlike or algorithmic costless scoring at the margin versus mergeability like scoring. Sweet scores are not trying to account for you know whether the code is spilled honable by by other people in future or whether it's matching quality considerations that aren't um considered by the unit tests. You know perhaps AIS really are performance according to SWEBenchl like scoring but not performance according to this kind of more holistic um uh holistic scoring that we might care about low versus high context baseliners. I I I mentioned I mentioned previously these are just much more skilled humans, you know, relative to those humans. Perhaps the AIs are less capable. Task distribution, maybe these are just different kinds of tasks, you know, in particular less less messy than the than the benchmark style task. Maybe that's explaining what's going on here. [snorts] Suboptimal capability elicitation. A huge amount of work has gone in at meter to making the agents as performant as possible given the underlying models on on our kinds of tasks. And um you know that involves churning through a load of AI tokens. Perhaps that's that's less the case for cursor in particular at the time when we completed the study. And then interdependence across tasks. Maybe it's the case that um you know if humans can complete task A and task B. AIS can only complete task A but not task B and of course can do task A faster. then it still makes sense to for humans to do task A and task B, not delegate task A because you know they they need to know the outputs. They need to know how how task A was completed in order to reliably complete task B. I think that's that's part of what's going on. You need to maintain context as you're working through these subtasks.

Um lastly I will say that we are hiring not just for this kind of work that you've um that you've seen being extended you know ever longer tasks ever more ambitious um RCTs um even more sources of evidence from which we can triangulate the truth about AI capabilities but also for for much more besides you can you can find this at meter.org/careers org/careers. In particular, I'm excited about research engineers, research scientists who might be um hiding in the current audience. We're excited not just, you know, for for research types with academic experience, but very much for scrappy startup people as well. And we're also hiring for a director of operations. And with that, thank you very much for listening. Heat