Joel Becker: METR stands for... first two letters, Model Evaluation. That is, we think about what their capabilities of AI models might look like today and tomorrow, as well as their propensities—what they'll actually do in the wild given that they have some level of capability. And then Threat Research, the final two letters. We try to connect those capabilities and propensities to particular threat models that we have in order to determine whether AI models pose enormous or catastrophic risks to society.

Allesio: Would you say that you've done a lot more ME and TR is like kind of the next phase, or is there TR side of work that I'm [missing]?

Joel Becker: I think there is TR. Some of the most publicized work does look more like the ME—looks like this Time Horizon stuff and the developer productivity RCTs. But there's this wonderful full report on our website, GPT-5 report, trying to make this more structured case that it doesn't pose these really large scale risks. Eventually coming to the conclusion that it doesn't, but it's worth thinking: why exactly is that the case? Why is it not able to do something really enormously wrong? Well, we find it's not capable enough on the basis of some of this capabilities evidence. But perhaps in future, we will think it's capable of doing pretty extraordinary things, the kinds of things that would be necessary to provide really serious threats. And then maybe you'd lean more on the propensities part—are the protections that we have against these dangerous capabilities sufficient for it not to pose an existential threat?

Swix: The model Time Horizon chart is probably the most quoted... What was the origin story of it?

Joel Becker: There's this internal METR PowerPoint from 2023 where we're trying to lay out our ambitions for what METR research might look like. There's this graph. It has a y-axis that's some measure of autonomous capabilities or dangerous capabilities, and then an x-axis labeled time or compute. And then it has a bunch of sort of scattered points that kind of go up and to the right. Many of METR's research bets have been trying to make this ever more concrete. And then when we actually did the full thing, when we had this y-axis—which turned out to be this task difficulty as measured by the length of time it takes for humans to do at which models can complete these tasks with 50% reliability—when we actually plotted that data over time, it turned out to be remarkably straight.

Allesio: How did you pick the tasks? You have some labels like "train classifier", "fix bugs". They all seem kind of arbitrary. What's the process of task selection?

Joel Becker: People are right to be worried about task selection. The aspiration was to pick economically valuable tasks relevant especially to sort of general autonomy and the threat models that we're primarily interested in. One misreading of the Time Horizon graph is this is referring to the full distribution of any tasks. In particular, tasks that are requiring of vision capabilities, they're probably much less capable today as measured by Time Horizon. We sample these tasks by having people inside of METR create them and by having a bounty so people from outside can provide them. In order to be able to scalably run our evals, it's helpful for the success on the tasks to be automatically gradable, which means some types of tasks are included and others are not.

Allesio: Any other disqualifiers? What are other things where you would expect the chart to be a lot worse?

Joel Becker: One thing is fairness—we want tasks to be in principle completable by a model. Could a low-context human who was sufficiently skilled at general skills, but maybe not the particulars in the background, be able to achieve success? That rules out a lot of real work because a lot of real work involves people having careful mental models of the situation that are not fully listed in an issue description. Another thing is that our tasks tend not to be so open-ended or interacting with the outside world—"messy", as we call it internally. Relative to tasks in the real world, our tasks are somewhat nicely scoped.

Swix: Let's talk about Opus 4.5. There's a very big jump. You're the first people to call out how much better Opus 4.5 was than the status quo... But it broke your trend line.

Joel Becker: There was some speculation that maybe the appropriate trend line to use is this faster 4-month doubling time, which Opus 4.5 would be perfect for. I was more of a believer in 7 months, so it is kind of falsifying my trend line in some way. I've seen some of the most talented engineers I know go from being picky about not using AI for coding to practically not writing a line of code. In some ways, I think the story of Time Horizon is that progress has been remarkably continuous over so many orders of magnitude of compute. But yeah, model capabilities are astonishing.

Swix: I would cosign what you said there with even very senior developers being finally pilled into agentic coding. Now very serious people are telling me that they want to commit to 100% agentic coding, which is not something that you would have said a year ago.

Allesio: How do you invalidate previous research? Take the developer productivity study—you had AI slowed people down. If you redo it with Opus 4.5, would you expect results to be dramatically different?

Joel Becker: We have been redoing it in the background. It is much harder to do it today. First, as AIs get better at coding, it's harder to find developers willing to be randomized to "AI disallowed". There's a selection issue. Second, a common workflow today is to work on multiple issues concurrently, which wasn't really true before. If you flip a single task to be AI allowed or disallowed, you're supposed to work on that single task. But that's not how developers are working today. These weren't threats to the previous study design in March 2025.

Allesio: Today I do a lot of just Claude Code and then review and iterate. It's much better and I don't know how to quantify it. People maybe overestimate... like 10x, but it's probably not right.

Joel Becker: There are a couple of side projects I have that I simply wouldn't be doing were it not for AI. In some sense, the speed up there is infinite, but these wouldn't really line up with additional value. I think that very bullish estimates of speed up today are inflated by what we document in that original paper—that people's expectations of speed up tend to be too optimistic. They also tend to be inflated by not quite grokking that the value of the additional tasks they're able to complete is lower than you might think. There's a reason they weren't doing them previously.

Swix: This concept of capability explosion... If you believe in emergence, it should be discontinuous in some sense.

Joel Becker: The thing that would really concern me is if AR&D was fully automated inside of a lab. The conditions are there for potentially a capabilities explosion. For things to be fully automated, 90% isn't enough; you need some full loop to be closed. Some people talk about software-only intelligence explosions—even holding hardware fixed, models improving themselves could lead to some extreme takeoff. Or instead, you need chip design or even chip production, and that's this larger loop that can close. Closing the chip production, software, and chip design loop is potentially very destabilizing.

Allesio: I think that is the actual paperclip factory. If you incentivize a model to go build its own compute, it will turn the planet into chips.

Swix: Do we want to talk about Manifold? You were the most profitable trader... what is your secret for Manifold markets alpha?

Joel Becker: The secret... actually mostly comes down to this one market where Manifold had opened up a charity program. The market is on how much is going to be donated. I noticed that you could manipulate this market by giving more to charity. So the strategy was to put a ton of Mana into the option that was above the linear projection. People keep betting against you because it doesn't look like that's happening. Eventually they caught on that someone's going to make this donation to move over the edge. I ended up donating something like $5,000. I won lots of fake internet points and became the number one most profitable trader.

Allesio: This is called prediction markets with high agency—you actually going [and making] the future what you make it.

Joel Becker: From METR, I think you're going to see more hopefully high quality capabilities evidence. We also have some monitoring research directions—thinking about if we can successfully apply safeguards to models attempting dangerous tasks. Maybe now is a good time to say that we are hiring. On my team, we're hiring for research engineers, research scientists, people from startup backgrounds. We are accepting a pretty wide range of people who get stuff done.

Allesio: Did you have a karaoke question? What is this karaoke thing that you organize?

Joel Becker: I've hosted a couple of these live band karaoke events. I think there's kind of transcendence to singing in person that the AI generator songs are not providing me. I look forward to seeing you both at the next one.