Julian: The talk about AI bubbles seemed very divorced from what was happening in frontier labs and what we were seeing. We are not seeing any slowdown of progress. We are seeing this very consistent improvement over many, many years where every, say, three, four months, it is able to do a task that is twice as long as before completely on its own. It's very hard for us to intuitively understand these exponential trends. If you manage to make everybody in society 10 times more productive, what kind of abundance can we achieve? What will we be able to unlock in the next 5 years? I think we can go extremely far.

Matt: Welcome to the Matt podcast. I'm Matt Turk from First Mark. Today my guest is Julian Schrittwieser, one of the world's most impressive AI researchers. Julian was a core contributor to DeepMind's legendary AlphaGo Zero and MuZero projects and he is now a key researcher at Anthropic. We covered the exponential trajectory of AI and his predictions for 2026 and 2027, the frontier in reinforcement learning and AI agents, and the science behind AI creativity and the famous move 37 from AlphaGo. Please enjoy this fantastic conversation with Julian.

Matt: Hey Julian, welcome.

Julian: Hey Matt, thanks for having me.

Matt: A couple of weeks ago, you wrote an incredible blog post that broke the internet entitled "Failing to Understand the Exponential Again." What is it that so many people are missing about the current trajectory of AI?

Julian: Yeah, it's funny that you bring up that blog post. I really didn't expect it to blow up that much. I actually had the idea when I was on holiday in Kyrgyzstan a few weeks ago on a very long car ride and then I started thinking about this and all the talk about AI bubbles I've seen on X and this discussion. And it seemed very divorced from what was happening in frontier labs and what we were seeing, and that made me start to wonder a bit like, is it that things are moving so fast that people maybe struggle a bit to extrapolate and understand intuitively.

Julian: "Oh, you know, maybe it's far away now, but it's doubling every so many months, which means that once it gets close to us, it's going to move past and become really good very quickly." And that reminded me a lot, in a different way, but like what happened during early COVID where we had a similar situation where, at the beginning, it's like very few cases. It's like, "Wow, it's never going to happen. It's only a few hundred people. Who cares?" But if you understand the math and if you look at it, right, it's like, "Oh, it's going to double every week, two weeks. Clearly, it's going to be a massive scale." But it's very hard for us to intuitively understand these exponential trends because it's just not what we're used to in our normal environment. And so that's what got me thinking, "Oh, is something similar happening here with AI?"

Julian: We are clearly, if you're looking at many benchmarks we have, many evaluations we have, we are seeing this very consistent improvement over many, many years where every, say, three, four months, it is able to do a task that is twice as long as before completely on its own. And so we can extrapolate this, right, and we see that, "Oh, in a year from now, maybe two years from now, the top models are going to be able to work completely on their own for like a whole day or more." Combined with this, combined with the fact that there's a huge number of knowledge-based jobs in the economy, knowledge-based tasks, and combined that in the frontier labs we are not seeing any slowdown of progress.

Julian: Just extrapolating those things together over a very short time like half a year, one year already, that is enough to know that there is going to be massive economic impact. That means if you look at current, if you look at OpenAI, if you look at Anthropic, if you look at Google, those evaluations, those revenue numbers are actually fairly conservative. I think some more thoughts, some things more I've seen more recently is that it's maybe actually even more interesting and more complex that while those frontier labs and frontier models are clearly very capable and on an extreme trajectory, there are a lot of other companies, right, that are trying to follow into the same AI sphere that may also have very high evaluation but not necessarily the revenues to support it.

Julian: And so it's possible that there may simultaneously be some sort of bubble in the wider ecosystem while at the same time the frontier labs are on a very solid trajectory, having a lot of revenue, making a lot of money. I think that may be quite of an unusual situation that in the past, you know, maybe in the dot-com bubble, maybe people were talking about the railroad rush and stuff like this, we did not see this bifurcation. So I think I've been thinking about this more and I think it's getting more and more interesting, the situation.

Matt: Fascinating. So you alluded to some of your predictions or extrapolations for '26 and '27. Do you want to unpack that? You had three of those.

Julian: Maybe calling it my prediction is giving myself too much credit, right? I will just say this. If you look for example at METER-Eval and you very naively extrapolate the linear fit, that's what you would expect to happen. And so I'm just going to be humble, right, and say like, "Oh, most of the time, I'm not going to be smarter than statistical models, statistical extrapolation of past trends that have been very consistent. So I'm just going to be very humble and despite what I might know all about research and what's happening." Probably the most likely, the best prediction I can make is actually just follow that data, that extrapolation and see where it's going to take us.

Julian: And yeah, in that case, if you roll this out, if you look at other benchmarks, I think we would have something like next year, maybe the models will be able to work on their own for a whole day worth of tasks. If you think of software, right, you might say like, "Oh, implement this entire feature, build out this entire set of the app." If you think of knowledge work, right, if maybe do like a whole research report, this kind of scale. The reason I think why task length specifically is interesting is because that's what allows you to delegate more and more work to language models to agents.

Julian: Even if you have a very clever model, but if it needs feedback or the interaction with you very often, then it really limits what you can delegate to it. If you need to talk to it every 10 minutes, right? Versus if you have something that can go for hours at a time, obviously, right? Then you cannot just have one copy of it. you can have a whole team that you delegate tasks to and manage them. And so I think that's why it's really critical that the models are actually smart enough, the agents are smart enough to work on their own to correct their own errors to you know iterate because that's really what allows you to delegate indeed.

Matt: Test length and time to complete as the metric for progress. So by mid-2026 you mentioned agents can work all day autonomously. Late 2026, at least one model matches industry experts across many occupations. And then by 2027, models frequently outperform experts on many tasks. So it's more time running and then generalization across the economy. And you mentioned GPQA-val, the OpenAI metric, as a benchmark to already see the progress towards multiple professions.

Julian: Yeah, I think the GPQA is like a super cool evaluation from OpenAI where they collected a lot of real world tasks from real domain experts to make sure it is actually representative of what you might do in the economy and then they evaluated a lot of models on those tasks. They compared them against real experts' performance to give us like a really good indication of how close, how far are we from having significant economic impact. So I think that's like a super cool evaluation.

Matt: The sort of obvious question is that GPQA-val and METER are carefully designed benchmarks. How do they predict production value once you add compliance, liability, messy data, the messy world, tool friction and all the things?

Julian: So I think like messiness and task length, like the time duration that you're able to work independently, are very similar or very correlated. So I think that's why it's interesting that METER tries to measure how long can the model go on its own because if you think about how do you come up with a task that takes a human 8 hours, 16 hours, right, you will have to include all these messiness and all this real world mess to even be able to measure it. But I think, ultimately, to go further, we really want benchmarks, we really want evaluations that come from the actual users, whether it's the industry, whether it's private users.

Julian: Because that's what ultimately matters, right? Like is the model helpful to you? Do you get something out of it, does your paperwork, helps you write something, fixes your codes, helps you study, right? I think that's the real proof. If you release a new model, right, do people start using it more? Do they really enjoy it?

Matt: Is there anything that would change your mind? Any kind of signal, whether that's real world adoption or benchmark performance, something that would make you more cautious about that exponential? Is there anything that would change your mind?

Julian: I mean many things, yes. I think like many of these things are internal only, right? I might look at our model pre-training, I might look at our fine tuning, I might look at RL things, you know, how do new runs go compared to past runs, do they match our expectations, the scaling continue. Then I might look at more public things of like are people actually able to use those models to be more productive. For example, at the beginning, right, there's always some adaptation period of, "Oh, you have a new tool like Claude." It takes you some time to figure out how to use it. But then in the medium term, in the long term, do people keep using it? Are they getting more and more productive using it? I think that's one of the things I look at, many, many signals.

Julian: I think when you do RL, when you do research, I think you get very much in the habit of looking for signals to prove yourself wrong because you often have ideas that you get attached to, but that's not a good way to do research, right? Most of your ideas are not good and they're not going to work. So you really want to figure out as quickly as possible whether this idea is any good or whether it's actually wrong. So you really get into this habit of like, "Oh, finding the fastest thing that will show that, 'Oh no, this is actually not true.'"

Matt: So by 2026-2027 in your extrapolation framework, AI becomes as good as humans. A key question of the moment is to which extent can it become better than humans? There's some chatter these days around move 37 and whether AI can create those like alien new paths to think and solve hard problems. So first of all, maybe remind the audience what move 37 is and then do you think that AI in its current state is going to be increasingly able to provide move 37 type thinking?

Julian: Yes. So I guess to give background, move 37, that was when we were building AlphaGo, an AI program to play the game of Go. That was in the year 2016, I think, and we're playing one of the best players in the world at the time because at that time, no AI program, no computer program had ever beaten the top human players at Go, and it was considered to be one of the most difficult board games, sort of like a real test of intelligence.

Julian: Move 37 happened during the second game of the five-game match where AlphaGo played a really unexpected, unconventional move that surprised many professional Go players. I think the commentator said it was truly creative, unexpected, and then ultimately AlphaGo ended up winning that game. And so I think that was for many people an early sign that AI is not just purely calculating, following an optimal path, but it can also do something that is truly novel and creative that you might not expect just from imitating its training data.

Julian: Yeah, I think that's very relevant in a modern context as well, right? Because as you alluded to, there is a lot of discussion of, "Oh, are LLMs just parroting the training data? Can they actually do novel things?" For me, as somebody who has been doing research a long time, I think it's pretty clear that these models can do novel things. And that's why they are so useful to many people, whether it's writing code for you, because obviously, right, you're not just writing code that you already have, that wouldn't be very interesting, or helping you write a paper. The way those models are trained, they're literally trained to generate a whole probability distribution, which means that when we sample from them, we can generate an infinite amount of novel sequences from them.

Julian: For the question of something like move 37, I think there it really comes down to, is it something that is sufficiently creative and impressive that we can easily recognize it? In the game of Go, that was pretty ideal conditions because it's very clean, very abstract, you can really, each move is very impactful, so you can really see it clearly. I think to have the equivalent for our modern models, you need the combination of a task that is sufficiently difficult and interesting, and a model that is both able to create sufficiently diverse and creative ideas and also able to evaluate accurately how good they are so that it can go down increasingly novel paths while making sure that this novel path is actually interesting and useful. Creating novel things is actually very easy with language models. The hard part is creating novel things that are useful and interesting.

Matt: Extrapolating this further, there's the idea of creating novel science. So not just one move but like a whole new idea, new concept. What's your current take on this? So I think AlphaCode and AlphaTensor proved that you can discover novel programs and algorithms. Very recently, I think last week, there was some news of Google DeepMind and Yale in the biomedical field coming up with brand new things as well. So do you think that's accelerating and that AI is in the process of discovering novel science?

Julian: So I think we're absolutely at the stage where it is discovering novel things and we're just moving up the scale of how impressive, how interesting are the things that it is able to discover on its own. And so I think it's highly likely that sometime next year we're going to have some discoveries that people pretty unanimously agree that, "This is super impressive." I think at the moment we're more at the stage of, "Oh, it came up with something, but there's debate about it." But yeah, I'm not very worried because I see this process continuing and then once it gets clear enough, there's less need to argue about it.

Matt: How far do you think we are from an AI winning the Nobel Prize?

Julian: Yeah, I think that's a really interesting question, right? Because we had a Nobel Prize for AI with AlphaFold, of course. And so I think the next very interesting point is going to be when can AI on its own make a breakthrough that is so interesting that it would win a Nobel Prize. And I think my guess for that level of capability might be maybe 2027.

Julian: I think we're probably not going to find out for quite some time afterwards because of the delay in getting prizes, but I think by 2027, 2028, I think extremely likely that the models will be smart enough and capable enough to actually have that level of insight on that level of discovery.

Matt: Amazing.

Julian: I think yeah, Nobel Prize, right? It's like the Fields Medal for math and all these kind of advances. I think that's what I'm truly excited about actually is AI that can help us advance science and really unlock both all the mysteries of the universe and all the improvements in living standards and abilities for us that we could have if we understood the world better.

Matt: All right. So extrapolating this even further then we get into the "AI 2027" thing that you probably saw. So this general idea of if AI can create novel science then AI can create AI researchers and basically AI can create itself which effectively leads to a discontinuity moment. So I don't know if that in the blog post that's the singularity or whatever. But does that strike you as somebody who's as deep in the field as possible as something that is possible in the short term or are they counterbalancing forces that makes that path to discontinuity harder as you get closer?

Julian: Yeah, I think a true discontinuity is extremely unlikely from, you know, obviously AI researchers are already using AI to accelerate themselves and so what's already happening and what is likely to continue to happening is that we see a smooth improvement of productivity. And then the main open question is how does the difficulty of improving AI keep scaling because a very common effect and a very common issue in many scientific fields is that we find all the easy problems first.

Julian: And then as we continue exploring the problem, the field, it gets more and more difficult to make advances. So in my mind the main question is do these two trends balance each other out so that the AI makes us increasingly more productive so that as it gets more difficult to make advances we just sort of about stay on trend and then we keep improving roughly linearly. Or it is still too difficult and then eventually after some time we still see a slowdown. But it seems quite unlikely to me that we improve in productivity so much that we can actually accelerate. That would be very unlike any other scientific field.

Julian: The normal course in many scientific fields is that we actually need to exponentially increase the research effort just to keep making progress and find new insights. For example, if you look at pharmacology discovering new drugs, it's nowadays in the range of billions of dollars to discover a new drug versus maybe 100 years ago a single scientist could discover the first antibiotic by accident. It's not that we will be surprised by a sudden takeoff in progress where, "Oh, we're just doing our research and suddenly our model is like 10x better." We will be seeing advanced signs of, "Oh, we're making faster progress every single week. We can see something is happening." Maybe we decide to pause if we don't understand what's happening.

Matt: Do you think the current approach to modern AI systems which effectively is pre-training plus RL does that take us to where we want to be? Whether we call it AGI, ASI, it's unclear what any of the things mean. But do you feel that this paradigm is the right one or do we need to come up with a different architecture all together post-Transformers or otherwise?

Julian: I think that's a great question and I think it hugely depends on what you mean by where do we want to be. So I think if you're thinking of, "Oh, we want some kind of system that can perform at roughly human level in basically all tasks that we care about productivity wise," then I think yeah, it's extremely likely that the current approach, pre-training or Transformers, is going to get us there. If what you care about is, "Oh, we want to have a model of intelligence that is conscious in the same way we are," or more abstract qualities like this, I think that's maybe more uncertain.

Julian: Right, and I think this is where a lot of the confusion and disagreement comes from. As you alluded to, AGI, ASI, people talk about very different things and they have very different things in mind when they say, "Oh, the current paradigm is going to get there, it's not going to get there." I often like to not use the term AGI or ASI and just talk very concretely about what problem are we solving, what task are we solving, what quality are we interested in, because I find that often makes the actual disagreement much more obvious. But yeah, I think if you're just thinking of in terms of like, "Is this going to help us be massively more productive? Is this going to massively accelerate scientific progress?" then I think definitely the current approach will get there.

Matt: And given how extremely deep you are, I cannot resist asking you sort of the trendy question du jour based on Richard Sutton's recent appearance on Dwarkesh podcast. Do you think that the models of the future will be trained in RL from scratch and that actually having pre-training in addition to RL is the wrong way to go?

Julian: Personally, I think that's unlikely. Not because pre-training is strictly necessary. I think we may well be able to train something completely from scratch as we've been able to do in other domains. But more because pre-training on this vast data sets we have just brings us so much value that we would from a practical point of view not want to give it up.

Julian: So we might well do some agents that are trained from scratch out of scientific interest. It could be very interesting to learn about what would a non-human intelligence look like. But from a pragmatic point of view, I definitely think we would keep using pre-training data. Not just from an efficiency point of view as well, but also I think there is interesting safety angles because by pre-training on all this human knowledge, we're implicitly creating an agent that has similar values as we do and I think that is quite valuable for aligning a highly intelligent agent. If you already start out by caring about sort of the same rough set of values, that makes things much easier than if you create an arbitrary alien intelligence that may have completely different values. Despite having done a bunch of from-scratch RL in the past, I think I'm often quite pragmatic about this.

Matt: I'd love to put a pin in that specific discussion about alignment and what we do to ensure safety to later in the conversation because I think that's a super interesting vein. But maybe to switch tacks for a minute. I'd love to go into a little bit of your story and then the monumental body of work that you've done at Google DeepMind before joining Anthropic around AlphaGo, AlphaZero, MuZero. So maybe just the three, four minute version of your personal story from when you were a kid. What was the path that led you to become a world-class AI researcher?

Julian: Yeah, actually when I was a kid, I didn't have any expectations of becoming an AI researcher. I was always very interested in computers and I grew up in the Austrian countryside in a small village. So, you know, it's not like there was a huge amount of things happening, but computers were always very interesting to me. It's like this connection to the wider world, to all these other interesting things.

Julian: And I was very interested in computer games as well. And I think that's the first time I became interested in programming because I wanted to make my own games, which I think is very common in people who get into programming. But I somehow I always got distracted by the technical aspect of, "I'm going to build a very general game engine that can run any kind of game." And so I never actually ended up making any game. I learned a lot about making game engines and different technologies and that's how I ended up studying computer science eventually in Vienna.

Julian: Yeah, there was like a classical computer science degree and then by chance after my first year in my first summer holidays I had an internship with Google and that's when I realized, "Oh wow, these guys are doing really interesting things." They, you know, that's where their big clusters, the tens of thousands of machines are. That's the first time I radically changed my plans from wanting to stay in academia and you know, I had originally thought, "Oh, maybe I do a PhD." That's when I changed, "Oh no, actually I just want to join these guys at Google and I will finish my degree as quickly as possible."

Julian: And so that's actually when I got my full-time position at Google, finished my degree the next year and then moved to London. So I was just working as a normal software engineer at Google, working actually in advertising, which I wasn't super excited or interested in. So the technology was interesting, right? It's like these huge systems and Google has famously great technology. But actually after a year-ish of this, I was pretty done on board of advertising and so I was actually planning to leave Google and thinking of maybe joining a hedge fund, going into finance.

Julian: When by chance I saw an email in my work inbox that this guy Demis was going to come to the office and give some talk about Atari and video games and AI. And it was actually a day off because I was visiting a friend somewhere else in England. But that email looked so intriguing that I was like, "Oh no, I'm going to have to like take the train back to the office right now and like see this talk."

Julian: And yeah, like I'm really glad that I saw this email and I did go back because that's the moment where I decided, "Oh no, I'm not going to go into finance. I'm going to move to DeepMind. I'm going to join these guys because this looks clearly super interesting, super amazing. They are doing really interesting research."

Matt: All right. Tell us the story of AlphaGo, AlphaGo Zero, Alpha Zero, MuZero. Because it feels like it's fundamental AI knowledge that everybody who has an interest in the space should know about, should understand the progression in particular. So starting with the beginning of AlphaGo, you alluded to it a second ago, but what did it do? How was it trained and then how did that evolve with each version?

Julian: AlphaGo, I think at that moment in time, Go in the machine learning community was this really big target where everybody felt like, "Oh, it's this big unsolved challenge." ImageNet had just happened before so clearly models were starting to do something with images and being able to recognize them and predict them. And if you look at the Go board the right way, it looks a lot like one of those images that you classify. So there was a lot of momentum around using neural networks to somehow play Go.

Julian: And then at the time, David Silver and Aja Huang had been working on Go, I think both of them had been working on Go for quite a while, had published some very interesting papers. And that's when the idea of using Monte Carlo tree search with deep networks came together. So the idea was that to train a deep neural network to predict which moves you might want to play and whether you're winning or losing the game, and then use the tree search to really make a big plan of what are all the possibilities in the game. How would it go for you if you chose a certain move or a different move? How would the opponent respond?

Matt: And to explain this in super plain English, the term search in this case is as you said is research, is not what people normally think of search which is searching a corpus. This is searching a series of options effectively. Is that the right way to think about it?

Julian: Yes. It's quite literally what you might do when you play a game of chess, when you play any board game. It's quite literally thinking of, "What move am I going to do? What move is my opponent going to do in return?" and then thinking about many possible moves like that and mapping out all the possibilities in the future.

Matt: So deep learning plus search. What was AlphaGo trained on?

Julian: Initial training phases of AlphaGo were on some human amateur games if I remember correctly.

Matt: Mhm.

Julian: So basically just predicting if you have humans playing many games of Go, try to predict at each turn in the game what move would they have played? And it turns out that if you train a deep network to do that, you can get something pretty decent, like amateur Go level.

Matt: Mhm. Mhm. But not good enough to actually beat a really strong player.

Matt: And by the way, just for the lore of it, did you guys have any sense that it was going to crush Lee Sedol? So, the famous Go player that you mentioned earlier in the conversation? Was it obvious before? Was that a surprise?

Julian: We thought we had a pretty good chance, but we were very nervous about like, "Are we going to win? Are we not going to win? Are we going to lose?" Yeah. We actually had some bets beforehand of like how many games we're going to win or lose. Like I think it was very ambitious to put the match as early as we did. If we had wanted to be a bit more safe, we may have tried to do it a few months later. And I think if we had done it a few months earlier, we would have probably lost.

Julian: So it was very knife edge of, I guess, which also made it much more interesting for us, right? Because it really means that each game is like a nail-biter of, "Oh, what's going to happen? Are we going to win? Are we going to play a dumb move? What's going to happen?" So that was very exciting.

Matt: AlphaGo Zero which was I believe the year after. How was that different? What was the progression?

Julian: Main change between AlphaGo and AlphaGo Zero was to remove all the human Go knowledge. So instead of starting by imitating human Go games, we were training it just from scratch playing only against itself and rediscovering basically all Go, completely figuring out from scratch how to play.

Matt: Did you give it the rules of the game?

Julian: We didn't give the rules of the game to the network per se, but we used the rules of the game to score the result. So basically, he would play and we would tell it who won, who lost, or you cannot make this move.

Matt: So the next hop was Alpha Zero which was a year or two later. How is that different?

Julian: So for Alpha Zero, the idea was, well, obviously Go is a really beautiful game but ultimately we would like to do something more general, right? So can we remove anything that is Go-specific and verify that the algorithm can actually solve more problems? And in that case, we did that by trying to solve both chess, Go, and shogi, which is a Japanese chess basically, with the same algorithm, the same network structure, just by running it in different games and also making it much simpler, elegant, and faster. So basically that was really laying the groundwork for applying the algorithms to solve real problems.

Matt: And then the next stop in the journey was MuZero. And just to bring it home for people, you were I believe second author on AlphaGo Zero and you were the lead author on MuZero which in the world of AI is as big a deal as it gets. So I'll say it so you don't have to say it. So MuZero, what was how was that different?

Julian: So the main motivation I had for making MuZero was that if you want to solve many real world tasks, you have no way of perfectly simulating what's going to happen. And you know, if you play a board game, obviously you know if you make this move, what's going to happen. It's like the piece is going to go there, it's going to take a piece, whatever, right? But if you actually want to solve something like a robotics task or anything more complicated, it's impossible for you to simulate what's going to happen accurately.

Julian: And also we as a human we don't do this right? We just imagine in our head of, "Oh, if I'm going to say this then he's probably going to respond in that way." This meant that Alpha Zero as it was could not be applied to such problems because it required some way of simulating the game, scoring the outcomes. And the idea with MuZero was that, well, we already have a deep neural network, right? These networks can learn a lot of things, so why not let it, why not teach it to predict the future of the environment, the future of the world? Why not make the model be able to learn for itself, what is going to happen after each action it takes?

Matt: After that, you also applied this to code and math. So that was AlphaCode and AlphaTensor. So like zooming out a little bit, that evolution of reinforcement learning in games and then code and then math. What did you learn about the general power of search and learning that is today relevant in modern agentic AI systems? How did that whole body of work translate to what you are doing today?

Julian: So games are a really good sandbox to learn very quickly about a lot of the reinforcement learning science. You know, the algorithms that work well, the kind of problems that we encounter, the even from a technical point of view, how do we build a learning system that spans many data centers, uses tens of thousands of machines? Because games are very clean sandbox, very clean environments, so we can make many good experiments.

Julian: And then now that we have a much more general model, right, the language models can do almost any task, but they're much more complicated. They're much slower to experiment with, we can apply those same lessons of, "Ah, you know, we know how to build a really robust reinforcement learning infrastructure." And now we can build the same one for language models or like, "You know, we know if you do this kind of RL, then the model will learn how to exploit the reward." And so we can apply the same lessons, the same mitigation techniques to the language models.

Matt: If I understand correctly I think MuZero had a learned world model.

Matt: Basically sort of rehearse the future for lack of a better expression. So do modern LLM agents have anything like that? Do they have an internal world model that lets them preview actions before they commit?

Julian: So I think yes, I would say that language models have not an explicit world model but they do have an implicit model of the world because to be able to predict, "You know, what is the next likely word in this sentence? How is this paragraph going to continue?" they need to internally model, "You know, what is the state of the world that makes this person say that thing?" And so it's actually somewhat similar to MuZero in the sense that MuZero also only had an implicit world model. You know, it was never trained to predict, "You know, what does the screen actually look like if you take an action."

Julian: It was also only trained to implicitly predict, "If I take this action, what is the next action I should take or is it going to be good or bad for me?" So in both those cases, you have an implicit representation of the world in your model that you can use to make predictions, but you're not actually reconstructing the full state of the world because reconstructing the full state of the world can be very expensive and complex.

Julian: If you think about super high resolution video, audio signals is a very large amount of data that probably you don't actually need. If you think of human attention, we are only aware of a very small subset of what's actually going on all around us all the time because that's the most relevant information that we actually need to make decisions.

Matt: And that goes back to the prior discussion about pre-training. So the reason why pre-training and RL work well together is that you have that world model that's implicitly embedded into the corpus. Although the argument against it is that it's what humans think the world model is as embodied by language versus what the world model actually is. And that's my understanding of the debate.

Julian: I mean for the debate, I think different people have different points of view so I don't want to speak for anybody. But yes, I think like pre-training on this rich knowledge gives you some representation of the world already so that when you actually start to act and interact with the world you can very quickly make meaningful decisions, meaningful actions.

Julian: I like to think of it in the similar way if you look many animals when they are born they very quickly know how to move, how to run even. Right? If you look at gazelles for example in the savannah, in a way that is like clearly they did not have time to really learn this from scratch, right, few minutes or hours. And in their cases they did not do pre-training but they have some evolutionary encoded structure in their brain because clearly it is very beneficial to have some sort of knowledge to make your learning more efficient.

Matt: Yeah. just RL in nature would lead to not so good results. Like if you're a gazelle and you have to AB test whether to run towards the lion or away from the lion.

Julian: Exactly. It's like the thousands of generations of gazelles acquired this knowledge over time. It was encoded in their genes and their brain structure in some way, right, and then you get to start on top of that. I think the main challenge or the main thing you need to watch out for is that you don't over-encode or you don't restrict your search base too much. If your pre-training, if your prior knowledge prevents you from exploring something that might be the correct course of action, that will be bad. So there, you know, there is some danger there you have to be aware of.

Matt: Let's spend a minute on eval and we touched upon this a little bit but just to give it some proper space. So there was in your blog post that we talked to at the very beginning of this conversation this concept of external benchmark and then you quoted your piece Goodhart's law. So what does, first of all, what is Goodhart's law and then how should labs compare results so that it doesn't end up with this kind of leaderboard theater that we've seen a little bit in the last couple of years?

Julian: Yeah. As a Goodhart's law basically says that any measure that becomes a target stops being a good measure and you can think of that intuitively that if you start paying for example programmers based on how many lines of code they write, well suddenly they will discover many ways to add more lines of comments which is completely useless. And this is a very general effect that obviously, right, if you give people an incentive that they should optimize, they will try very hard. Yes.

Julian: And we also see this with language model benchmarks. Of course, people want to get promoted. They want to launch their model. So any benchmark that is too easily measured or that has a lot of attention on it, people will optimize very hard for it. Which means that probably the model will look very good at that benchmark. But if you then use it for your own task, you might get different performance. Yeah. You ask about what do we do about this? It's very hard to prevent people from optimizing on the benchmark.

Julian: So one possibility is just periodically create completely new held out benchmarks that nobody has seen before and that gives you a fairly good estimate of model performance. So I know for example like a lot of researchers have their own toy problems that they use to test all the models precisely for that reason. So that you know this is a problem, a set of problems that nobody has seen. you have a pretty good guess that it's going to give you an unbiased estimate. If you're like an individual of your company trying to decide which model to use, it's probably something similar. Just make your own internal benchmark that really represents what you care about and then measure on that. And I think that's likely to be the most objective, most accurate way of measuring.

Matt: Internally. What does that look like at a place like Anthropic or previously DeepMind? Is there, I mean I know there are teams that are focused on evals. How do you think about what works, what doesn't in terms of internal evals?

Julian: It definitely used to be easier to have good evals. You know, 5 years ago the tasks were doing, I think it was easier to measure model performance. I think nowadays it's much more difficult and I think we try to not over rely on evals so much because it's quite hard for example to measure how good is this model really at writing code. Yeah, I think it's one of the big unsolved or very important problems in the field of making really good evals that are both cheap to run, reliable, and accurate.

Julian: Because it's easy-ish to make an eval that takes one of those but to get all three is quite hard. For example, in the beginning we were talking about OpenAI's GPQA eval. And that one is like, you know, it's very accurate and unbiased but it's very expensive to do because what it actually involves is taking human experts, having them do the task and then compare the model task to the experts and rate it with multiple people. So it's very accurate but it's like extremely expensive to do.

Matt: And related to that topic of evals, what's the latest in terms of our ability or I should say your ability to truly understand how models work? To the general field of mechanistic interpretability. You alluded to the fact earlier that RL, if I understood correctly, sometimes make it a bit harder because it does things occasionally in a more inscrutable way. My words maybe not yours. So what is the latest and indeed does RL make things harder or easier?

Julian: Oh, so what I meant before is that debugging RL in general, completely unrelated to interpretability, is harder because there are more moving parts. But it is also true that if you're not careful with RL, you can make interpretability harder.

Julian: For example, one common thing with modern models is they do reasoning with the chain of thought. You could look at the chain of thought to see what are the model internal thoughts and then you could also have a thought that, "Oh, maybe I should use that as a reward signal in RL and punish the model if it thinks the wrong thing." But then suddenly you completely destroyed your interpretability angle. So you sort of have to be careful that you don't do RL on the signals that you actually want to use to interpret what the model is thinking of doing.

Julian: That said, I think yeah, there's some extremely exciting interpretability things happening including mechanistic interpretability. I think last year, I think before Johnic maybe even, there was a super cool Golden Gate Claude model where they found the neurons in Claude that were responsible for the Golden Gate concept and then modified them to make a version of Claude that really loved the Golden Gate Bridge in San Francisco. And so that's like a really vivid example of, "Ah, you know, we really understand what's happening in this model." And what better way is there to verify that understanding than actually changing the behavior of the model? And so I think that's like a super important direction for safety.

Julian: As the models get smarter, we really need to be able to understand what is the model thinking internally. You know, what is the values it has? Is it lying to us? Is it actually genuinely following the instructions? And so I think like, you know, definitely extremely important area to invest in and work in. I think that especially if like, you know, people interested in working AI or doing AI research, I think interpretability is a great area to get into.

Matt: Yeah, perfect segue for the last part of this conversation. I'd love to zoom out and talk about the impact of AI. So if we think that we are on the exponential and that things are going to only accelerate from here, what does that mean? And certainly safety and alignment which is a core value at Anthropic, hopefully in other parts of the field as well, but like Anthropic is particularly vocal about safety and alignment, let's say. How does that actually manifest? So we just talked about interpretability. What, for people who are concerned that this is going too fast and that we collectively are creating a monster, can you give us a glimpse into the kind of work that is done for alignment and safety at a place like Anthropic?

Julian: Yeah, I think like the focus on sort of safety alignment pervades all of Anthropic and there's very rigorous processes where we train a model, whenever we want to release a model, both to analyze the capabilities of the model, verify the alignment of the model. Ensure that it does not do harmful things on its own. Ensure that it does not enable malicious users to do harmful things. And to the point where if we are unsure about the safety of model, we will delay the launch and until we're sufficiently sure that is actually harmless, we will not launch and release a model, which may show that people take the safety much more seriously than any financial return or revenue.

Julian: I think yeah, also in terms of research and resources, the teams working on safety and interpretability are a big focus of the company which gives me a lot of confidence that we're actually care about this and put a lot of effort into it.

Matt: And at a more technical level and to tie back an earlier part of the conversation around when we're discussing the pre-training and safety. So is safety and alignment an RL problem? And by that I mean the beauty of having pre-training is that you import that world model as we're discussing but arguably you also import into your brain a lot of bad stuff if you collect data from the internet as we know there's good things but also a lot of toxic content. So is alignment largely using RL to get rid of the bad stuff that is built into the pre-training?

Julian: We can definitely use RL to like shape the model behavior and ensure that for example given adversarial, given bad input, it sort of behaves safely or knows that he can refuse or is robust to attempts to hack the model. Yeah, I wouldn't view it alignment just like an RL problem. I think it sort of it goes throughout the whole stack. You might for example filter the pre-training data in some way. You might after training you might have classifiers that look at the model, monitor the model behavior to ensure that it is actually aligned.

Julian: You might when you write the system prompt for the model that you use. You might put safety guidelines in there. So I think safety alignment it really pervades the whole of research and the whole of you know product and deployment it's not just isolated into any one part.

Matt: And then another super interesting topic in the same vein of like the impact of AI is obviously the discussion around jobs. So if as per the GPQA discussion the agents are becoming just as good or better than humans obviously what does that mean for all of us in terms of our jobs? What have you learned after the experience of Alpha Zero, AlphaGo that could give us a glimpse into what may happen once we all have super powerful agents do our jobs?

Julian: So I think the first thing that we didn't talk about yet so far is that artificial intelligence is quite, I mean this may sound a bit simplistic, but it's quite different than human intelligence. So we can see that right that the model may be much better at us on some tasks like calculation obviously and like much worse than us at other tasks.

Julian: So it is not, I don't think it is at all going to be any like one for one replacement. It's going to be much more complimentary of, "You know, the model is really good at something that maybe I really don't like doing or I'm not interested in or I'm very bad at," and then I'm much better than the model as a mother part. And so I think it's going to be like a gradual process of we're all going to incrementally start using models more and more to improve our own productivity.

Julian: Rather than have a model that one for one is able to do exactly the set of things we can do. And so for example, I use Claude all the time to for example refactor code or maybe write some front end code that I don't want to write or at the same time there's other parts where I'm clearly much better putting than Claude still. So there is a synergy of use the best most productive skills. I think I guess economists call it like comparative advantage but like there is this long process of we'll both sort of improve our productivity incrementally.

Julian: And I think that process is going to give us some time of figure out politically and figure out economically how do we want to you know benefit from this massive productivity increase. You know, even independently from AI, right? The promise of technology has long been that, "Oh, we're going to be all so productive, so wealthy that we need to work much less." Yet, mysteriously, right, we all have like 40 hours working week for decades. And so, you know, I think it's much more like a political social problem of like figuring out how do we actually benefit from all these improvements and like, you know, bring the increases in wealth and productivity to everybody and it's much less a technological problem.

Julian: Mhm. Also means that we can't really solve it with technology. We have to solve it at like a sort of a democratic political level. How do we spread these benefits?

Matt: Do you think that that increases inequality? So, as you think about the impact of AlphaGo and MuZero, what happened to the top Go players and what happened to the top chess players? Did they disappear or did they get enhanced and better?

Julian: Yeah, I think at least in the case of chess and Go there has been like more interest and it has become much easier for people to study how to play Go, how to play chess because now you don't need to find an expert tutor, anybody can practice on their own, right, spend a lot of time and I guess like chess streamers are very popular on Twitch, right? And similarly, right, like a lot of students are using language models to study. I think also for coding, right?

Julian: Claude, code, these agents, they raise the bar of what anybody who has an idea can accomplish on their own. I think the larger picture whether it increases or decreases inequality is quite hard to forecast. It both sort of raises the floor of what any person can accomplish but it also gives very productive people an ability to be even more productive. It's possible that we see quite a difference between countries depending on the taxation, social redistributive system that they have in whether inequality increases or decreases for example.

Julian: Overall I'm quite excited that it is very much non-zero sum, is very much increases the total wealth available in society. I think if you think about progress, if you think about prosperity, that is the most important thing. Like redistributing the pie is kind of a loser's game. To get more wealthy, we really need to grow the pie. You know, if you think of the agricultural revolution, the industrial revolution, the reason why we have much better lives nowadays is because we're so much more productive. We have so much more wealth.

Julian: And so that's the key step we want to unlock. If we manage to make everybody in society 10 times more productive, what kind of abundance can we achieve? I think that's the key question, right? What advances does that unlock in medicine? You know, curing diseases, halting aging. What does it unlock in terms of energy? Obviously, right, we have like climate crisis. We need more energy rights to sustain our lifestyle.

Julian: What advances in material science can we have? All of those are basically bottlenecked on how much intelligence we have access to and how can we apply it. So I'm yeah incredibly optimistic about like what will we be able to unlock in the next 5 years. I think we can go extremely far.

Matt: Well that feels like a wonderful place to leave it. Thank you so much Julian. This was absolutely fantastic. Thank you for spending time with us.

Julian: Yeah, thank you for all the exciting questions and giving me the time.

Matt: Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks, and see you at the next episode.