Today I'm speaking with Andrej Karpathy. Andrej, why do you say that this will be the decade of agents and not the year of agents?

First of all, thank you for having me here. I'm excited to be here. The quote you've just mentioned, "It's the decade of agents," is actually a reaction to a pre-existing quote. I'm not actually sure who said this but they were alluding to this being the year of agents with respect to LLMs and how they were going to evolve.

I was triggered by that because there's some over-prediction going on in the industry. In my mind, this is more accurately described as the decade of agents. We have some very early agents that are extremely impressive and that I use daily—Claude and Codex and so on—but I still feel there's so much work to be done. My reaction is we'll be working with these things for a decade. They're going to get better, and it's going to be wonderful. I was just reacting to the timelines of the implication.

What do you think will take a decade to accomplish? What are the bottlenecks?

Actually making it work. When you're talking about an agent, or what the labs have in mind and maybe what I have in mind as well, you should think of it almost like an employee or an intern that you would hire to work with you. For example, you work with some employees here. When would you prefer to have an agent like Claude or Codex do that work? Currently, of course they can't. What would it take for them to be able to do that?

Why don't you do it today? The reason you don't do it today is because they just don't work. They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working. It will take about a decade to work through all of those issues.

Interesting. As a professional podcaster and a viewer of AI from afar, it's easy for me to identify what's lacking: continual learning is lacking, or multimodality is lacking. But I don't really have a good way of trying to put a timeline on it. If somebody asks how long continual learning will take, I have no prior about whether this is a project that should take 5 years, 10 years, or 50 years. Why a decade? Why not one year? Why not 50 years?

This is where you get into a bit of my own intuition, and doing a bit of an extrapolation with respect to my own experience in the field. I've been in AI for almost two decades. It's going to be 15 years or so, not that long. You had Richard Sutton here, who was around for much longer. I do have about 15 years of experience of people making predictions, of seeing how they turned out. Also I was in the industry for a while, I was in research, and I've worked in the industry for a while. I have a general intuition that I have left from that.

I feel like the problems are tractable, they're surmountable, but they're still difficult. If I just average it out, it just feels like a decade to me.

This is quite interesting. I want to hear not only the history, but what people in the room felt was about to happen at various different breakthrough moments. What were the ways in which their feelings were either overly pessimistic or overly optimistic?

Should we just go through each of them one by one? That's a giant question because you're talking about 15 years of stuff that happened. AI is so wonderful because there have been a number of seismic shifts where the entire field has suddenly looked a different way. I've maybe lived through two or three of those. I still think there will continue to be some because they come with almost surprising regularity.

When my career began, when I started to work on deep learning, when I became interested in deep learning, this was by chance of being right next to Geoff Hinton at the University of Toronto. Geoff Hinton, of course, is the godfather figure of AI. He was training all these neural networks. I thought it was incredible and interesting. This was not the main thing that everyone in AI was doing by far. This was a niche little subject on the side. That's maybe the first dramatic seismic shift that came with the AlexNet and so on. AlexNet reoriented everyone, and everyone started to train neural networks, but it was still very per-task, per specific task.

People became very slowly interested in agents. People started to think, "Okay, maybe we have a check mark next to the visual cortex or something like that, but what about the other parts of the brain, and how can we get a full agent or a full entity that can interact in the world?"

The Atari deep reinforcement learning shift in 2013 or so was part of that early effort of agents, in my mind, because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments. At the time, this was Atari games. I feel that was a misstep.

It was a misstep that even the early OpenAI that I was a part of adopted because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games, and OpenAI was doing a lot of that. That was another prominent part of AI where maybe for two or three or four years, everyone was doing reinforcement learning on games. That was all a bit of a misstep.

What I was trying to do at OpenAI is I was always a bit suspicious of games as being this thing that would lead to AGI. Because in my mind, you want something like an accountant or something that's interacting with the real world. I just didn't see how games add up to it. My project at OpenAI, for example, was within the scope of the Universe project, on an agent that was using keyboard and mouse to operate web pages. I really wanted to have something that interacts with the actual digital world that can do knowledge work. It just so turns out that this was extremely early, way too early, so early that we shouldn't have been working on that.

Because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, your reward is too sparse and you just won't learn. You're going to burn a forest computing, and you're never going to get something off the ground. What you're missing is this power of representation in the neural network. For example, today people are training those computer-using agents, but they're doing it on top of a large language model. You have to get the language model first, you have to get the representations first, and you have to do that by all the pre-training and all the LLM stuff.

I feel maybe loosely speaking, people kept trying to get the full thing too early a few times, where people really try to go after agents too early, I would say. That was Atari and Universe and even my own experience. You actually have to do some things first before you get to those agents. Now the agents are a lot more competent, but maybe we're still missing some parts of that stack.

I would say those are the three major buckets of what people were doing: training neural nets per-tasks, trying the first round of agents, and then maybe the LLMs and seeking the representation power of the neural networks before you tack on everything else on top.

Interesting. If I were to steelman the Sutton perspective, it would be that humans can just take on everything at once, or even animals can take on everything at once. Animals are maybe a better example because they don't even have the scaffold of language. They just get thrown out into the world, and they just have to make sense of everything without any labels. The vision for AGI then should just be something which looks at sensory data, looks at the computer screen, and it just figures out what's going on from scratch. If a human were put in a similar situation and had to be trained from scratch… This is like a human growing up or an animal growing up. Why shouldn't that be the vision for AI, rather than this thing where we're doing millions of years of training?

That's a really good question. Sutton was on your podcast and I saw the podcast and I had a write-up about that podcast that gets into a bit of how I see things.

I'm very careful to make analogies to animals because they came about by a very different optimization process. Animals are evolved, and they come with a huge amount of hardware that's built in. For example, my example in the post was the zebra. A zebra gets born, and a few minutes later it's running around and following its mother. That's an extremely complicated thing to do. That's not reinforcement learning. That's something that's baked in. Evolution obviously has some way of encoding the weights of our neural nets in ATCGs, and I have no idea how that works, but it apparently works.

Brains just came from a very different process, and I'm very hesitant to take inspiration from it because we're not actually running that process. In my post, I said we're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution. We're doing training by imitation of humans and the data that they've put on the Internet. You end up with these ethereal spirit entities because they're fully digital and they're mimicking humans.

It's a different kind of intelligence. If you imagine a space of intelligences, we're starting off at a different point almost. We're not really building animals. But it's also possible to make them a bit more animal-like over time, and I think we should be doing that.

One more point. I do feel Sutton has a very... His framework is, "We want to build animals." I think that would be wonderful if we can get that to work. That would be amazing. If there were a single algorithm that you can just run on the Internet and it learns everything, that would be incredible. I'm not sure that it exists and that's certainly not what animals do, because animals have this outer loop of evolution. A lot of what looks like learning is more like maturation of the brain. I think there's very little reinforcement learning for animals. A lot of the reinforcement learning is more like motor tasks; it's not intelligence tasks. So I actually kind of think humans don’t really use RL, roughly speaking.

Can you repeat the last sentence? A lot of that intelligence is not motor task…it's what, sorry?

A lot of the reinforcement learning, in my perspective, would be things that are a lot more motor-like, simple tasks like throwing a hoop. But I don't think that humans use reinforcement learning for a lot of intelligence tasks like problem-solving and so on. That doesn't mean we shouldn't do that for research, but I just feel like that's what animals do or don't.

I'm going to take a second to digest that because there are a lot of different ideas. Here’s one clarifying question I can ask to understand the perspective. You suggest that evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world. The difference is that evolution has to be titrated in the case of humans through three gigabytes of DNA. That's very unlike the weights of a model. Literally, the weights of the model are a brain, which obviously does not exist in the sperm and the egg. So it has to be grown. Also, the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA. Evolution seems closer to finding the algorithm which then does the lifetime learning. Now, maybe the lifetime learning is not analogous to RL, to your point. Is that compatible with the thing you were saying, or would you disagree with that?

I think so. I would agree with you that there's some miraculous compression going on because obviously, the weights of the neural net are not stored in ATCGs. There's some dramatic compression. There are some learning algorithms encoded that take over and do some of the learning online. I definitely agree with you on that. I would say I'm a lot more practically minded. I don't come at it from the perspective of, let's build animals. I come from it from the perspective of, let's build useful things. I have a hard hat on, and I'm just observing that we're not going to do evolution, because I don't know how to do that. But it does turn out we can build these ghosts, spirit-like entities, by imitating internet documents. This works. It's a way to bring you up to something that has a lot of built-in knowledge and intelligence in some way, similar to maybe what evolution has done. That's why I call pre-training this crappy evolution. It's the practically possible version with our technology and what we have available to us to get to a starting point where we can do things like reinforcement learning and so on.

It's subtle and I think you're right to push back on it, but basically the thing that pre-training is doing, you're getting the next-token predictor over the internet, and you're training that into a neural net. It's doing two things that are unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the internet, it boots up all these little circuits and algorithms inside the neural net to do things like in-context learning and all this stuff. You don't need or want the knowledge. I think that's probably holding back the neural networks overall because it's getting them to rely on the knowledge a little too much sometimes. For example, I feel agents, one thing they're not very good at, is going off the data manifold of what exists on the internet. If they had less knowledge or less memory, maybe they would be better.

What I think we have to do going forward—and this would be part of the research paradigms—is figure out ways to remove some of the knowledge and to keep what I call this cognitive core. It's this intelligent entity that is stripped from knowledge but contains the algorithms and contains the magic of intelligence and problem-solving and the strategies of it and all this stuff.

There's so much interesting stuff there. Let's start with in-context learning. This is an obvious point, but I think it's worth just saying it explicitly and meditating on it. The situation in which these models seem the most intelligent—in which I talk to them and I'm like, "Wow, there's really something on the other end that's responding to me thinking about things—is if it makes a mistake it's like, "Oh wait, that's the wrong way to think about it. I'm backing up." All that is happening in context. That's where I feel like the real intelligence is that you can visibly see. That in-context learning process is developed by gradient descent on pre-training. It spontaneously meta-learns in-context learning, but the in-context learning itself is not gradient descent, in the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution but our learning during our lifetime is happening through some other process.

I don't fully agree with that, but you should continue your thought.

Well, I'm very curious to understand how that analogy breaks down.

I'm hesitant to say that in-context learning is not doing gradient descent. It's not doing explicit gradient descent. In-context learning is pattern completion within a token window. It just turns out that there's a huge amount of patterns on the internet. You're right, the model learns to complete the pattern, and that's inside the weights. The weights of the neural network are trying to discover patterns and complete the pattern. There's some adaptation that happens inside the neural network, which is magical and just falls out from the internet just because there's a lot of patterns.

I will say that there have been some papers that I thought were interesting that look at the mechanisms behind in-context learning. I do think it's possible that in-context learning runs a small gradient descent loop internally in the layers of the neural network. I recall one paper in particular where they were doing linear regression using in-context learning. Your inputs into the neural network are XY pairs, XY, XY, XY that happen to be on the line. Then you do X and you expect Y. The neural network, when you train it in this way, does linear regression.

Normally when you would run linear regression, you have a small gradient descent optimizer that looks at XY, looks at an error, calculates the gradient of the weights and does the update a few times. It just turns out that when they looked at the weights of that in-context learning algorithm, they found some analogies to gradient descent mechanics. In fact, I think the paper was even stronger because they hardcoded the weights of a neural network to do gradient descent through attention and all the internals of the neural network. That's just my only pushback. Who knows how in-context learning works, but I think that it's probably doing a bit of some funky gradient descent internally.

So then it's worth thinking okay, if in-context learning and pre-training are both implementing something like gradient descent, why does it feel like with in-context learning we're getting to this continual learning, real intelligence-like thing? Whereas you don't get the analogous feeling just from pre-training. You could argue that. If it's the same algorithm, what could be different? One way you could think about it is, how much information does the model store per information it receives from training? If you look at pre-training, if you look at Llama 3 for example, I think it's trained on 15 trillion tokens. If you look at the 70B model, that would be the equivalent of 0.07 bits per token that it sees in pre-training, in terms of the information in the weights of the model compared to the tokens it reads. Whereas if you look at the KV cache and how it grows per additional token in in-context learning, it's like 320 kilobytes. So that's a 35 million-fold difference in how much information per token is assimilated by the model. I wonder if that's relevant at all.

I kind of agree. The way I usually put this is that anything that happens during the training of the neural network, the knowledge is only a hazy recollection of what happened in training time. That's because the compression is dramatic. You're taking 15 trillion tokens and you're compressing it to just your final neural network of a few billion parameters. Obviously it's a massive amount of compression going on.

So I refer to it as a hazy recollection of the internet documents. Whereas anything that happens in the context window of the neural network—you're plugging in all the tokens and building up all those KV cache representations—is very directly accessible to the neural net. So I compare the KV cache and the stuff that happens at test time to more like a working memory.

All the stuff that's in the context window is very directly accessible to the neural net. There's always these almost surprising analogies between LLMs and humans. I find them surprising because we're not trying to build a human brain directly. We're just finding that this works and we're doing it. But I do think that anything that's in the weights, it's a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. That's a very powerful analogy to think through things.

Stepping back, what is the part about human intelligence that we have most failed to replicate with these models?

Just a lot of it. So maybe one way to think about it, I don't know if this is the best way, but I almost feel like — again, making these analogies imperfect as they are — we've stumbled by with the transformer neural network, which is extremely powerful, very general. You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns and they're very powerful, and it works really well. That to me almost indicates that this is some piece of cortical tissue.

It's something like that, because the cortex is famously very plastic as well. You can rewire parts of brains. There were the slightly gruesome experiments with rewiring the visual cortex to the auditory cortex, and this animal learned fine, et cetera. So I think that this is cortical tissue. I think when we're doing reasoning and planning inside the neural networks, doing reasoning traces for thinking models, that's kind of like the prefrontal cortex.

Maybe those are like little checkmarks, but I still think there are many brain parts and nuclei that are not explored. For example, there's a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning. But where's the hippocampus? Not obvious what that would be. Some parts are probably not important. Maybe the cerebellum is not important to cognition, its thoughts, so maybe we can skip some of it. But I still think there's, for example, the amygdala, all the emotions and instincts. There's probably a bunch of other nuclei in the brain that are very ancient that I don't think we've really replicated. I don't know that we should be pursuing the building of an analog of a human brain. I'm an engineer mostly at heart.

Maybe another way to answer the question is that you're not going to hire this thing as an intern. It's missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. So it's not fully there yet. You can look at it as not all the brain parts are checked off yet.

This is maybe relevant to the question of thinking about how fast these issues will be solved. Sometimes people will say about continual learning, "Look, you could easily replicate this capability. Just as in-context learning emerged spontaneously as a result of pre-training, continual learning over longer horizons will emerge spontaneously if the model is incentivized to recollect information over longer horizons, or horizons longer than one session." So if there's some outer loop RL which has many sessions within that outer loop, then this continual learning where it fine-tunes itself, or it writes to an external memory or something, will just emerge spontaneously. Do you think things like that are plausible?

I don't know that I fully resonate with that. These models, when you boot them up and they have zero tokens in the window, they're always restarting from scratch where they were. So I don't know in that worldview what it looks like.

Maybe making some analogies to humans—just because I think it's roughly concrete and interesting to think through—I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day. But when I go to sleep, something magical happens where I don't think that context window stays around. There's some process of distillation into the weights of my brain.

This happens during sleep and all this stuff. We don't have an equivalent of that in large language models. That's to me more adjacent to when you talk about continual learning and so on as absent. These models don't really have a distillation phase of taking what happened, analyzing it obsessively, thinking through it, doing some synthetic data generation process and distilling it back into the weights, and maybe having a specific neural net per person. Maybe it's a LoRA. It's not a full-weight neural network.

It's just some small sparse subset of the weights that are changed. But we do want to create ways of creating these individuals that have very long context. It's not only remaining in the context window because the context windows grow very, very long. Maybe we have some very elaborate, sparse attention over it. But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We're missing it. I do also think that humans have some very elaborate, sparse attention scheme, which I think we're starting to see some early hints of. DeepSeek v3.2 just came out and I saw that they have sparse attention as an example, and this is one way to have very, very long context windows. So I feel like we are redoing a lot of the cognitive tricks that evolution came up with through a very different process. But we're going to converge on a similar architecture cognitively.

Yeah. I was about to ask you a very similar question about nanochat. Since you just coded it up recently, every single step in the process of building a chatbot is fresh in your RAM. I'm curious if you had similar thoughts about, "Oh, there was no one thing that was relevant to going from GPT-2 to nanochat." What are some surprising takeaways from the experience?

Of building nanochat? So nanochat is a repository I released. Was it yesterday or the day before? I can't remember. It's trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone. So you have all of the steps, not just any individual step, which is a bunch. I worked on all the individual steps in the past and released small pieces of code that show you how that's done in an algorithmic sense, in simple code. But this handles the entire pipeline.

What is the best way for somebody to learn from it? Is it to just delete all the code and try to reimplement from scratch, try to add modifications to it?

That's a great question. Basically it's about 8,000 lines of code that takes you through the entire pipeline. I would probably put it on the right monitor. If you have two monitors, you put it on the right. You want to build it from scratch, you build it from the start. You're not allowed to copy-paste, you're allowed to reference, you're not allowed to copy-paste. Maybe that's how I would do it.

But I also think the repository by itself is a pretty large beast. When you write this code, you don't go from top to bottom, you go from chunks and you grow the chunks, and that information is absent. You wouldn't know where to start. So it's not just a final repository that's needed, it's the building of the repository, which is a complicated chunk-growing process. So that part is not there yet. I would love to add that probably later this week. It's probably a video or something like that. Roughly speaking, that's what I would try to do. Build the stuff yourself, but don't allow yourself copy-paste. I do think that there's two types of knowledge, almost. There's the high-level surface knowledge, but when you build something from scratch, you're forced to come to terms with what you don't understand and you don't know that you don't understand it. It always leads to a deeper understanding. It's the only way to build. If I can't build it, I don't understand it. That’s a Feynman quote, I believe. I 100% have always believed this very strongly. Don't write blog posts, don't do slides, don't do any of that. Build the code, arrange it, get it to work. It's the only way to go. Otherwise, you're missing knowledge.

You tweeted out that coding models were of very little help to you in assembling this repository. I'm curious why that was.

I guess I built the repository over a period of a bit more than a month. I would say there are three major classes of how people interact with code right now. Some people completely reject all of LLMs and they are just writing by scratch. This is probably not the right thing to do anymore. The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the autocomplete that's available now from these models. So when you start writing out a little piece of it, it will autocomplete for you and you can just tap through. Most of the time it's correct, sometimes it's not, and you edit it. But you're still very much the architect of what you're writing.

Then there's the vibe coding: "Hi, please implement this or that," enter, and then let the model do it. That's the agents. I do feel like the agents work in very specific settings, and I would use them in specific settings. But these are all tools available to you and you have to learn what they're good at, what they're not good at, and when to use them.

So the agents are pretty good, for example, if you're doing boilerplate stuff. Boilerplate code that's just copy-paste stuff, they're very good at that. They're very good at stuff that occurs very often on the Internet because there are lots of examples of it in the training sets of these models. I would say nanochat is not an example of those because it's a fairly unique repository. There's not that much code in the way that I've structured it. It's not boilerplate code. It's intellectually intense code almost, and everything has to be very precisely arranged.

The models have so many cognitive deficits. One example, they kept misunderstanding the code because they have too much memory from all the typical ways of doing things on the Internet that I just wasn't adopting. Maybe one example? You have eight GPUs that are all doing forward, backwards. The way to synchronize gradients between them is to use a Distributed Data Parallel container of PyTorch, which automatically as you're doing the backward, it will start communicating and synchronizing gradients. I didn't use DDP because I didn't want to use it, because it's not necessary. I threw it out and wrote my own synchronization routine that's inside the step of the optimizer.

The models were trying to get me to use the DDP container. They were very concerned. This gets way too technical, but I wasn't using that container because I don't need it and I have a custom implementation of something like it. They just couldn't internalize that you had your own. They couldn't get past that. They kept trying to mess up the style. They're way too over-defensive. They make all these try-catch statements. They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it's okay. I don't need all this extra stuff in there. So I feel like they're bloating the code base, bloating the complexity, they keep misunderstanding, they're using deprecated APIs a bunch of times. It's a total mess. It's just not net useful.

The reason this question is so interesting is because the main story people have about AI exploding and getting to superintelligence pretty rapidly is AI automating AI engineering and AI research. They'll look at the fact that you can have Claude Code and make entire applications, CRUD applications, from scratch and think, "If you had this same capability inside of OpenAI and DeepMind and everything, just imagine a thousand of you or a million of you in parallel, finding little architectural tweaks." It's quite interesting to hear you say that this is the thing they're asymmetrically worse at. It's quite relevant to forecasting whether the AI 2027-type explosion is likely to happen anytime soon.

That's a good way of putting it, and you're getting at why my timelines are a bit longer. You're right. They're not very good at code that has never been written before, maybe it's one way to put it, which is what we're trying to achieve when we're building these models.

Here's another reason this is really interesting. Through the history of programming, there have been many productivity improvements—compilers, linting, better programming languages—which have increased programmer productivity but have not led to an explosion. That sounds very much like the autocomplete tab, and this other category is just automation of the programmer. It's interesting you're seeing more in the category of the historical analogies of better compilers or something.

Maybe this gets to one other thought. I have a hard time differentiating where AI begins and stops because I see AI as fundamentally an extension of computing in a pretty fundamental way. I see a continuum of this recursive self-improvement or speeding up programmers all the way from the beginning: code editors, syntax highlighting, or checking even of the types, like data type checking—all these tools that we've built for each other. Even search engines.

Why aren't search engines part of AI? Ranking is AI. At some point, Google, even early on, was thinking of themselves as an AI company doing Google Search engine, which is totally fair. I see it as a lot more of a continuum than other people do, and it's hard for me to draw the line. I feel like we're now getting a much better autocomplete, and now we're also getting some agents which are these loopy things, but they go off-rails sometimes.

What's going on is that the human is progressively doing a bit less and less of the low-level stuff. We're not writing the assembly code because we have compilers. Compilers will take my high-level language in C and write the assembly code. We're abstracting ourselves very, very slowly. There's this what I call "autonomy slider," where more and more stuff is automated—of the stuff that can be automated at any point in time—and we're doing a bit less and less and raising ourselves in the layer of abstraction over the automation.

Let's talk about RL a bit. You tweeted some very interesting things about this. Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment, and in ways that seem almost irrespective of the final reward at the end of the episode? If somebody is starting a business, and at the end of 10 years, she finds out whether the business succeeded or failed, we say that she's earned a bunch of wisdom and experience. But it's not because the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted. Something much more deliberate and rich is happening. What is the ML analogy, and how does that compare to what we're doing with LLMs right now?

Maybe the way I would put it is that humans don't use reinforcement learning, as I said. I think they do something different. Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse because previously we were just imitating people, so it has all these issues.

In reinforcement learning, say you're solving a math problem, because it's very simple. You're given a math problem and you're trying to find the solution. In reinforcement learning, you will try lots of things in parallel first. You're given a problem, you try hundreds of different attempts. These attempts can be complex. They can be like, "Oh, let me try this, let me try that, this didn't work, that didn't work," etc. Then maybe you get an answer. Now you check the back of the book and you see, "Okay, the correct answer is this." You can see that this one, this one, and that one got the correct answer, but these other 97 of them didn't.

Literally what reinforcement learning does is it goes to the ones that worked really well and every single thing you did along the way, every single token gets upweighted like, "Do more of this." The problem with that is people will say that your estimator has high variance, but it's just noisy. It's noisy. It almost assumes that every single little piece of the solution that you made that arrived at the right answer was the correct thing to do, which is not true. You may have gone down the wrong alleys until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as, "Do more of this." It's terrible. It's noise.

You've done all this work only to find, at the end, you get a single number of like, "Oh, you did correct." Based on that, you weigh that entire trajectory as like, upweight or downweight. The way I like to put it is you're sucking supervision through a straw. You've done all this work that could be a minute of rollout, and you're sucking the bits of supervision of the final reward signal through a straw and you're broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It's just stupid and crazy. A human would never do this.

Number one, a human would never do hundreds of rollouts. Number two, when a person finds a solution, they will have a pretty complicated process of review of, "Okay, I think these parts I did well, these parts I did not do that well. I should probably do this or that." They think through things. There's nothing in current LLMs that does this. There's no equivalent of it. But I do see papers popping out that are trying to do this because it's obvious to everyone in the field.

The first imitation learning, by the way, was extremely surprising and miraculous and amazing, that we can fine-tune by imitation on humans. That was incredible. Because in the beginning, all we had was base models. Base models are autocomplete. It wasn't obvious to me at the time, and I had to learn this. The paper that blew my mind was InstructGPT,

because it pointed out that you can take the pretrained model, which is autocomplete, and if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training. This blew my mind because I didn't understand that stylistically, it can adjust so quickly and become an assistant to a user through just a few loops of fine-tuning on that kind of data. It was very miraculous to me that that worked. So incredible. That was two to three years of work.

Now came RL. And RL allows you to do a bit better than just imitation learning because you can have these reward functions and you can hill-climb on the reward functions. Some problems have just correct answers, you can hill-climb on that without getting expert trajectories to imitate. So that's amazing. The model can also discover solutions that a human might never come up with. This is incredible. Yet, it's still stupid. We need more. I saw a paper from Google yesterday that tried to have this reflect & review idea in mind.

Was it the memory bank paper or something?

I don't know. I've seen a few papers along these lines. So I expect there to be some major update to how we do algorithms for LLMs coming in that realm. I think we need three or four or five more, something like that.

You're so good at coming up with evocative phrases. "Sucking supervision through a straw." It's so good. You're saying the problem with outcome-based reward is that you have this huge trajectory, and then at the end, you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit.

Given the fact that this is obvious, why hasn't process-based supervision as an alternative been a successful way to make models more capable? What has been preventing us from using this alternative paradigm?

Process-based supervision just refers to the fact that we're not going to have a reward function only at the very end. After you've done 10 minutes of work, I'm not going to tell you you did well or not well. I'm going to tell you at every single step of the way how well you're doing. The reason we don't have that is it's tricky how you do that properly. You have partial solutions and you don't know how to assign credit. So when you get the right answer, it's just an equality match to the answer. It’s very simple to implement. If you're doing process supervision, how do you assign in an automatable way, a partial credit assignment? It's not obvious how you do it.

Lots of labs are trying to do it with these LLM judges. You get LLMs to try to do it. You prompt an LLM, "Hey, look at a partial solution of a student. How well do you think they're doing if the answer is this?" and they try to tune the prompt. The reason that this is tricky is quite subtle. It's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable. If you're reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed. So you can't do this for too long. You do maybe 10 steps or 20 steps, and maybe it will work, but you can't do 100 or 1,000.

I understand it's not obvious, but basically the model will find little cracks. It will find all these spurious things in the nooks and crannies of the giant model and find a way to cheat it. One example that's prominently in my mind, this was probably public, if you're using an LLM judge for a reward, you just give it a solution from a student and ask it if the student did well or not. We were training with reinforcement learning against that reward function, and it worked really well. Then, suddenly, the reward became extremely large. It was a massive jump, and it did perfect. You're looking at it like, "Wow, this means the student is perfect in all these problems. It's fully solved math." But when you look at the completions that you're getting from the model, they are complete nonsense. They start out okay, and then they change to "dhdhdhdh."

You're looking at it, and it's like, this is crazy. How is it getting a reward of one or 100%? You look at the LLM judge, and it turns out that "dhdhdhdh" is an adversarial example for the model, and it assigns 100% probability to it. It's just because this is an out-of-sample example to the LLM. It's never seen it during training, and you're in pure generalization land. It's never seen it during training, and in the pure generalization land, you can find these examples that break it. You're basically training the LLM to be a prompt injection model. Not even that. Prompt injection is way too fancy. You're finding adversarial examples, as they're called. These are nonsensical solutions that are obviously wrong, but the model thinks they are amazing.

Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

I guess what I'm saying is, say we have a chapter of a book and I ask an LLM to think about it, it will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same. You can't just keep scaling "reflection" on the same amount of prompt information and then get returns from that. Any individual sample will look okay, but the distribution of it is quite terrible. It's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse.

I think that there's possibly no fundamental solution to this. I also think humans collapse over time. These analogies are surprisingly good. Humans collapse during the course of their lives. This is why children, they haven't overfit yet. They will say stuff that will shock you because you can see where they're coming from, but it's just not the thing people say, because they're not yet collapsed. But we're collapsed. We end up revisiting the same thoughts. We end up saying more and more of the same stuff, and the learning rates go down, and the collapse continues to get worse, and then everything deteriorates.

Have you seen this super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse? The reason dreaming is evolutionary adaptive is to put you in weird situations that are very unlike your day-to-day reality, so as to prevent this kind of overfitting.

It's an interesting idea. I do think that when you're generating things in your head and then you're attending to it, you're training on your own samples, you're training on your synthetic data. If you do it for too long, you go off-rails and you collapse way too much. You always have to seek entropy in your life. Talking to other people is a great source of entropy, and things like that. So maybe the brain has also built some internal mechanisms for increasing the amount of entropy in that process.

You've talked about how you were at Tesla leading self-driving from 2017 to 2022. And you firsthand saw this progress from cool demos to now thousands of cars out there actually autonomously doing drives. Why did that take a decade? What was happening through that time?

One thing I will almost instantly push back on is that this is not even near done, in a bunch of ways that I'm going to get to. Self-driving is very interesting because it's definitely where I get a lot of my intuitions because I spent five years on it. It has this entire history where the first demos of self-driving go all the way to the 1980s. You can see a demo from CMU in 1986. There's a truck that's driving itself on roads. Fast forward. When I was joining Tesla, I had a very early demo of Waymo. It basically gave me a perfect drive in 2014 or something like that, so a perfect Waymo drive a decade ago.

I thought it was very close and then it still took a long time. For some kinds of tasks and jobs and so on, there's a very large demo-to-product gap where the demo is very easy, but the product is very hard. It's especially the case in cases like self-driving where the cost of failure is too high.

Many industries, tasks, and jobs maybe don't have that property, but when you do have that property, that definitely increases the timelines. For example, in software engineering, I do think that property does exist. For a lot of vibe coding, it doesn't. But if you're writing actual production-grade code, that property should exist, because any kind of mistake leads to a security vulnerability or something like that. Millions and hundreds of millions of people's personal Social Security numbers get leaked or something like that. So in software, people should be careful, kind of like in self-driving. In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it's almost unbounded how terrible something could be. I do think that they share that property.

What takes the long amount of time and the way to think about it is that it's a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that's just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don't know what it is, but multiple nines of iteration. There are still more nines to go. That's why these things take so long.

It's definitely formative for me, seeing something that was a demo. I'm very unimpressed by demos. Whenever I see demos of anything, I'm extremely unimpressed by that. If it's a demo that someone cooked up as a showing, it's worse. If you can interact with it, it's a bit better. But even then, you're not done. You need the actual product. It's going to face all these challenges when it comes in contact with reality and all these different pockets of behavior that need patching. We're going to see all this stuff play out. It's a march of nines. Each nine is constant. Demos are encouraging. It’s still a huge amount of work to do.

There's another objection people make to that analogy, which is that with self-driving, what took a big fraction of that time was solving the problem of having basic perception that's robust, building representations, and having a model that has some common sense so it can generalize to when it sees something that's slightly out of distribution. If somebody's waving down the road this way, you don't need to train for it. The thing will have some understanding of how to respond to something like that. These are things we're getting for free with LLMs or VLMs today, so we don't have to solve these very basic representation problems. So now deploying AIs across different domains will sort of be like deploying a self-driving car with current models to a different city, which is hard but not like a 10-year-long task.

I'm not 100% sure if I fully agree with that. I don't know how much we're getting for free. There's still a lot of gaps in understanding what we are getting. We're definitely getting more generalizable intelligence in a single entity, whereas self-driving is a very special-purpose task that requires. In some sense building a special-purpose task is maybe even harder in a certain sense because it doesn't fall out from a more general thing that you're doing at scale, if that makes sense. But the analogy still doesn't fully resonate because the LLMs are still pretty fallible and they have a lot of gaps that still need to be filled in. I don't think that we're getting magical generalization completely out of the box, in a certain sense.

Let's talk about education and Eureka. One thing you could do is start another AI lab and then try to solve those problems. I’m curious what you're up to now, and why not AI research itself?

I guess the way I would put it is I feel some amount of determinism around the things that AI labs are doing. I feel like I could help out there, but I don't know that I would uniquely improve it. My personal big fear is that a lot of this stuff happens on the side of humanity, and that humanity gets disempowered by it. I care not just about all the Dyson spheres that we're going to build and that AI is going to build in a fully autonomous way, I care about what happens to humans. I want humans to be well off in the future.

I feel like that's where I can a lot more uniquely add value than an incremental improvement in the frontier lab. I'm most afraid of something depicted in movies like WALL-E or Idiocracy or something like that, where humanity is on the side of this stuff. I want humans to be much, much better in this future. To me, this is through education that you can achieve this.

So what are you working on there?

The easiest way I can describe it is we're trying to build the Starfleet Academy. I don’t know if you’ve watched Star Trek. I haven’t. Starfleet Academy is this elite institution for frontier technology, building spaceships, and graduating cadets to be the pilots of these spaceships and whatnot. So I just imagine an elite institution for technical knowledge and a kind of school that's very up-to-date and a premier institution.

A category of questions I have for you is explaining how one teaches technical or scientific content well, because you are one of the world masters at it.

With respect to Eureka, one thing that is very fascinating to me about education is that I do think education will pretty fundamentally change with AIs on the side. It has to be rewired and changed to some extent. I still think that we're pretty early. There's going to be a lot of people who are going to try to do the obvious things. Have an LLM and ask it questions. Do all the basic things that you would do via prompting right now. It's helpful, but it still feels to me a bit like slop. I'd like to do it properly, and I think the capability is not there for what I would want. What I'd want is an actual tutor experience.

A prominent example in my mind is I was recently learning Korean, so language learning. I went through a phase where I was learning Korean by myself on the internet. I went through a phase where I was part of a small class in Korea taking Korean with a bunch of other people, which was really funny. We had a teacher and 10 people or so taking Korean. Then I switched to a one-on-one tutor.

I guess what was fascinating to me was, I think I had a really good tutor, but just thinking through what this tutor was doing for me and how incredible that experience was and how high the bar is for what I want to build eventually. Instantly from a very short conversation, she understood where I am as a student, what I know and don't know. She was able to probe exactly the kinds of questions or things to understand my world model. No LLM will do that for you 100% right now, not even close.

But a tutor will do that if they're good. Once she understands, she really served me all the things that I needed at my current sliver of capability. I need to be always appropriately challenged. I can't be faced with something too hard or too trivial, and a tutor is really good at serving you just the right stuff. I felt like I was the only constraint to learning. I was always given the perfect information. I'm the only constraint. I felt good because I'm the only impediment that exists. It's not that I can't find knowledge or that it's not properly explained or etc. It's just my ability to memorize and so on. This is what I want for people.

How do you automate that?

Very good question. At the current capability, you don't. That's why I think it's not actually the right time to build this kind of an AI tutor. I still think it's a useful product, and lots of people will build it, but the bar is so high and the capability is not there.

To the extent you're willing to say, what is the thing you hope will be released this year or next year?

I'm building the first course. I want to have a really, really good course, the obvious state-of-the-art destination you go to to learn, AI in this case. That's just what I'm familiar with, so it's a really good first product to get to be really good at it. So that's what I'm building. Nanochat, which you briefly mentioned, is a capstone project of LLM101N, which is a class that I'm building.

That's a really big piece of it. But now I have to build out a lot of the intermediates, and then I have to hire a small team of TAs and so on and build the entire course. One more thing that I would say is that many times, when people think about education, they think more about what I would say is a softer component of diffusing knowledge.

I have something very hard and technical in mind. In my mind, education is the very difficult technical process of building ramps to knowledge. In my mind, nanochat is a ramp to knowledge because it's very simple. It's the super simplified full-stack thing. If you give this artifact to someone and they look through it, they're learning a ton of stuff.

It's giving you a lot of what I call eurekas per second, which is understanding per second. That's what I want, lots of eurekas per second. So to me, this is a technical problem of how do we build these ramps to knowledge. So I almost think of Eureka as maybe not that different from some of the frontier labs or some of the work that's going on there. I want to figure out how to build these ramps very efficiently so that people are never stuck and everything is always not too hard or not too trivial, and you have just the right material to progress.

Motivation-wise, before AGI motivation is very simple to solve because people want to make money. This is how you make money in the industry today. Post-AGI is a lot more interesting possibly because if everything is automated and there's nothing to do for anyone, why would anyone go to a school? I often say that pre-AGI education is useful. Post-AGI education is fun. In a similar way, people go to the gym today.

We don't need their physical strength to manipulate heavy objects because we have machines that do that. They still go to the gym. Why do they go to the gym? Because it's fun, it's healthy, and you look hot when you have a six-pack. It's attractive for people to do that in a very deep, psychological, evolutionary sense for humanity.

Education will play out in the same way. You'll go to school like you go to the gym. Right now, not that many people learn because learning is hard. You bounce from material. Some people overcome that barrier, but for most people, it's hard.

It's a technical problem to solve. It's a technical problem to do what my tutor did for me when I was learning Korean. It's tractable and buildable, and someone should build it. It's going to make learning anything trivial and desirable, and people will do it for fun because it's trivial. If I had a tutor like that for any arbitrary piece of knowledge, it's going to be so much easier to learn anything, and people will do it. They'll do it for the same reasons they go to the gym.

But it sounded like you had a vision also that this education is relevant to keeping humanity in control of AI. That sounds different. Is it entertaining for some people, but then empowerment for some others? How do you think about that?

I do think eventually it's a bit of a losing game, if that makes sense. It is in the long term. In the long term, which is longer than maybe most people in the industry think about, it's a losing game. I do think people can go so far and we've barely scratched the surface of how much a person can go. That's just because people are bouncing off of material that's too easy or too hard.

People will be able to go much further. Anyone will speak five languages because why not? Because it's so trivial. Anyone will know all the basic curriculum of undergrad, et cetera.

Can I ask some questions about teaching well? If you had to give advice to another educator in another field that you're curious about to make the kinds of YouTube tutorials you've made. Maybe it might be especially interesting to talk about domains where you can't test someone's technical understanding by having them code something up or something. What advice would you give them?

That's a pretty broad topic. There are 10–20 tips and tricks that I semi-consciously do probably. But a lot of this comes from my physics background. I really, really did enjoy my physics background. I have a whole rant on how everyone should learn physics in early school education because early school education is not about accumulating knowledge or memory for tasks later in the industry. It's about booting up a brain. Physics uniquely boots up the brain the best because some of the things that they get you to do in your brain during physics is extremely valuable later.

The idea of building models and abstractions and understanding that there's a first-order approximation that describes most of the system, but then there're second-order, third-order, fourth-order terms that may or may not be present. The idea that you're observing a very noisy system, but there are these fundamental frequencies that you can abstract away. When a physicist walks into the class and they say, "Assume there's a spherical cow," everyone laughs at that, but this is brilliant. It's brilliant thinking that's very generalizable across the industry because a cow can be approximated as a sphere in a bunch of ways.

So because of that training, I always try to find the first-order terms or the second-order terms of everything. When I'm observing a system or a thing, I have a tangle of a web of ideas or knowledge in my mind. I'm trying to find, what is the thing that matters? What is the first-order component? How can I simplify it? How can I have a simplest thing that shows that thing, shows it in action, and then I can tack on the other terms?

Maybe an example from one of my repos that I think illustrates it well is called micrograd. I don't know if you're familiar with this. So micrograd is 100 lines of code that shows backpropagation. You can create neural networks out of simple operations like plus and times, et cetera. Lego blocks of neural networks. You build up a computational graph and you do a forward pass and a backward pass to get the gradients. Now, this is at the heart of all neural network learning.

So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency. Everything else is efficiency. There's a huge amount of work to get efficiency. You need your tensors, you lay them out, you stride them, you make sure your kernels, orchestrating memory movement correctly, et cetera. It's all just efficiency, roughly speaking. But the core intellectual piece of neural network training is micrograd. It's 100 lines. You can easily understand it.

So I love finding these small-order terms and serving them on a platter and discovering them. I feel like education is the most intellectually interesting thing because you have a tangle of understanding and you're trying to lay it out in a way that creates a ramp where everything only depends on the thing before it. I find that this untangling of knowledge is just so intellectually interesting as a cognitive task.

It also makes the learning experience so much more motivated. Your tutorial on the transformer begins with bigrams, literally a lookup table from, "Here's the word right now, or here's the previous word, here's the next word." It's literally just a lookup table. That’s the essence of it, yeah. It’s such a brilliant way, starting with a lookup table and then going to a transformer. Each piece is motivated. Why would you add that? Why would you add the next thing? You could memorize the attention formula, but having an understanding of why every single piece is relevant, what problem it solves. You're presenting the pain before you present a solution, and how clever is that?

You want to take the student through that progression. There are a lot of other small things that make it nice and engaging and interesting. Always prompting the student. There's a lot of small things like that are important and a lot of good educators will do this. How would you solve this? I'm not going to present the solution before you guess. That would be wasteful. That's a little bit of a…I don’t want to swear but it’s a dick move towards you to present you with the solution before I give you a shot to try to come up with it yourself. Because if you try to come up with it yourself, you get a better understanding of what the action space is, what the objective is, and then why only this action fulfills that objective. You have a chance to try it yourself, and you have an appreciation when I give you the solution. It maximizes the amount of knowledge per new fact added.

What is your advice as a student to other students, if you don't have a Karpathy who is doing the exposition of an idea? If you're reading a paper from somebody or reading a book, what strategies do you employ to learn material you're interested in in fields you're not an expert at?

I don't know that I have unique tips and tricks, to be honest. It's a painful process. One thing that has always helped me quite a bit is—I had a small tweet about this—learning things on demand is pretty nice. Learning depth-wise. I do feel you need a bit of alternation of learning depth-wise, on demand—you're trying to achieve a certain project that you're going to get a reward from—and learning breadth-wise, which is just, "Oh, let's do whatever 101, and here's all the things you might need."

Which is a lot of school—does breadth-wise learning, like, "Oh, trust me, you'll need this later," that kind of stuff. Okay, I trust you. I'll learn it because I guess I need it. But I love the kind of learning where you'll get a reward out of doing something, and you're learning on demand. The other thing that I've found extremely helpful. This is an aspect where education is a bit more selfless, but explaining things to people is a beautiful way to learn something more deeply. This happens to me all the time.

It probably happens to other people too because I realize if I don't really understand something, I can't explain it. I'm trying and I'm like, "Oh, I don't understand this." It's so annoying to come to terms with that. You can go back and make sure you understood it. It fills these gaps of your understanding. It forces you to come to terms with them and to reconcile them. I love to re-explain things and people should be doing that more as well. That forces you to manipulate the knowledge and make sure that you know what you're talking about when you're explaining it.

That's an excellent note to close on. Andrej, that was great. Thank you.