So we have a momentum that carries us forward. We have really great people that work together. We have a good operating way of structuring research and can borrow a lot from Silicon Valley how to get things done quickly. And people are generally very excited about work. Everyone feels the weight and potential of what we are doing, what we are trying to do, and because of that, people at OpenAI have a tendency to work pretty hard. And then, having great people excited about what they are doing, all working together reasonably well, results in doing a lot of things. We understand that there's only one time in history where AI is being built and deployed and developed, and people want to do it in the best way that is possible.

Thank you for all of that. Let's switch tags and go back to how all of this works. So is the right way to think about modern AI systems at OpenAI, by modern I mean as of October 2025 versus the old days of, you know, 9 months ago. So is the right way to think about it as a combination of pre-training and RL? First of all, is that the right way to think about it? And second, if so, just at a high level, how does the articulation between both of those work?

Today's language models can basically be thought of as first they are pre-trained, then you do reinforcement learning on it. The reinforcement learning would not work without pre-training. And I think in a similar way, pre-trained models have a lot of limitations that are very hard to resolve without doing something that looks like reinforcement learning. So I think both of those bits are here to be and to stay. I think the way how they are combined and do may and probably will evolve in the future. Nothing should be treated as dogmatic and fixed, and we need to keep generally figuring out the way how to train better models, and this is what we are trying to do.

The interesting thing, and you know, I can credit that to Ilia, how much foresight that he had, but whenever I started at OpenAI in early 2019, I remember there was a research all-hands or something like that where Ilia came on stage and talked about what is OpenAI's research program, what are we trying to pursue. And what he said at the beginning of 2019 was to train a large generative model on all data we can and then do reinforcement learning on it. That was the OpenAI research plan at the beginning of 2019, and this is exactly what we are doing today. The algorithms change, architectures change. I don't think he was even thinking about Transformer at that moment. The GPT was, there was some GPT, but it was like a toy example that someone was playing with. But the goal of training a large generative model on all the data in the world and then doing reinforcement learning with it was already there at the core DNA of OpenAI.

So let's do a little bit of reinforcement learning 101 to make this really interesting to a broad group of people listening to this. So in very simple terms, like explain it to me like I'm 10. What is reinforcement learning?

Yeah. I usually the metaphor and the analogy I have to reinforcement learning is like training a dog. It's very, very close. I used to have a dog when I was a teenager, and even what I remember my parents did, I didn't know anything about raising a dog. What they kind of invited through some friend of a friend, a fireman who I think was working with service dogs, and he came to me and he basically told me a little bit about how do you train your dog. And what most dog owners that are ambitious about training their dogs know, it is always extremely important to have a bag of treats in your pocket. That's what you always do.

And whenever you see your dog behave well, what you should be doing, you should smile and you should give your dog a treat. Whenever you see your dog do something bad, you basically give your attention away, turn away, and become sad. And through the years of breeding, the dogs discover it's a bad reward and bad behavior. And this is exactly doing that, but with models. We elicit a lot of different behaviors in a model, put them in challenging situations, and then we give them a cookie if they do something we want. If they do a good thing, and give them some kind of punishment and negative reward if they do something that we don't want and that we don't like.

In a good way, the good way to do RL is if you balance those things. So if you kind of give cookies half of the time and punish the other half the time. But this is almost like a mathematical kind of aspect of it. But that's the most important part, which is: elicit behaviors, reward the good ones, and then going forward, the model will be more likely to do what you want and less likely to do what you don't want. And through that, it improves. It is the way how to train models to elicit actual behavior that is not first, it's next to prediction. If you pre-train the model, you literally train the model to predict the next token. RL is like a completely different gradient, a completely different set of what we want to get out of the model.

And getting the model to do what you want, just for some vocabulary and semantics, that's you hear sometimes the term policy. So in RL you hear terms like agent, environment, action, reward and policy. So I think a lot of those are sort of self-explaining but policy is what? That's a strategy, that's a behavior of the model.

Yeah. Policy is the behavior of the model, as the model weights represent what it does when put in a different thing. Like, a model in the end is a mathematical object, and you can define it. And policy is a mathematical function that maps observations to actions. You see what you see, and then what you do with what you see.

Can you give us a little bit of a bird's eye view of the evolution of RL over the years? What, you know, mostly how does modern RL differ from sort of historical RL?

Yeah. Again, there was super historical RL, so not even that old, but the main tectonic shift was when combining neural networks with reinforcement learning. The reinforcement learning predates neural networks as a general mathematical method of optimizing behaviors in mathematically defined environments and as a method of study.

That's what is known as deep reinforcement learning. Is that right?

Yes. Then the deep reinforcement learning, that basically DeepMind's invention of combining neural networks with reinforcement learning to the DQN moment I talked to you about. And then from there, there was a moment where there was a pretty active area of research of reinforcement learning on games. Like when even when I started in 2019, reinforcement learning was kind of fashionable at that moment, although not very successful, but we were, the reinforcement learning was able to solve a lot of games.

But the bottleneck was there that the models were not pre-trained in any way. We were training a lot of behaviors playing games. We even got the AlphaGo moment out of that, which a lot of people got very excited about it, but it was still like learning behaviors without the models that were meaningfully smart about those behaviors. There was still a lot of kind of, you don't want to call it caveman intelligence, but something in that regard about models not being really smart even though being pretty heavily reinforced. And there was a long research in that, and a lot of cool results and theoretical understanding of RL comes from those days because people were researching RL actively. But in some way, it was a dead end of doing RL without pre-training.

And in my moment when I finished working on robotics, I started working on teaching language models to code, but having pre-trained models was a really big deal. And the GPT era of scaling and of large-scale ingesting lots of data to really train great models enabled us already at that moment to start RL. And that was one of the first things I did almost immediately whenever GPT-3 was trained, I tried to do RL on it. And there were always bottlenecks. The systems were kind of clunky. It was hard to figure out what are the right algorithms, what are the right problems to work on it, and what is the right algorithm to train it on.

And what OpenAI did at that moment, and what's kind of how research goes, we kind of cargo culted a lot of things that were used for games and almost the same things for robotics. And at the first RL I was doing on large models was kind of the same PPO we used for everything. And it gave some results, but those early results weren't completely mind-blowing in RL. And there was a long time where we were keeping on investing in it, and personally I always believed there will be a really, really big moment for RL and language models, but the early trials and errors weren't super successful.

At the moment when we trained GPT-4, and there was an interesting moment where we trained GPT-4, and everyone today thinks, "Oh, GPT-4 is such a great model," but when we trained GPT-4, we were pretty underwhelmed internally. And then there was a lot of moments, "Oh, we trained this model, we spend a lot of money on it, and it's kind of pretty dumb." At least, we have GPT-3, GPT-3 already does all that stuff, and GPT-4 doesn't really seem to be that much better.

And we had this kind of question, like, it kind of seemed smart on evals that were one token long. It seemed to be able to give pretty detailed answers to complex questions where it was one token. But if you actually let it speak for longer, it wasn't very coherent or really gave very long answers. And we need to ask this question: How do we actually make the language model that seems to have some smartness in its weights actually sound smart and actually be good in talking to it?

And that was the moment where a technique that was developed already a few years earlier really shone, which was called RLHF, which is basically doing PPO on large language models with the reward given from human preferences of seeing two parts of the text.

Thumbs up and thumb down.

Yeah, thumbs up, thumbs down, whatever human preference is. That's a very good reward because there are a lot of things how the model can generate bad text, and how early GPT-4 was generating bad text in a lot of ways. And RLHF was able to catch those things and correct it, and you know, reinforce good behaviors, reinforce generating good text, and then punishing bad text. And in the end, GPT-4 plus RLHF together as a package delivered the GPT moment to the world that everyone sees. And as much as it is a big success of pre-training, it actually also was a pretty big success of RL in the form of RLHF.

Okay. Great. All right. So going back to RL, you tweeted the other day, "The GRPO release has been in a large way, has accelerated the research learning program of most US research labs." So what is GRPO?

Uh, yeah, it was a little bit of a tongue-in-cheek moment. And I am extrapolating here a little bit what exactly happens because I haven't been in most US research labs, but I have some mental model of what happened and how. And long story short, GRPO was the open source release from DeepSeek. And there was a, like everyone who is terminally online and follows AI discourse knows that DeepSeek moment of whatever it was when the Chinese company that seems to be doing really, really great work released a new model, and it was also a pre-trained model, a reasoning model. They open-sourced the algorithm. They open-sourced a lot of things they did. Overall, a really, really great and technically excellent release.

And there was a lot of discourse about that they pre-trained their model particularly cheaply, and that was part of the discussion about the DeepSeek moment. But the other part of the discussion was that they released their reasoning process. It was like not very far after our O1 release. And as far as I know, our O1 release mostly caught a lot of US labs by surprise. They didn't have a similarly advanced RL research program, to my knowledge. Basically no one.

And I think the only company in the world that I am aware of, and there's probably a lot of things I didn't know, but you talk to people, sometimes you hear rumors. So this is my version of the world, is that if you look at the other papers of DeepSeek, that company was doing pretty similar, in some ways, RL research to what we are doing. And I have to clarify, what OpenAI is doing is not exactly GPO, it is slightly different in many different ways, but some parts are definitely similar. And what's most important, those are both like large-scale policy gradient algorithms.

And DeepSeek was doing, the company was doing research in a slightly adjacent area. They were not very far. And whenever we released O1 and we told the world that you can get pretty great results with scaling up reinforcement learning on language models, I think it was not a very big hop for the DeepSeek company to kind of realize, "Okay, we are not very far from getting similarly good results." And they did it. They trained their reasoning model and they released it, and they told the world how, pretty much not very much later than we released O1. And I think for a lot of US research labs that didn't yet know, didn't have a research program how to train reasoning models, they looked, "Oh, there's this Chinese company, they released how to do it." It helped us kickstart and train reasoning models much faster than we would have to otherwise if we would have to find all those bits ourselves.

What does it take to scale RL? So if there was a phase where OpenAI was very focused on pre-training, and then if I understand correctly, the last 12-18 months or whatever time period where the emphasis has been on the sort of second part of the plan, which is scaling RL. Is that a question of just giving RL more compute, more data, more labeling as we were saying? What does it take?

The first thing that is important to know and understand is RL is hard. Conceptually, if you think about it, and there's still a lot of depth to it, but conceptually, mathematically speaking, pre-training is dead simple. It is the kind of the simplest thing you can do. And there has been a lot of thought and a lot of optimization already put through a few years of optimizing and doing very well at very large scale a very simple mathematical operation.

In terms of RL, it is much, much more complex. There are many more things going on in a reinforcement learning run. There's many more things that can go wrong in doing it, especially as you scale up, and many more types of bottlenecks, failures. It's a much more delicate thing and there's much more room for error. In some way, I don't want to go too deeply in the parallel because it's a little bit overblown, but in some way, just to give some coincidence, you can have a steel factory which makes steel, and the process is relatively standardized, and you make blocks of steel and they are uniform and nice and well-defined what it is, versus like building semiconductors, which there are very, very few companies in the world that can do it because there are so many things that can go wrong and you have to put a lot of attention to details to make great semiconductors, and it's very, very complex internally.

And in many ways, this is kind of like, I don't want to diminish because there's a lot of very hard technical difficulty to do pre-training well at a large scale. But there's just many more moving pieces and many more elements of the reinforcement learning stack that need to get right to get a large-scale run successful.

You mentioned working on agents, like agentic AI. Where does that all fit? The tool use, the whole agentic autonomy versus reasoning, RL, like help us to reconcile what does what and what impacts what.

What's important is I believe, and I think that there can be a lot of positive impact of AI on our world and our lives through automation, through problem solving, and through AI doing good things for us, the things that we want. And for a long time, and for a long time again, it's not that long, but the last two years or so, or maybe approaching three, we've been living in this world where we kind of ask questions to AI and it gave us an answer, at the beginning instantly. Now it can think for like a minute or two, which feels long.

But in many ways, what can you do for two minutes if you think of how many problems you must solve? And AI is probably a little bit faster in the things it can solve, but it's still a limit of what it can do. There are still a lot of tasks that would take AI to do much, much longer. When I prompt Codex, it works for a while, again, a few minutes. There are a lot of things we have internally and we are doing that allow the model to work for much longer. We still didn't figure out the right product to deploy them. But the models can think for like 30 minutes, an hour, two hours these days on certain types of tasks and problems, even longer than that.

And they generally are capable of doing so. And we need to figure out how to make that process more useful and more being able to actually come to various problems in real life, whatever it is, coding or booking travel or making plans or even designing houses or new electronic devices or whatever else we would like models to do, we would like them eventually to be able to do for us. A lot of this comes through the models thinking independently for longer periods of time and considering more of our alternatives, bits, and just sometimes going through a slog of very long lists of tasks.

All right. So maybe to zoom out to close this conversation. You said the other day, you tweeted, "We all collectively believe AGI should have been built yesterday. And the fact that it hasn't yet is mostly because of a simple mistake that needs to be fixed," which is super awesome as a tweet. Do you think that the combination of pre-training and scaled RL takes us to AGI?

There's always an interesting question of what do we consider something that is not pre-training? And where is the limit? I generally think something that we are doing like pre-training today is necessary. I think something that we are doing like RL today is necessary. And there will surely be a few things more. And we have a lot of very ambitious research programs on some of those things.

And I don't want to go into debates whether it's the same or not, but we are and want to be constantly changing the way how we train the models to represent what we think the right form of intelligence is and the most useful form of learning is. And we are constantly researching various things. And the distance from AGI is also a very complex question. I really like someone said it to me, but I think it is right that if you talk to someone from 10 years ago and show them ChatGPT from today, they would probably call it AGI. But we are not today because it still has a lot of limitations, and we are all very aware of those limitations, and we are pretty sure we can resolve those limitations.

There will be probably some further limitations of the future models that will need to be fixed. There is an ultimate question which is very hard to answer: when is the moment that the model can improve itself without that much external output and without humans working on it and fixing it? And I think it is a very hard question. It is a serious question that humanity needs to try to answer because most of that will still largely depend on our infrastructure and our systems, but we will be able to start fixing itself without us having to fix it. And that's when the predictions of what really AI will be able to do and will be able to solve at that moment start becoming a little bit murkier than what we can do right now, which I think we still can do that pretty well.

But philosophically, you may have heard Richard Sutton, you know, the other day on the Dwarkesh podcast, which is a wonderful episode that people should really listen to, effectively saying that the only path to AGI was going to be pure RL and that fundamentally LLMs, and maybe I hope I'm characterizing what he said appropriately, but that LLMs were a flawed premise because effectively that was imitation of reality, whereas RL was enforcement of reality. Do you have any thoughts sort of like philosophically on that question?

Yeah. I haven't had a chance to fully listen to that episode yet, so I also don't get all the details of that thought. But what I can say is that we are doing quite serious RL on language models these days. And in terms of a pure RL, I don't think really pure RL makes sense. RL needs pre-training to be successful, and I think pre-training, as I said before, needs RL to be successful as well. I don't think without RL it would make sense, the research program we are doing.

But we are, OpenAI is, and I'm pretty sure all other AI labs as well, very serious about doing a lot of reinforcement learning on our models. And what a lot of people are saying, whether LLMs are an on-ramp or off-ramp of AGI, very often they do mean pre-training. But it's also clear that the current way how we are doing things is also not yet enough and it's not yet everything, and there will need to be further changes to the setup.

But sometimes people say, "Oh, if you are doing RL, it's not an LLM, it's something else." Sometimes they say, "Oh, if you can write a program in your roll-out and it's a chain of thought, it's not in this neural network only, it's a neural-symbolic system." So it's easy to get, some people consider something an LLM and the other thing not. But personally, my view is what we have is a pretty good foundation for the next step. We did Transformers first trained for translation, then we were pre-training them on large-scale data, then we were doing RLHF on them, now we are doing large-scale reinforcement learning. We'll do a few more and more complex things.

There is a chance somewhere along the line the architecture will start changing more or less significantly. And I personally think we are on the right path, and it will feel less like completely turning around and more like keep on adding more things and maybe dissolving some old elements that carried us to that particular level of intelligence and were not needed anymore.

Well, that feels like a wonderful place to leave it. You've been very generous with your time and thoughts and giving us a glimpse into OpenAI, what you work on, what it looks like behind the scenes, and the key aspects of pre-training and scaling reinforcement learning. So, it's been a wonderful conversation. Jerry, thank you so much. Really appreciate it.

Thank you very much. I enjoyed being here a lot too.