John Schulman: Before ChatGPT, OpenAI had these instruction-following models. And, uh, that was, B, the idea. There was, um, we had base models and people can, um, prompt them in elaborate ways, um, but, uh, they're also kind of hard to prompt. You had to, uh, they basically do autocomplete, so you have to set up a very good prompt with some examples. So, uh, people at OpenAI, uh, were working on, um, just taking the base models and making them easier to prompt, so that if you just wrote a question, it would answer the question instead of giving you more questions or something. Uh, so we had these instruction-following models which were kind of like base models but a little easier to use. Um, and those were the original ones deployed in the API, uh, after, um, GPT-3. Those were the next, uh, generation of models.

John Schulman: Um, then at the same time, there were definitely a lot of people thinking about, um, chat. So, uh, so Google had some papers, uh, like they had, uh, Lambda and, um, earlier Mina. So they had these chat bots, and it was more like, um, uh, like you had a, it was more like a base model that was really specialized to, um, the task of chat, really good at chat. And, uh, like, I think, at least looking at the examples from the paper, it was more, uh, used for sort of fun applications, like, um, where the model would, would, uh, like take on some persona and pretend to be that persona. It was not so functional, like, um, like, help me refactor my code. Um, so, yeah, there are definitely people thinking about chat. I had worked on a project before, uh, looking at chat called, uh, Web GPT, which was more about doing question answering with the help of web browsing and retrieval. And, well, when you do question answering, uh, it really wants to be in a chat because, um, you always want to ask follow-up questions, or sometimes you need a clar, the, the model should ask a clarifying question because the question is ambiguous. So it was kind of clear after we did the first version of that that we should, the next version should be conversational.

John Schulman: So anyway, we started working on, uh, like a conversational chat assistant. Um, and, uh, we, uh, this was built on top of GPT-3.5, which was done training at the beginning of '22. And, uh, that model was quite good at language and code. So we quickly realized that it was actually, uh, quite good at coding help, and that was one of the things we were excited about. So, yeah, we worked on that. Uh, we worked on that for, for most of the year. And we had, we had browsing, um, as another feature, and it, though we ended up, uh, like deemphasizing that later on because the, like, the model's internal knowledge was so good that we didn't, that the browsing, um, wasn't the most interesting thing about it.

John Schulman: Um, and then, uh, we were thinking about, we had it out for beta testing or to friends and family for a while, and we were thinking about doing a public release. Um, but, um, at that time, uh, actually GPT-4 finished training in August, or, um, yeah, in August that year. And, um, actually, the, um, like the flagship RL effort at OpenAI was the instruction-following effort because that was the models that were being deployed into production. So, um, like the first fine-tunes of GPT-4 used that, um, that whole stack, and that was, um, yeah, those models were really good, and everyone got really excited about that after seeing the, uh, like, instru, fine-tune GPT-4s. Uh, but so they were really, really good. They would occasionally give you amazing outputs, but they were also, like, a little bit, the model was clearly, like, pretty unreliable. Like, it would sometimes hallucinate it a lot, and it was like pretty, it would sometimes give you pretty unhinged outputs. So it was clearly not quite ready for prime time, but it was, like, obviously very good.

John Schulman: Um, and, uh, yeah, so I guess that, um, people forgot about chat for a little while after that. C, about this, like, alternative branch. Uh, but then we, we ended up, um, we pushed it further, and we ended up, like, mixing together all the datasets, like the instruct and the chat data, and to try to get something that was the best of both worlds. And, uh, I think the, yeah, the models we, the chat models were like, uh, were clearly more, um, like, it was an easy, easier to use. It was sort of more, um, it sort of, uh, like, automatically had much more sensible behavior in terms of, like, the model knowing its own limitations. The other thing about chat was that when we had these instruct models, uh, like the task of, uh, complete this text, but in a nice way, or in a helpful way, that's like a pretty poorly defined task. So I think, uh, like, I think that task is like both confusing for the model and for the human who's supposed to do the data labeling. Whereas for chat, um, I think people had an intuitive sense of, uh, like, what a helpful robot should be like. So I think it was, uh, just much easier to tell people, uh, like, uh, to, to get, for people to get the idea of what, what the model was supposed to do. Yeah.

John Schulman: Um, and, uh, so that, so as a result, I think the, um, like the model had a much more coherent personality and, uh, like it was much, like, easier to get, um, like, robust, like, sensible behavior. Robustly. Is it the case that anybody could have made ChatGPT using your publicly available fine-tuning API? Not exactly. I mean, uh, they could have, um, I don't remember the status of which models were available for fine-tuning. Uh, you, assuming we had 3.5 available for fine-tuning at the time, you could have made something pretty decently close. But I'm not sure you would have, um, I don't think you would have been able to do just one iteration of fine-tuning where you have like, purely human-written data, and you fine-tune on that. I think you would want, like, you'd want to do several iterations. Like if you're not going to do RL, um, which, which we did, um, you'd want to do some kind of iterative supervised fine-tuning where you have like, humans edit the model-generated outputs. Because it's really hard to get people to, like, if you train on human-generated data, even if it's really high quality, it's just hard for a model to fit that data perfectly because it might not be, like, it might not be something a model is capable of outputting. Uh, so you need to do something iterative that looks a little bit more like RL.

John Schulman: Uh, so I think if you had done that, you could have gotten something pretty close. But, um, that would have been kind of non-trivial. Um, but we also had another, uh, like instruction-following model trained with RL that was released a little before ChatGPT. So I think if you put a chat, like wrapper on that, you would get something decently close. Uh, but it, like, that model, um, like if you just prompted it with chat, um, so, but that model had some, uh, differences in, uh, strengths. Like, it was like that model was pretty good at writing and poetry and so forth, but it wasn't, uh, it sort of, it wasn't as good at knowing its limitations and, uh, at factuality and so forth.