Speaker: The fraction of compute spent on training, specifically pre-training versus post-training, is changing significantly in favor of post-training in the future.

Speaker: Yeah, there are some arguments for that. I mean, right now it's a pretty lopsided ratio, but you could argue that the output generated by the model is high quality, or higher quality than most of what's on the web. So, it sort of makes more sense for the model to think by itself instead of just training to imitate what's on the web. So, I think there's a first-principle argument for that.

Speaker: And, um, I would say we've found a lot of gains through post-training. So, um, I'm not sure, but I would expect us to keep pushing this methodology and probably increasing the amount of compute we put into it.

Speaker: The current GPT-4 has an ELO score that is like 100 points higher than the original one that was released. Is that all because of what you're talking about, with these improvements brought on by post-training?

Speaker: Yeah, I would say that most of that is post-training.

Speaker: Interesting. Um, so there are a lot of different axes for improvement. Like, you can, uh, yeah, so we think about data quality, data quantity, just doing more iterations of the whole process of deploying and collecting new data, and like changing the kind of annotations you're collecting. So, there's a lot of things that stack up, but together they give you a pretty good effective compute increase.

Speaker: How much of a moat is better post-training? Currently, companies distinguish themselves by, well, how big is our model and so forth. Will it be a big moat for whoever has figured out all the finickiness that you were talking about earlier with regards to all this data? I think there's something of a moat because it's just a very complex operation. And there's, uh, so it takes, uh, you have to have a lot of skilled people doing it. And, uh, so there's a lot of tacit knowledge and, uh, um, there's, uh, a lot of organizational knowledge that's required. So, um, so I think, um, yeah, I think post-training, uh, like to create a model that actually has all the functionality people care about, uh, is pretty complicated. It requires a pretty complicated effort. Um, so, and this requires a lot of, this is basically an accumulation of a lot of R&D.

Speaker: Um, so I would say, um, I would say that makes it somewhat of a moat, that it's not trivial to spin this up immediately.

Speaker: Uh, it does seem like the same companies that are putting together the most serious pre-training efforts are also putting together the serious post-training efforts. So, uh, it seems like it is somewhat possible to copy or to spin up more of these efforts. Um, there's also like one force that sort of makes it less of a moat: is that you can, uh, like distill the models, or you can take someone else's model and, uh, clone the outputs, or you can, uh, use someone else's model as a judge to, like, do comparisons. So, I think, uh, like the more big league people probably aren't doing that because it goes against, uh, terms of service policies. But and it would also be a sort of hurt to their pride, but I would expect some of the smaller players are doing that to get off the ground.

Speaker: What makes for somebody who's really good at doing this sort of R&D research? Uh, I hear it's super finicky, but like, what are the sort of intuitions that you have that enable you to find these ways to mess with the data and set up these environments?

Speaker: I'd say I just, um, have a decent amount of experience at this point from, uh, like the different parts of the stack, from like, uh, RL algorithms, obviously, since I've worked on those since, uh, grad school, uh, to like, uh, the data collection, um, like, the annotation process, uh, to, um, like, playing with language models. So, I mean, I'd say I've just dabbled with these things. And, uh, I'd say the people who, um, do well at this kind of research, uh, have some view of the whole stack and have a lot of curiosity about the different parts of it. And, uh, also sort of think about, um, well, you want to be both empirical, um, and, uh, like use experiments, let experiments update your views. But you also want to think from first principles somewhat, like, uh, what, um, like, assuming that, um, like, learning works, uh, like, what would be the ideal type of data to collect, yeah, and that sort of thing.