Yann Dubois: You need to reach this level of reliability to really make any of these AI tools very useful and I think we just crossed that probably December last year at least at OpenAI now we can trust these models to do a lot of the work that we are doing the last few months have been pretty well we moved from like competitions to uh usefulness to users and that's what we are feeling right now I think most of the time the BIC is the the last mile there will always be a lot of space left for this last mile in different verticals and I would highly encourage people to continue working on that.

Matt Turk: Hi, I'm Matt Turk. Welcome to the Mad Podcast. My guest today is Yan Dubois who co-leads the post-training frontiers team at OpenAI. The recent release of GBT 5.5 was yet another major milestone in AI and Jan's team helped build it alongside OpenAI's prior top reasoning models including 03 and GBD5 thinking. Before OpenAI, Yan was at Stanford where he co-authored Stanford Alpaca, the landmark project that kicked off much of the modern post-raining research community. In this conversation, we go deep on what's actually new in GBD 5.5, why reinforcement learning is moving from math and coding competitions into messy real world work, why AI progress can feel like a sudden step function, and why continual learning remains one of the big unsolved problems in AI 3 years after Jet GPT. Please enjoy this fantastic conversation with Yan Dubois.

Matt Turk: Hey Yan, welcome.

Yann Dubois: Hi Matt, thanks for having me.

Matt Turk: It's been another wild uh last few weeks in the world of Frontier AI with the release of GPD 5.5 of Claude Mythos preview. So it feels like uh we have unlocked yet another step function in progress particularly in cyber security agent coding. What's the best way to think about this from your perspective? Are things accelerating? What is happening?

Yann Dubois: Yeah. Uh the last few months have been pretty wild. Um internally we also really feel it. Uh and I think anyone who's working with uh anyone who's work who's coding basically is really feeling it right now. Um I think that's really because of three reasons. Uh the first one is even though the in my mind everything the progress is actually pretty continuous you need to reach this level of reliability to really uh make any of these AI tools very useful and I think we just crossed that probably December last year at least at openi that's why I thought we really crossed that threshold uh where now we can trust these models to do a lot of the work that we are doing so it feels like a stem function even though I actually in terms of capability it's like it's pretty continuous.

Yann Dubois: Um so that's the first thing. The second the second reason is um once you start having models that are really good you accelerate yourself. Um especially in terms of coding given that we all code internally uh you you accelerate yourself both for having these models like train the other models but also like build like the tooling that we need as researchers to like do our job and and all this acceleration I think means that we saw these last few months going faster and faster. The third thing that I I think we are uh feeling is all of last year uh we really built on um like these reasoning models and we really like sawing pushing a lot on on reinforcement learning and initially when we had like 01 um 01 preview even 03 um these models were still like optimized for uh what we call verifiable rewards things where we actually have access to ground truth and like it's easy to test whether you're correct or not uh That is for example the case in like math questions or like comp like like coding competitions. And what I think we are realizing now is that we were able to take many of the tools that we built for these like verifiable reward cases and we were able to use them more generally in uh on for reinforce learning on like real use cases. And I think that's like really why we're feeling that right now in like um uh just real world coding rather than like competition. So we move from like competitions to uh usefulness to users and that's what we are feeling right now.

Matt Turk: Okay. Fascinating. So we're going to unpack a lot of this particularly on the on the RL side. For the first thing that you mentioned reliability is that a engineering is that models like what makes a model reliable in in in the way you meant it.

Yann Dubois: It's a little bit of everything. But in general, given that these are agentic models, h the longer, if you just think about it as like every two minutes, there's like a certain probability that they're wrong. Uh the longer that they run, the the higher the probability that like the final answer is going to be wrong. Um so it's just something inherent in like agentic models. And what we've been pushing a lot on is like making sure that the model like we decrease this probability of being wrong every like two minutes. So purely from a model point of view of course there's a lot of reliability that is also happening on the applied side and the team at open air has been doing an an amazing job um on that um but I'm I'm even talking only about reliability of our models and like making sure that like basically we decrease the probability of being wrong.

Matt Turk: Great. So 5.5 formerly known as spud was uh as mentioned a big deal uh is a big deal and I'm just a curious from the inside what was what are you guys the most proud of? What did you find the most challenging? Give us some some some color on like how you all uh felt uh you know releasing this.

Yann Dubois: We're all really excited by 5.5 to be honest. It is one of these models where everyone in the company was extremely involved uh in building um and I think that we really feel it now that's like we got a lot of attention because of 5.5 and it's uh it seemed like all the stars were aligned. That doesn't always happen. Um and I was just like a great model for the for this. Um so we we did feel it. It's kind of funny because in general with every model that is looking really good early on, we have a model, we all get really excited about it and then there's like tons of doubt that start uh coming up because it's like oh like everyone is so hyp is like hyping this thing internally but actually it's like bad at all these other things and then there's another wave where like people start uh um underhyping it and it kind of goes through through waves and it depends like when we actually ship it how like people feel about it internally. But that's true with like most models that we have. Um, so 5.5 was not that different in this case, but it definitely maybe had like a a higher amplitude of the wave. So people were very excited then very not as excited and and we shipped it and and people are happy externally.

Matt Turk: How long does that process take? Like you know you including the waves of going up and down and of of excitement. I guess it depends on the on the on the release and the importance of each release, but like is that a is that a few weeks? Is that a few months?

Yann Dubois: It really depends. I so I cannot talk exactly about what what went into uh 5.5 but it kind it kind of depends um which part of the pipeline is training parts of the model. So we really have like different sub teams uh including pre-training and you have like the mid-training stage and like you have some post- trainining and usually the closer you get to to products like postraining being the last one the faster the iteration cycle is. Um and if you're more upstream, the slower the iteration cycle is. Uh so it could go from let's say from months to to days.

Matt Turk: Uh basically 5.5 was particularly good um on a gentic coding, computer use, knowledge work and early scientific research. How does that work internally? Do do different people focus on those different parts? How do you get to that result?

Yann Dubois: Yeah, we definitely have different teams that are working on specific use cases and are pushing on these use cases. Uh my team specifically is actually the one that is kind of taking all these vertical improvements and try to put them together in the final model. You could see it as a team that is doing both kind of the smoothing function. So you have all these improvements, but you need to make sure that the model is doesn't feel too spiky, doesn't feel the differently in different on on different verticals. And also you you need to have some teams that are working and that's basically what my team is doing on all the horizontal improvements. So there are many things like instruction following function calling or like thinking about how much should a model think for on different uh problems those are very horizontal and that kind of impacts all these use cases. So we have both these more vertical teams and these more horizontal ones. Um and both are very important uh to to to improve the to improve on the model.

Yann Dubois: Um, and the good thing is that these things can kind of be improved orthogonally. So you might have like multiple different teams that are working on certain verticals and maybe for one model there's only half of these teams that made integrations basically in the last run and like improved the model on on these capabilities and maybe for the next model it'll be the other half. So that's kind of at a high level how it works. uh one thing which I will say because you asked also about uh what are the things that we are really proud about for this model I would say two things number one is the efficiency of the model u we really really improved uh the efficiency of the model and like we most of the tasks and we basically performed I would say like 2x faster now with this model um so that's great uh and the other one that I already mentioned before but it's kind of this alignment of the company and making sure that like everyone is working towards the same goal. Uh and that really takes the entire the entire company working towards like this northstar of building one good model uh uh in in like specific timelines. So very very proud of how that happened.

Matt Turk: Great. And then speaking of efficiency, how do you optimize for that? We're talking about efficiency per per token. Are we also talking about latency in serving the model? What what what part is AI research versus engineering?

Yann Dubois: So that's what that's what I I mean when I say it's the entire company is that it really comes from everywhere. It has to come from like inference optimizations. Um it has to come from the model being more efficient in its thinking time. So you have basically every token that you think for uh basically the the usual plot that you should be looking at is x-axis the number of tokens that you think for and y-axis um the the performance. So this is the these test time scaling curves that we look at. Um and research basically tries to move this curve to the left. So think less uh to be the same level or more correct. Um and then inference also deals with with this x-axis but switches switches it from number of tokens to actual latency. Um and the final thing that people care about is latency on x-axis performance on y-axis. And this is where everything comes together. And this is really what happened with 5.5. Um, so yeah, that's why I always say I'm really proud of the company for this one.

Matt Turk: Okay, great. Let's talk about you for for for a minute. So you are in the post trading frontiers team. What? So that that team you described as horizontal. So what does the the team do in general?

Yann Dubois: Yeah, I would say there's three things that we do. So in a broad broad sense, we are on the porching org. Uh, and my team is the porching uh frontiers one. Um so there are three things that my team does. Number one is we kind of decide what goes into the final run. Um so as we talked before there's like money verticals and someone needs to decide like what can go in what cannot and also provide the the science experiments for people to to iterate on something that's going to be representative of the final run. So this is the first thing that that my team does.

Yann Dubois: The second thing that my team does is bringing everything together and actually doing the big run. So this has as you might imagine like we train on a good amount of GPUs. So there's a lot of infra work that is needed but also there's a lot of ML work that is needed by putting everything together and making sure things work well together. And then the third thing that my team does is uh horizontal improvements to the models. Basically there are some things that like these vertical schemes will not usually look too much at. For example the thinking time as I said before. So how much should the model think for on certain answers? um or like instruction following, function calling, uh things like memory and like general improvements to the model that are really across the stack. Um so that's what the pushing frontiers team does and uh and I'm leading that team.

Matt Turk: Okay, great. And uh what was your journey to open AAI?

Yann Dubois: Oh, it's a long story, but I'll try to keep it really short. Uh basically, I did my undergrad in biomedical engineering um in Switzerland. Um, I'm from Switzerland and then I went on an exchange in Canada and I learned about word tove vec. So I don't know if you heard about this algorithm but it basically takes words which is like something discrete uh and puts it in a in a vector space. So puts it basically in a way to think about it as a plane where if words that are more similar to one another will be closer to one another. So it it brings these like discrete words into like some continuous space that is semantically meaningful and I was absolutely blown away by that algorithm and that's when I decided that I I wanted to work on natural language processing and just like understanding language.

Yann Dubois: Um at that time I was very wrong but I thought that uh English uh NLP was basically solved or like close to being solved that was in 2017. So that was uh uh right when transformers started. I was actually right before transformers. So I was very wrong but I decided I wanted to work on under research languages and basically um I wanted to improve um NLP on languages where we don't have that much data. Uh so I went to uh work uh for Grab in Singapore and I was basically building the natural language processing uh pipeline for them. Uh working with Kum with Bahasa with Thai Vietnamese and all these different languages. And then I'm skipping a little bit. I had I did more academic type of work in different countries and I ended up at Stanford did my PhD there. Um and after this um had a small stint into startups and then went to open air.

Matt Turk: Yes. And I I remember seeing on uh I think your blog or your page uh a note to for for quant firms to not reach out to you because you were not interested in hedge fun work.

Yann Dubois: Yeah. by I always think it's very important for me to think about the positive impact that I'm having in the world or at least that I'm trying to have. Um so so that's that's why this note is there.

Matt Turk: Yes. And as we were saying just before we started uh recording uh people uh may have seen you in the GBT 5 video announcement and you did this uh very funny uh demonstration of um an app that was built on the fly to teach your partner how to speak French. So like people should go check that out.

Yann Dubois: Exactly. Um that was that was a fun one. That was a fun one. It was TPT5 was not that reliable. So, I was a little bit stressed that it wouldn't work, but uh but it ended up working.

Matt Turk: So, this was truly live and and presumably very very rehearsed, but but truly live.

Yann Dubois: Actually, the um right before we did that, like the last rehearsal, it did not work. So, I got slightly stressed about that. But, uh but yeah, seems like live life ended up working well.

Matt Turk: Yeah, no pressure, but yeah, that that uh that landed perfectly. Okay. Uh, very cool. All right. So, let let's uh unpack uh some of the things we alluded to in the intro. So, we started effectively talking about reasoning, and I'm I'm curious what reasoning means in 2026 that's any different from, you know, a conversation we could have had about 01 or or or three. Um, in particular, one of the claims uh of 5.5 and and also my experience as a user is that it's particularly good with with messy data, which seems to imply that um it needs to reason through ambiguity more. Um, what has changed?

Yann Dubois: What I would say is that 01 and 01 preview uh were really really breakthroughs um in in the research community about having model that can think and the longer they thought for the more like the higher likelihood they would be of being correct. Um so that was really a breakthrough but initially and if you look at like old blog posts you would mostly see like math uh math evals and also like maybe coding competitions but things that are really easy to test whether you're correct or whether you're not. Um and it also gives you like some suggestion about like how we were training some of these models.

Yann Dubois: Um, and how I see maybe all of last year and especially the end of last year and the beginning of this year is that we were able to take these algorithms that work with uh verified rewards like things where we can say you're correct or you're not uh to the messy real world um and really optimize for the utility that we provide to users and like making them more productive. Uh so I think that's what really changed.

Matt Turk: Okay. So it's the post-training reinforcement learning part largely.

Yann Dubois: Yeah, I would say that's a I mean there's also there's also another big part of it. Uh number one basically the first thing is that of course when you develop a new method uh the method is kind of fragile um and is not that reliable and like it's hard to to basically productionize. So this bot also improved a lot. Um but then it's also really basically we had a tool that we could start optimizing for for different things and initially when we were developing this tool uh we were optimi we were making a lot of simplifying assumptions of in the real world basically and and now we are removing these simplifying assumptions and and at least in posting we are able to optimize really like user utility and make sure that these models are useful and the tasks that we're looking at are useful and that's why also now current evals look much more realistic Um I mean if you think about GDP val or even if you look at like threebench pro or threebench these look way more uh realistic than let's say some code force or like coding competitions that we were looking at with 01.

Matt Turk: And still on the topic of of reasoning um what's ultimately the difference between 5.5 thinking versus 5.5 pro? Is that is that just more test time compute, more tokens and more time invested in solving a a problem?

Yann Dubois: Yes, basically it's just a question of of uh how much test time compute we pour into the model uh or we pour into this entire uh system that we're shipping. Um so we we've seen again and again the longer the model thing for uh the better answers we will get. The problem is that this these curves that we're talking about um are not are definitely not linear and like they there's some plateauing effect and they kind of look um lo logarithmic on some in some sense um or depending on which evals. So you can pour like two times more compute and actually only get like small performance gains. Um I personally don't use pro that much because I really don't like weight. I'm pretty impatient, so I don't like waiting for that long. And uh and I know that the probability of being correct definitely improves, but it doesn't improve like enough for for me to use it. Um but there are some people who use pro and who really love it especially actually for academic research and uh I know especially a lot of mathematicians who are using it and that's because they're kind of just have this in the background that is running for maybe one hour uh two hours and they don't really need to like iterate really quickly with the model um and pro is really good for that.

Matt Turk: I'd love to reconcile this with uh what you were mentioning about efficiency earlier per token. So is the idea that um you would be able to think longer but also be more efficient therefore solve the task better like how do those the the the time aspect and the efficiency uh aspect sort of interact?

Yann Dubois: Yes. Uh so if you go back to like the plot that I was talking about I was thinking about uh where on the x axis we have latency and y axis we have performance. We're basically moving this curve when we say that we improve efficiency more and more to the left. So we're becoming more efficient uh or like we spend less time to achieve the same performance. Um but what pro does is that it extends this curve. So it says like um I'm going to think for much longer but I will have a higher likelihood of being correct. But every iteration of the pro model also moves to the left. So it also becomes more and more efficient. The important part is um there will always be tasks where uh you just want to maximize the probability of correctness and you don't really care about latency. Uh for example, if I if I start a job before going to sleep um I mean the model has like eight hours like it should just think for as long as it as it can. Um and this is what kind of promo gives you

Matt Turk: and uh in layman's term like what what what does that mean practically or how does that work practically? If the model goes in a wrong direction uh then it would interrupt itself earlier is that is that one of the axis

Yann Dubois: so for the effic Okay so there's two things are you asking for the efficiency what does it mean? Um let me give you um maybe a metaphor from like humans. Uh if if you have a someone who know like someone who's an expert in certain domain uh and you compare them to like some undergrad that is like starting in that domain, uh the undergrad doing that task will probably take might take like one day, two days and we'll have to think through a lot of like the um the possibilities and like investigate because it never did a certain problem. while someone who's an expert in that in in that field will usually just like know what direction to take and it will it will not spend the time on like investigating 10 different directions because it knows that there's like one that is more likely to be correct. So this is the type of efficiency that we're talking about.

Yann Dubois: It's basically models that we where we optimized more on like real world problems. Um and as a result it was kind of trained to uh to figure out with a higher likelihood which paths of reasoning are are more likely to be correct. Um so this is this is a part on on efficiency. There's also what what you suggested is that um part of it is the model knowing when it's going down the wrong path. Uh but this is also something that we can um that the the model can be trained for with reinforcement learning is like knowing okay like that seems like not a great path. let me backtrack and let me go and and test something else. Um, and if you train the model less, it might realize it's in the wrong path much later.

Matt Turk: Okay. All right. So, it seems like um a lot of this uh goes back to uh reinforcement learning and post- training. So, let's uh talk about uh how the different components of modern AI systems work. Uh so let's talk about pre-training, mid-training and and post- trainining and spend more time on post-training since it's so important. Uh starting with pre-training uh first at at a high level and and realizing that you may or may not be able to talk about how the things are are done or what happened in the context of 5.5 specifically. You know, big narrative of last year was that pre-training was hitting a wall uh and was not going to yield much progress. that seems to not be the case at all uh in 2026. Uh can you walk us through some some ideas for what is happening in pre-training and why it's progressing now in a way that people hadn't predicted uh last year

Yann Dubois: for pre-training I I can't talk in a lot of details about what is happening internally. Uh besides that um the team has been really doing a lot of good work um and our models are really getting better and better. Um one one thing that I do want to um highlight when we're talking for example with efficiency um if you have larger models uh the amount of thinking time so the amount of tokens that will think for um will usually decrease. And the way that you can think about it is that um metaphorically the model already thinks through its weights when it generates a certain token. Um so you can you can decrease the number of like tokens that it needs to generate for thinking by kind of like increasing the size of the model uh that you're training. Um so so oftentimes if you just increase the the model size if you basically train uh pre-train larger models uh you will get better efficiency.

Yann Dubois: Um and the good thing with larger models is that they can be paralyzed better on on at inference time. So the even though you might think okay you actually generated fewer tokens but by a larger model uh so you actually might decrease the uh the overall efficiency of the system. This is not true because the larger the model is the the more chances you have to actually uh optimize for optimize basically for inference on on on GPUs. So you will be able to uh um to make them the overall system like more efficient. So that's that's one thing I wanted to say with like larger models that are actually giving you a lot of efficiency.

Yann Dubois: Um otherwise in terms of pre-training I think it's very interesting. I actually also thought maybe two years ago that pre-training was kind of hitting a wall. Um and when we see for example if we talk just about entropic I mean myth is seems like clearly just a much bigger model when you look at the cost um uh the cost of the model. Usually that's how you know by the way if it's a if it's a bigger model you just look at the cost um per token and and clearly they're getting very good performance uh just by increasing the size of the model. Um so I think the field was a very at least part of the field was surprised uh about that. There were a lot of conversation about hitting uh data walls and it seems like we did not quite hit it. So the larger the model is the more data it needs to ingest to be trained. Um and it seems like different companies kind of found different ways um to overcome the fact that that we don't have that much data on the internet.

Matt Turk: Is the next frontier or the current frontier uh for for data multimodel data? Is it synthetic data?

Yann Dubois: I think synthetic data can probably work well in a in a data um data limited regime. Um I think multimodal is an interesting one. Uh I I definitely cannot talk about what we do internally but like I used to work on multimodal uh representation learning back in the days and I always thought that it would really help uh kind of your reasoning abilities if you have a lot of multimodal data. Um and I still think this but but for example like if you look at entropic models they tend to not be that good on multimodal and they are still really smart. Um, so it seems that uh it's not as necessary as at least I would have thought in the past. Um, I still believe that once we go to embodied agents, embodied AI, you will learn a lot about the world. Um, and you will kind of improve general intelligence and usefulness to users um by learning how how the world interacts with itself. uh but at least looking for example entropic models it seems like they they don't need that much multimodal data to have strong models

Matt Turk: and by embodied intelligence so you mean so potentially robotics and so if you use a video uh that shows how gravity works and how a robot evolves in space then presumably that would be more useful is that is that the thought

Yann Dubois: yes the the idea the the intuition that I think many people had and I I definitely felt for a long time is that it's hard to understand the uh only through text and and they will be um it's hard to understand what like what physics is without really seeing what like for example you can't understand gravity without really seeing things falling um and when you look at our models I mean it's they kind of understand gravity without having seen that but it still seems not obvious like it seems still seems like they would get it more and like they are still kind of missing some common sense aspects. Um, so I do feel like we will improve the common sense of our model by having them interact in the real world. Um, but we are we're still pretty far from that I think. And by we, I mean just generally the academic community and and the AI community seems pretty far from that.

Matt Turk: Okay, great. All right, so going back to pre-training, mid training, post- training. Let's talk about mid-training. It's might be something that people have heard about a bit less. The term comes up a bit less. What is it and why is it important? Mid-training

Yann Dubois: um is just this this idea of something that's between pre-training and as you might realize from the name and kind of the post-training um part of this part of the pipeline. Uh and really the idea is if you have high quality data um that is more representative of what you really want in your final model um you should overtrain on that data. Um so taking a step back here pre-training what is it pre-training it's basically trying to learn everything from the world by learning everything from internet at a high level. Um the problem is that most things on internet are not really useful. Uh if you think for example about Wikipedia or like GitHub which is like coding data um it just seems like there's way more information in there than some random forums. Uh um yeah, some some random farms that may maybe not like have that much information like for example ads there's also lots of ads on the internet like you probably don't want to train too much on that. Um but in pre-training we train on everything and in mid-training we basically overweight this type of high quality data that we think is more useful uh for for training the final model and this is something I I can't talk about what's happening everybody but this is like something that that is happening definitely in all the academic community right now and in all the open source uh models have this stage of mid training

Matt Turk: great post training let's start uh at a high level by by defining what that is so there's reinforcement learning but that's not the only part of post training. What What else is there?

Yann Dubois: It kind of depends how you define the term uh and where you put the boundaries. In my mind, post training um including let's I'll take it from a very broad sense which includes all the reinforcement learning and like the training for our reasoning models. Um is just the idea of having something that knows everything about the world to making something that is useful to people. Um so pre-training I think about it or the metaphor that I like giving is uh you go in the library and you have a lot of books about everything and in theory you can find all the information that you want in the library but it's much more useful to talk to an expert who has learned these books and that you can ask questions to and they can answer they can answer and like they can understand like what you're actually looking for. Um so this is kind of the goal of of pushing at a at a very high level is like making something that is useful um uh to users and is like easier to interact with.

Yann Dubois: Um so there are multiple uh stages I I'll talk mostly well I'll talk only about things that are happening outside of OpenAI and kind of the the usual stages. Um there's usually some SFT that is happening which is supervised finetuning. Supervised finetuning. Yes. uh supervised fine tuning and that's um that's actually what early on most of the uh models that were portraying were only doing supervised fine tuning. This is the idea is that if you have humans that can give you uh the desired final answer. Um so if you have if yeah if you have humans that give you the gold answer you can basically clone the behavior of the human. H so this is what we call behavior cloning. Um the problem with this is that you will never get better than what your ground truth gives you. Uh and humans are actually pretty limited in many in many sense. So you will never like overcome uh the the the the human labelers that you you're working with.

Yann Dubois: Uh the reinforcement learning or reinforcement learning stage goes from behavior cloning to really like optimizing rewards. So the idea is I don't know what the ground truth is. I don't know what the perfect answer is, but here here's how I would say whether the answer is correct or not. And here are the things that I I want in the answer. And what you do is you start optimizing. You start having a model that tries to to get more reward basically optimize more um uh this this reward function that that's how we call it. Um and it goes beyond what you currently have what what like humans can do or at least the humans that you're working with can do. Um so this this I would say is the two big stages. Then in reinforcement learning uh that depends in like which models are being trained. At least in the open source community it seems that there are there are different ways of doing that. Reinforcement learning when you have verifiable rewards. So, uh, reinforcement learning where it's really easy to say whether something is correct or not and you can really kind of have a binary reward for this and that goes back to how we talked about 01 uh and 01 preview in the past.

Yann Dubois: Um, and then you have reinforcement learning without verifiable rewards where maybe I could do pair wise comparisons. I can say this this answer is better than this this other one, but I don't really know. I cannot quite say this is the perfect answer. Um so of course like it's a continuum and there's everything in between but I would say these are like the three uh high level uh things to think about when you think about post training in general um and how people are usually doing it in the open source world is that they take SFT uh they clone the behavior that you you can collect online or from humans and then once it's already at a pretty good level they just do this reinforcement to go beyond what we currently have because if you just started from reinforcement learning it would be very inefficient. Uh because the problem with reinforcement learning is that you have to stumble across the right answer basically because how how reinforcement learning works is you sample many times essentially from uh from the model that you're training and you say this one is correct this one is not and you say do more of the one that is correct. So you have to stumble across the right solution. So you're much better off first getting as much as closer as possible to the best you can do. Um and this is this behavior cloning and then doing reinforcement.

Matt Turk: Does reinforcement learning uh create new capabilities uh or does it make the model better at existing capabilities?

Yann Dubois: It's really hard to say because retraining when it's trained on all of the internet arguably already has all capabilities in it. Um so it's it would be even hard to answer this question scientifically. Um because arguably everything is is already there. What I would say is that if you look uh at models that we were training or that we're pushing like two years ago in the open source world uh for example I worked on one of them alpaca where we used 50,000 examples for SFT and like now when you look at reinforcement learning from from models like Kimmy or or or from deepseek models it seems that they are closer to 1 million data points. So definitely people scaled up a lot the reinforcement learning stage. Um and from this it seems that they've learned like new capability like this reasoning aspect this fact that you can check your answer and and uh and try to improve it. Um so you can you can really think for longer to get to get a more correct answer. So all this to say that arguably everything is already in pre-training but we were definitely able in the last one year and a half even even in the open source world um to have more capabilities after reinforcement learning that we used to uh before

Matt Turk: I heard several times that reinforcement learning is pretty finicky and and hard to scale and part of the reason why we as an industry didn't do uh reinforcement learning as part of the initial kind of LLM sort of progress curve was was precisely that that it was hard to to make work. What is hard about scaling RL? Is that a question of uh data sets knowing where the rewards are? Is there is that or something else?

Yann Dubois: I would say most people who did not work in reinforcement learning in the academic and like in research community up to two years ago probably thought reinforcement learning would like just doesn't work and is like too finicky to to work with. Uh I used to be that type of person and actually when I saw Chad GPT come out they had the this blog I was not at openi at the time. Uh I saw this blog that says that they use reinforcement learning and my first thought was I can do the same without reinforcement learning because this is just an over complicated method and this is actually the problem that we started working on with alpaca was exactly let's try to reproduce that only using SFT just by doing this behavior cloning.

Yann Dubois: Yeah. And like for example, Yanuk famously like gives like this metaphor of like oh the reinforcement is like the cherry on the top. So I think that was really like the intuition that most people had. Um it seems that after crossing a certain scale of um models that know basically everything about the world and what we call like good priors about the world. It seems that reinforcement learning just started to work and this is not only with LMS. robotics seems to have or get seems to be entering the same stage uh where they're realizing that actually it used to be very finicky but now that we use models that like know already everything about the world it actually learns pretty well um

Yann Dubois: now to answer your question about what is still complicated with reinforcement learning one is an infra aspect um so just like systems in general uh reinforcement learning you have at a very high level basically to sample as I said for many answers and say like what is correct um and what what is not and um and like this sampling is just very expensive and you have you have to do it at scale. Um the other issue that that also in the open source world we uh people are seeing right now is that when we are training more agentic uh systems you only know whether you're correct at the end of your very long roll out. Um so you get very little information per token of whether you were correct or not. And it's hard to say uh it's it's hard to basically do attribution. It's hard to say what part of your entire answer was the one that led you to being correct. So that's more of a of an issue on the machine learning side. It's uh the the the ideal world in machine learning is when I can say exactly like this thing was good do more of that. And the problem again with with uh with these agentic systems and and reinforcement learning aentic system is that you don't really know which part was good or not until you arrive at the end. That's another big issue from from from re reinforcement learning.

Matt Turk: What's the current uh frontier of reinforcement learning? It it seems like there's a jungle of acronyms like GRPO and uh other techniques. What uh what are you using? What are you excited about? What do you think is promising?

Yann Dubois: So, I can't talk about what we're using, but like for example uh in the open source world, GRPO seems to be working very well. Um and people used to have different methods like PO and and DPO and like people seem to have really converged to this one. Uh the big the big difference with others other methods is that um you again you do this like simple method that I told you about like sampling as many answers as possible and you say which one is correct. Uh so in some way GPO is a very simplistic method. uh and in general we saw over and over again in machine learning that the the simplest method that where you can scale up in terms of compute usually is the one that ends up working the best and that is kind of what is happening here um at least in the open source world

Matt Turk: as you described some of the challenges question crossed my mind uh you know you often hear that AI systems are not built they grown how you'd characterize it as well what part is science versus a craft or trying multiple things and then just keeping what works best in your day-to-day life.

Yann Dubois: Yeah, that's that's a great question. I think how it usually works is that it starts being craft. Uh people just try out many things and and they start building a mental model of what works and what doesn't. And over time we move to like from this like craft land uh to more science. science uh is or like more scientific approach is are rarely the ones that like first end up working. It's hard. It's very rare that uh you you take a really scientific approach and and you say um uh like this is the optimal the optimal thing to do and you do it and it just works. Like people just there's some sense of alchemy. people just have like a good flare for something and they make it work and then other people or that person uh starts trying to improve what we are doing by being very scientific. Um and I would say this this happens over and over um in uh in machine learning. Uh so first craft then science and both are really important uh but it's different stages of the pipeline.

Matt Turk: Great. Let's talk about evals for a minute. Uh that's uh a hugely important topic. Maybe to start, why is it so hard to evaluate a model in the first place?

Yann Dubois: Evaluation has been harder and harder as models become better. Um and that's because the tasks that we ask to the model become uh more and more general um and more and more open-ended. So like now I maybe just say like build me a website that does X. Well before in the in the past I would just be like hey like is there a specific bug in this in in this like implementation that you have and it's like much easier to say whether there's a bug because I can I can extract I can know a I can have a human that says here are all the bugs that you have and then you can apply that automatically. um while the the website one is very hard to know uh what is like the optimal answer because there are many good answers. There are many good ways of of building a certain website. This open-ended nature of models really makes evals harder.

Yann Dubois: Um there's also another issue is that models in specific axes are becoming better than the majority of humans and so we have fewer and fewer humans that can actually evaluate these models in particular axis. Uh so that's definitely a constraint. Another one to be honest is kind of cultural. Um most people want to improve the model and they they think that the best way to do that is kind of training the model when in reality finding issues and like making sure that we can quantify improvements is just as important if not more important. But there's always this like cultural gap. Um that was especially true I would say in the academic world up to like two years ago when evals were always fixed benchmarks were always fixed uh and even data sets were kind of always fixed maybe let's say four years ago um and there was like a mentality shift of like okay data is actually critical and now there's a lot of people working on data and I think eval was still not quite there people don't really fully everyone knows that it's important but like people don't really understand like how impactful it could be to work on evals

Yann Dubois: Um, so actually my first first product at opening I just came in and I was like I want to work on data and evals because I know that this is the thing that no one is is working on and as a result I know that's like super impactful to work on that. Um, and yeah the tide is shifting but like not fast enough. And is the pace of progress in model as a judge and AI evaluating AI is that is that moving as fast is that a distinct part of research or is that fundamentally the same idea or the same techniques? It's really fundamentally the same method. It's like nothing also most of the things that we do in in eval especially now that we have reinforcement learning could just be applied nearly exactly as is during training. So that's another reason actually why eval are so complicated is that every time you build an eval you actually build a way to build training data sets. Um so now you're going to optimize that training data set. Well not even if it's not that eval it's going to be the same type of data and now you're going to do super well because we have this generalization of of uh of capabilities that I was telling you about. You will learn that on that other data set and now you'll become really good at that eval and that eval will become u obsolete really quickly. Um so so that's also an issue with ULS. But yeah to come back to your question um the model as a judge it's really important and I think it's one one probably of the most important things because as we get like better models uh we have this self-reinforcing loop and we have this this like capability flywheel where better models become better teachers for other models. Um and this is really important for training but then you can also do the same thing for evaluation. So I a lot of my team works on that and I think it's really critical is to work on this uh module model as a as a judge kind of um uh framework.

Matt Turk: Okay. Fantastic. Um all right so as we get towards the end of this conversation I'd love to to zoom out a bit and get your sense for where things uh might be heading. Obviously, it's incredibly hard to make predictions on on AI uh you know, years out, but let's call it the next 12, 18, maybe 24 months. Is your sense that things are going to continue progressing or are we heading towards something that could feel more like a discontinuity?

Yann Dubois: In terms of progress, as I was said before, it's I think it's always continuous. Now, the feeling of discontinuity will happen. It did happen three months ago with coding or four months ago with coding and I think that will happen now in every other domains like most people are not feeling uh the same way like the like kind of the capability of our model and the usefulness of our models the same way as like coding um and like software engineering is feeling right now. Uh so this will definitely permeate I think through many other verticals. Um now in terms of like capability bump in terms of let's say the the verticals that we're already looking at uh I think it will be more continuous um and they will not be there will never be uh big discontinuities like most of them are always local discontinuities but you zoom out and it always just feels pretty smooth. Um it's not always like this but like that has been the case most of the time and I can definitely not predict when is the next big discontinuity.

Yann Dubois: I'm extremely excited about continual learning. I think we haven't quite cracked it. I mean, we have uh we have like Codex memories and that that is helpful, but it's definitely not like the the end state. Um I have a friend who always like tells me about uh kind again another type of plot that we should be looking at which is x-axis time y-axis utility that you provide to users and right now or like or or like usefulness basically of the models and right now actually most models at day zero if you just drop them in a company um arguably they are more useful than most new employees so they start higher at t0 Um but then across time they are mostly constant because they don't really learn kind of company knowledge. Uh they don't really learn like to be more efficient uh over time on on doing the things that they are doing. uh while humans learn really quickly and what is important is kind of this integral uh or like kind of the area under the curve um uh of these curves and as a result that I think like humans are still more useful uh in many cases um and that's why what we will need is to make like continual learning is to make the the this curve um now monotonically increasing over time uh and basically make models more and more useful the longer they work in a certain environment. So, I'm extremely excited about it. I'm actually surprised that we're not quite there yet. Uh, three years ago when Chip PT came out, I remember I was doing a startup with friends. Uh, and we were thinking about working on on continual learning and like personalization and and like memories in general. And we're like, ah, OpenAI is going to do that in in the next six months. Like they have all the data. They're going to figure it out and they have all the users and the models are going to learn super quickly from users. And uh, three years later, I don't think we're there yet.

Yann Dubois: Yeah. Um I think harnesses can really improve the capability of a model right now. Um I think given that we seeing this this really fast progress in terms of capability uh I personally wouldn't push that much on the harness that unless it's like the harness is um is something for like very concrete goal that you're trying to achieve right now. Um so certain companies like if they are focused on like a specific vertical they want to go from this like 80% uh maybe reliability to maybe the like 85% and like harnesses will give them that and I think that's like very important but like they will they will they need to do it while knowing that they will have to retune that harness in the future and I think that's that's totally fine. Um if you try to have like a general harness to that will like sustain over time uh I don't think that will work the harnesses for specific uh domains as a short-term thing that you need to do. I think it's there will always be so much you can do in harnesses and if anything I think everyone should do more of that if they have a specific problem in mind because we're leaving so much on the table without a good harness. Uh, arguably if we just I think if we froze the models that we have right now and you really worked on the harness and like maybe like we also spend more time like training with like a great harness. Um, I think people would really feel the AGI in every single domain or could already feel that in every single domain. Um, but given that we're not freezing it and we're going to continue training better and better models, uh, I think the harness we don't really understand what the final harness will be and it's not and like it will always change. ... There's so much space left for this last mile in different uh in different verticals and uh I would highly encourage people to continue working on that and maybe one day when we stop making horizontal progress which I don't think is anytime soon maybe we will start focusing on that but yeah that's not what we're doing now.

Matt Turk: Okay, well that feels like a very uh optimistic note at least for the startup ecosystem to uh end up on. Thank you so much, Yan.

Yann Dubois: Great. Thanks, Matt.

Matt Turk: Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.