There was a big change in leadership at Meta and Llama's future is unknown. So there's this big vacuum of influence which has been absorbed by the likes of Quen, Deepc, Kimmy Moonshot in terms of like who's trying to build things with open models and that's a big shift.

We're launching Almo 3 family today and just like every single models that we released before, we're not just releasing the final models. We're putting out all the details. It's like the first fully open reasoning model where we show doing RL and base models and distilling from bigger thinking models and there's a lot of discussion within the US that there's like good reason that we should own the whole technological stack and that includes open models. There are people that are really starting to wake up to this.

Hi, I'm Matt Turk. Welcome to the Mad Podcast. Today we have a special episode with Nathan Lambert and Lucas Sanei from the Allen Institute for AI for the release of the OMO3 model family. At a time when most open source releases are just open weights, AI2 is going all in on real openness, models, data, recipes, and intermediate checkpoints. In this conversation, we break down almost 3's architecture, the rise of thinking models, and the increasingly high stakes race between US open source efforts and fast advancing Chinese powerhouses like Quinn, Deepseek, and Kimmy. This is a rare, fully transparent look at how modern AI models actually work. Please enjoy this great episode with Nathan and Luca.

Guys, welcome to the pod. Uh, big announcement today and a big day for open source AI. Walk us through what it is that you're releasing today.

Thanks for having us. Yeah, we're launching OMO 3 family today. Um, so this is our latest uh family of open source models. We have a 7B model, a 32B model. We have models that can think, models that can follow instruction and use tools. Um and just like every single models that we released before um we're not just releasing the final models we are releasing you know the entire recipe we follow to get this model. So the data the intermediate states the evaluation frameworks all the details all the bits that uh people need to know to make uh models like

specifically this uh 3 base uh 7B and 32B. So what what are those? probably we have say five you know flagship uh checkpoints that we're putting out. Two of them are base models. That means there these are models before they get trained to respond to user instruction. Um so these are really good for folks who want to take sorry the bulk of our compute that we spend in pre-training these models and then they want to customize them for their use cases. So these are a two-based model. There's smaller one, there's more efficient uh that takes about one GPU to fine-tune for use case. And then there's a larger 32B that takes about one box of AGPUs to fine-tune.

And then on top of that, we have our fine-tune our uh post train models for various use cases. So there's models, there are a couple of models that are thinking models. Um so there's almost 7B think and almost 32B think. Um these are models that um you know just like a lot of the reasoner or like prom models out there uh they can spend um compute power uh inference time um to sort of think through a problem and solve it and then give you an answer at the end. Um and also we are releasing a 7B instruct model. This is a more like immediate model that gives you faster um responses. So, it's really good for like bulk data processing or or use cases where like you want to have like low latency in your responses.

I want to add more color to these things. I think Luca is underelling their base model. Um, we're going to talk more about this, but over this year a lot more people have been releasing open models and especially large open models, but um some people are starting to like not release base models. We have a bunch of like deepseek size giant MOE base models and a bunch of small base models. But for example, Quen 3 which everyone accepts as like a research standard and an industry standard like they don't have this 32b base model.

So um this base model is similar in quality to the best available which is like quens 2.5 32b is was still the best base model. Um the upside is that we have all the data so people can actually do some sort of like continued pre-training and hopefully make it a bit easier to modify and understand the behavior. So that's exciting for us though like the actual potentially best-in-class thing. It's also a fully open thing which is not something we get to say a lot at AI too.

Sometimes it's like oh we replicated this and now you can do it yourself. Like this is actually a good thing. And then 7B models which Luca was saying used to be like this huge industry standard where there's just so many of them. It's still a standard size category, but there aren't quite as many models there as there used to be. And this like especially the instruct models are less common. And this is up there with one of the best in the world there at that size category again. And I I just think of this like llama 3.18 is one of the most used models on hooking pace of all time. And this should be better. We're in our measurements, we see it as being better than llama 3.18. Hope that holds up for people. We can release more and fix it. But that's just like trying to give we might not be at this frontier scale but these are things that are still widely used in the world

and then it's like the first fully open reasoning model where we show doing RL and base models and distilling from bigger thinking models and all these things that people have seen a ton of times throughout the year. I think get another thinking model is like what is this one for? But we have all the data and we show people what to use for. So I think a lot of times with our especially open post training it's just like the data sets become a standard. So it's like our two loot 3 data set from last year which we used for mode 2 is in like the thinking machines tinker API and we want people to use this data modify it how they need to and look at the different training stages.

Great great uh fantastic. So uh to the data point uh talk about DOMA 3.

DOMA 3 is the data that um we use in pre-training uh for 3. So it's what we use for to create the base model. Um it's really three parts. There's the sort of pre-training pool. Um, this is like a pool of about 10 trillion tokens from which we have like an algorithm also fully open source to like sample about 6 trillion tokens that we use during training. Uh, and it's you know we have kind of new techniques there. is kind of interesting of like we uh do this technique where um you know instead of repeating at random documents to get more training tokens, we intelligently repeat the tokens that have the most value.

So we have that part uh there's smaller subset uh that we use during this mid-training phase. So this is like a more focused data set um with a lot of math high quality codes um sort of knowledge tidbits that you want the model to pick up um and then finally we have a set of um documents that are particularly useful to make uh models able to work with long context.

really excited about this one because like historically um of the data that is available openly out there for people to build their language model um you don't have a lot of like long document data so these are documents uh that we crawl ourselves they're PDF um scientifically they're mostly like science PDFs they're openly available um on the internet for crawl um we have a pipeline but it's also open source so everything's open source to turn them into plain text. Um, and of those we have uh like instead of like web pages that are like kind of short like 95% of web pages are um below 3,000 tokens. Um these are quite long. Uh we have about 600 billion tokens are longer than 8,000 tokens. Um so these are really good for people to like develop other ways for models to understand very long inputs. uh which is typically it's something that people are not able to do today in the open unless they are a big lab and they can you know acquire data that it's long enough to to do this phase.

Okay, great. Thank you. Um you alluded to some of this but talk about uh performance and efficiency.

Performance you know it's kind of it's very hard to measure performance a base model. So for like the instruct and the thinking uh Nathan will have more info about like comparative benchmark but like the base model is really good. It's a level of as Nathan was saying not that many people release the base model. So we're kind of limited there in terms of comparison but it's at the level of like Quen 2.5.3 or Gemma 3. um certain capabilities they have maybe a little bit better on some capabilities with better than some others. Uh but ser in the ballpark absolute performance of base model don't matter so much as as um an instruct model at the end you want to be like in the right band where the model is capable enough that then your post training team can do like magic on the checkpoint and make it really really good.

I would say in post training we're the best models that don't start with quen 3 and we're like reasonable to say that they are comparable to quen 33 like on some benchmarks we beat them and on some benchmarks they're way ahead. I think a lot of people are like we don't know what quen 3 puts exactly in the in the training data so we don't know if like some benchmarks they benchmark to max a little harder than we did.

I mean like we try to hill climb on benchmarks to make our model good. I think there's like always some some level of this but in that it's like in the same ballpark in plenty of things. are hoping that there's use cases where people that use Quen 38B or 32B are willing to switch over and get some value out of this and maybe modify it to their own use cases. But it's like Gwen also releases great models. So it's like ever a never- ending up uphill battle that motivates you to do better to try to get just like get close and compete with what they're doing. It's like they release these Quen 3VL their vision models and like on text only benchmarks it's way better than the models they released in April. So it's like h okay like that's the new baseline and most people don't know about it because they think it's just a vision model but it's actually a much better text model and it's like okay like the bars the bar is always rising

but at 7B scale Nvidia had NEAtron Nano V2 which is a 9B hybrid model which I think is like almost equivalent to our like our 7B model is pretty much equivalent to that. These are good models. There's not that many of them that are at in these size spans that are really strong. So it's like I'm I think we're happy to be there and happy to point out other people are doing great work here. It's not it's not like we can ignore Quen. That's a losing strategy.

Luka, just to uh drive it home, the concept of open source in AI, this different flavors of it. Uh walk us through what that means and where you guys are at.

That's always like a a a topic that gets um sort of overlooked a little bit uh in discussion. Uh but yeah, like um when when it comes to models, there is different level of um what people consider open source. majority of uh models that get um released um I think the best term to describe them is open weights um your Quen your Gemma your llama um you know Kimmy um it's what gets released is a set of weights uh that correspond either to the final state of model that's the most common or maybe final state of the instruct model final state of the base model

and you know there are plenty of cases where that's enough and you can build great software on top of it um there's an equivalent large set of cases from research to um application where that's just not enough. Um you want to have like intermediate state of the model so that you can customize it better. You want to have access to the data so you can maybe redo a step of the training while infusing your own data. you might want to have access to the pre-training data because you have this incredible research project that's going to change how we think about language models but you need to know what a language model is trained on

um so we want to support those use cases um so when it comes to OMO um if we can release it um we will release it um so you know we can't release I don't know our GPUs out to the world that's not doesn't work but uh when it comes to the data, the the intermediate checkpoints, the benchmarks, the software, anything we can, we'll put it out.

Um, if people ask, hey, um, you described this part of your pipeline, but you haven't put it out, we'll release that part as well. Like we've always got questions about like intermediate checkpoints during SFT or other fine-tuning stages. And like now we have intermediate checkpoints during our supervised fine-tuning for reasoning and for instruct and then also for like our multi-day RL runs at the end of these we have intermediate checkpoints. So people that are looking like a lot of people like to understand checkpoints and do research on them but don't have compute to train and now it's like okay this is this is all there.

Before we dive into the specifics I'd love to uh take a step back. Uh it's been a very intense year in the world of open source AI. Uh the deepseek moment feels like it was three years ago, but uh in reality that was at the end of January, so so 10 months ago and a lot has happened since. Nathan, could you help us uh maybe recap the the key events of 2025 for people to understand what what's happened?

Yeah, if I try to make a list of actual models, I'm going to forget some because there's so many that are notable. I think starting with deepseek as you mentioned is definitely the important thing and then if you talk to people building models in China that a lot of the consensus is like deepseek showed us that AI could be a big deal and then a lot of these companies were like oh we should we should do what they did. So there's just kind of a ton of labs that have popped up over the year. I think known players in in addition to Deep Seek Quake Quen and I think like Z.AI AI and Kimmy Munch had already kind of existed and like these really stepped up to be much more known names

especially if you're following western like SF centric discourse like these these are things that people are using and talking about which is a kind of big change but there's just this huge mass of models coming from China you have everything like like ant group is releasing trillion parameteres with really strong benchmarks mtoan which is like the Chinese equivalent of door dash which is just like another big tech company in China like the standard way of developing language models has become to release them openly and like that whole ecosystem is going forward with this figuring this out

when at the same time like there was a big change in leadership at meta and llama's future is less unknown which was really like the paradigmatic the like the definition of open source AI and that kind of that that line of thought just ended so there's this big vacuum of influence which has been absorbed by the likes of Quen, Deepseek, um Kimmy Moonshot in terms of like who's trying to build things with open models and that's a big shift and

I think there's a lot of discussion within the US that there's like good reason that we should own we should at least have influence over the whole technological stack and that includes open models and because realistically it's the big tech companies in the US that'll capture the downstream value of that from having the researchers be in close proximity and speaking the same language and used to the infrastructure. I think this is something that we've seen for decades in the tech industry. So I don't think I need to explain it that much and there are there are people that are really starting to wake up to this.

I think in June July is when the Chinese like model providers were really becoming like you cannot ignore them. That's when we had the Kimmy K2 instruct Quen was releasing a lot of their big models like Quen Coder um GLM 4.5 from Z.AI and that's kind of just continuing now. So I think when we're recording and releasing this podcast, there's a lot of interest of like what are the US companies going to do to respond to this. I know that like Nvidia is making a lot of noise here. They invested a lot of money in reflection and there are other players that are trying to get going.

But like urgency and we don't have a lot of compute AI too, but like if we can make a dent in this and some model sizes that people actually use, I think that like we focus on researchers. I think dense models are great for researchers. They take a little bit less compute and engineering resources to use and it's like I I do I do think that there's more if you look at this podcast in the coming months I do think there's going to be look like there's a lot more labs in the US participating. I mean open AAI has released some models. So it's it just takes a long time for the norms to shift in in the US where they're just established in a different way.

And then Quen is is widely widely used in a way that people may not have completely realized, right? There was a as an anecdote the the example of Airbnb talking about using Quen over Chat GPT a few weeks ago. But do you have any sort of stats or anecdotal evidence on on the usage of Quen?

The other famous quote was a Martin Casado quote in the economist where he said 80% of companies are building on Quen. That has been corrected where it's 80% of companies building with open models are using Quen which is like 16 to 24% of the his portfolio which is still a lot like like it's a meaningful amount of people are trying open models for things and most of them are using Quen

and then there's the likes of like cursor released their own model like composer 2. It's accepted that it is built on a large Chinese whether of some sort that was released openly. Um there's some obvious tales of like it switching to Chinese and things like this, but that's just like that is the sort of company that doesn't want to pre-train their own models, but has immense value in specifying models for their use case that is just going to build on these on these great models.

And I think they would want more options to choose from as they try to sell into more markets. I think realistically it's a thing where a lot of US companies don't want to deploy Chinese models. I think currently a lot of the stated reasons are just unknown unknowns and things you can't prove. Like you can't prove that the models aren't doing certain back doors where I'm fairly certain they defin they they aren't now. But like just because you can't prove it makes this kind of weird market dance which is like yes these are stocastic things that are kind of amorphous.

And it's like I don't I don't love being in the middle of this as a researcher. But it's like I would like to just provide information and good things that people actually really want to use and leave all of the geopolitical and other messaging to people that have probably realistically like way more on the line than I do. Like I don't know. We work in a nonprofit. I have my dog

Why why do you think this uh happened uh that the ecosystems developed in in this way that the US was uh very uh commercial closed source and China very open source?

Historically the US has a lot more willingness to pay for services. I hear anecdotes from people that know China a lot more than I do that are like yeah mediumly large like billion dollar plus valuation companies in China will just like pirate SAS software. I don't like it that sounds worse than it is, but it's just like I think that the thing is that US companies are used to paying for services and an API model and paying for tokens has been proven as a very good like selling tokens is a good business in the US right now.

I think there's a lot of debate over profitability but the demand and usefulness of these tokens is high. So I I have a lot of belief that there can be profitable businesses from selling tokens where I think that like AI will be embedded in very different ways when it comes to Chinese companies. And I've talked to a few of these labs and they're like in order to sell into the US market like they will not pay for they've said this they like US companies will not pay for services. So like they don't expect enterprises to sign up for the the Kimmy coding plan in mass, but they're like we have a chance that they'll use our models and it's like that is a practical way to influence and getting a piece of the sharing pie.

And it's like the people building these models in China know the same things about the different ecosystems. It's like that's why I've enjoyed starting to talk to them. I was like these people the same thing. They they see the same constraints. It's not that complicated. So they're smart enough to know that if they drop really good models like people in the US can't ignore it and like that's their way to like have a part in this ecosystem. So there's a mix of like the deepseek standard and then they're kind of like yeah like this this is this is something that works for us. Let's let's keep doing it. It's it's getting them a lot of mark like mind share and some use in prominent ways. So so I think it makes sense.

Today's release and you guys work is a part of that uh American response to uh China's rise in open source AI.

Like I would say I launched Adam in July and thought it would get more visibility but now I'm getting like a crazy amount of media inbound and press and brown and everybody wants to share the plot. So I was like, "Okay, I guess I was just four months too early, but that's what I really am." It's like just today I saw Bloomberg like published a post that pretty much had the same title as my Kimmy post from July. I was like, "Okay." Like I'm gl I'm glad that people are paying attention now is like better late than never.

Okay. Well, congratulations. Best form of flattery. All right. Switching tax and um in an effort to to to make those conversations educational for a broad group of people. Uh so one of the key aspects of the release is the thinking model. Could you remind folks uh what a thinking model actually is versus other forms of models or prior generations of models?

A lot of people have heard about inference time scaling which makes sense which if you spend more compute at inference time you get a better answer. A thinking model is really a way to train the model to exploit that a lot. So you spend a lot of tokens which is the tokens are usually hidden from the user as like a long chain of thought and the model therefore kind of has this step change where it's way better at math tasks, coding tasks, agentic tasks.

I think we we have some like our future plans are adding more tool use to the model. So we're not talking a lot about like agentic search or agentic code execution on the fly and stuff for this model. But like building thinking models is the gateway to doing a lot more interesting things like cloud code or like maybe we'll have all code next year and all these things that we want to do like like the thinking model has just been the thing in 2025 that use a lot more compute per answer model gets way better at various things.

I don't like thinking models but it's fine. No, they're good. They're very useful. Um it's um thinking models are really like work mode and uh regular instruct models are usually more fun to build. They can be more quirky. Uh but yeah, I think that's that's really what where they're like 90% of the cases especially like userf facing cases folks I'm okay spending time waiting for these models to craft a better answer. Um there's still a space for models that can respond faster. Like you see like stats that Google released about like adoption of Gemini Flash. Um and that's where like non-thinking models that can at least approximate have a good approximate first answer are really useful. They're also more fun to build. Uh but yeah, thinking models like where the future is especially when it comes to like um agents integration.

before we go into uh the pipeline very specifically of the Almo family um because as as you alluded to that's one of the amazing things about open source is that we can uh in a discussion like this uh truly understand uh how the model works versus you know other conversations with commercial players. So before we go into the pipeline, I'd love to talk a little bit about uh you guys, your your backgrounds and AI2 which is a very important player in the ecosystem that um people may or may not have heard about. So who wants to go first?

I sort of stumbled into this role uh by just picking problems that are interesting. Uh so my background originally from Italy, moved to US for PhD. My PhD is in information retrieval. how do you build search engine to just simplify a lot. I slowly got into more and more uh the sort of natural language. Um first joined after grad school I joined Amazon. I was working on Alexa at the beginning working on like the search part of Alexa and then I got way the the actual part where the users talk Alexa is the interesting part.

So slowly moving towards that initially joined um AI2 working on um a project called semantic scholar is still active um it's a search engine for academic paper um and there the interesting interesting bits were actually like the interacting with users and less so the actual uh text of of the papers that you were searching on and then the the way I got into LLM is uh and then building language model is really intertwined of how AI2 got into building language models. Um there was it all started around um uh what is it November of 2022. This is around the same time um charge got released a bunch of um researchers AI2 this is like individual contributors um not uh the it was not direction from the top as a bunch like a very grassroot initiative too a bunch of researchers got really interested in build building a model that would be um fully open

um AI2 um had already built um sort of proto language age models around 2017 2018. Um so a lot of the interest was in you know recapturing expanding that line of work. Um so a bunch of us got together start started planning um got in touch with um a few companies who might be interested in supporting these initiatives. We got an initial grant from um AMD at the time that was about 2 million GPU hours and so had the idea had the researchers interested. We had the compute. So we went to leadership at that time and sort of told them hey we're going to go do this thing. Um I hope you're okay with it. One of the nice thing about it too is like at heart we're like a research lab. Uh, so everyone was like, "Sure." Um, it you figure everything out. Uh, just have fun.

Great. All right, Nathan. How about you? So, you you're a man of uh many talents. You do research. You write uh this uh very interesting um uh blog newsletter called Interconnects. Uh you do podcasts, you do like a bunch of different things. So, tell us about your journey.

Yeah, I say I wear many hats to try to get the things that I want to do done. Um, I showed up to Berkeley as a EE mostly PhD admit in 2017 and then I saw that AI was happening and I decided that I want to try to do this which started by going to all the names that people know like Sergey Levan and Peter and asking to the BN group and then they respectfully say no and then starts the long process of becoming um learning how to actually do it without being directly embedded in these elite groups which was a mix of robotics and reinforcement learning. and finding my way there. So my PhD was in mostly modelbased reinforcement learning and then my one research avenue job was to go do join hugging face when they said they were going to make an open source version of deep mind to do a bunch of research.

Realistically my job was not that impactful or useful at hugging face until chatbt came out and then I was like oh I should maybe just learn about rof and that got very immediate traction as somebody trying to work in public with the team there. So like Lewis Tonstall and other people at Hugging Face are still doing a great job on this and we worked together for a while and then mostly I was just like getting burnt out on remote work and met Luca in Hawaii at a fun conference and was like wow I could have real life friends and I joined the AI2 to work in person and tried to do the same thing

which kind of takes an evolution of the OMO story which was just like I had a lot of motivation on trying to figure out these what what was mostly reinforced learning from human feedback at the time and make versions of these post-training techniques public and then that kind of evolved through both Elmo and we have our like post- training methods that was named Tulu which is like we we spent a long time to try to replicate um what we thought was close to llama 3 post training with multiple stages and optimizers which is the project that like came up with the name reinforce learning with verifiable rewards with a bunch of people so it's kind of this evolving journey at AI2 to and in in search for impact which is what we think people are actually doing and then like the EOS largely the opportunity that Luca and I and others AI too fill is that like there's so much money in AI and it only becomes increasingly so that the amount of people that can talk about these things in public and educate and um get more people involved by spreading knowledge is ever smaller. So I I describe my career journey as a lot of it is filling that vacuum and and thinking about what's impactful there. So it it kind of pulls you when when there's such a when there's such a void, it it has a sort of gravity to make it clear what you should be doing.

AI2 was uh started by Paul Allen right? AI2 stands for Allen Institute uh for artificial intelligence. U this uh you you mentioned some grant Luca I u and I think earlier in the conversation we talked about a recent grant as well. I saw uh that it was 152 million from NSF and Nvidia. So uh what is AI2? Uh how did it start? Has it founded at a high level? So we alluded to some of it but maybe a few words about uh AI2. Just maybe one last question like in terms of size like what are we talking about? Like how many of you guys are there?

200 people between you know the research stuff, engineering, um cons um and and other support roles.

All right, let's let's uh as as previewed, let's uh switch tax uh and uh go into uh Almo3 uh/lo thinking/mo reasoning, whatever you guys end up uh calling it. Uh and I think uh it's it's it's a perfect um opportunity to talk about how uh those reasoning models actually work. uh you know in prior episodes of of this podcast we've had great conversation with folks like at anthropic or open AI but um not not surprisingly there's only so much they can talk about and the beauty of what you guys do u which is the very uh essence of it all is to make it open and um and accessible to everyone. So I I'd love for us to talk about uh the whole pipeline from pre-training to post-training the different parts and um and make that super educational and explain in plain English what part does what. So uh could either of you start with just a high level of architecture of what the various uh you know subcategories of the pipeline are and then we'll go into those one by one.

So we're going to talk about six stages. One is like large scale pre-training which is this training on all the internet predicting next tokens. Two is what we call mid-training which is debatable whether or not it actually should exist. What technically it is is you train on higher quality web data and with a change in the learning rate. Three is long context extension which is absolutely essential for these reasoning models because they generate so many intermediate tokens before um sharing an answer with you.

And Luca has a lot of battle stories from that. And then we go into post- training which in our case like like those three building blocks of pre-training are I would say more set and super essential. And then post- training you when you approach this you have a bag of tools which are like optimizers and you apply them in the order that suits your model depending on size capabilities you want. So we'll talk about things that we did which is like instruction tuning, preference tuning and then we did some reinforce learning with verifiable awards again. But if like if we were to train a model that was 10 times as big like all this post- trainining stuff would change but the pre-training and mid-training and long context I think would actually become looking pretty similar. So it's kind of a difference across the two phases of training where post-training is like a bit of an art and you have to do what is best for your specific use case and and that'll change but we'll kind of go we can go through these too.

Okay, great. And uh Luca, you're the pre-training guy and Nathan, you're the post-training guy, right? Is that fair?

I think the the way I like to think of it is um the pre-training phase um it's really like um a very expensing expensive initialization of the model. Um, right. You want to like when I think of like, oh, what do we want to um what is a good uh final set of weights that I can pass to to Nathan and the rest of the post training team is well, I want a model that has great knowledge about the world. Um, and it also can sort of you can start seeing sparks of of of capabilities uh that you will want to model that then you know you want to chat with have have great capabilities. So um it is a very expensive and very compute inensive uh way to like create an initial models out of like what is essentially like random parameters. Uh but it's all about like yeah let's let's have this model have like a lot of knowledge of of world facts and and information. Um and let's have it so that it can start behaving a little bit um like a chat model so that when we pass it to post training and you have this reinforcement learning uh there is some some behavior to reinforce uh and to give rewards on so the model can pick it up.

I would say that generally the reason why discussions are hard right now on whether or not people should care about pre-training or post- trainining is that we optimize pre-training for multiple years and then there's a lot of um kind of untapped potential on this type of RL where like a couple like what is said is that open AI figured out a whole bunch of tricks to get 01 to work and then it kind of showed that this area was possible and then this year has been a race to capture lowhanging fruit on RL where I think that's kind of the biggest story is why we have all these crazy new models that appear like 03 with this thinking and tool use which are just downstream of like oh we could do very different things because we've done such a we have such a good platform as these malleable pre-trained models that we've been iterating on for a long time that this RL stuff we just kind of could have tapped into it much earlier but there's a lot of potential so like yes the rate of improvement right now in RL is higher but in in the end of the day it's going to be a dance between both of them where you need a better base model like it's said very commonly that a better base model and a bigger base model is much easier to improve with RL. So like if you take that as one of the core things of doing RL research, it's pretty obvious that pre-training is very important to enabling that.

Thanks for that. So uh let's take those uh six modules turn by turn. So let's talk about pre-training. Uh what did you guys do specifically for this model?

Pre-training is very interesting. um the way uh we sort of planned so a a good background is to have is that um pre-training all that happens during we have to be very methodical in how we do it uh because uh first of all it takes a long time to pre-train I think it's standard practice among the uh frontier labs to try to cap your big final pre-training run to two months uh not more than that uh but to get to like something that will not crush and burn in two months uh during these two months uh you have to do a lot of preparation around it. So we're really um everyone who works on prey is fairly methodical um and just to sketch out how that works. is usually you have a sense of okay the duration of this running is fixed. Um the number of of GPUs I have available is will be fixed. Um and therefore you know you write the fastest possible code to train this model. Uh you have these three you can figure out okay how much data can I show my model. Um in our case that number was like six trillion tokens. Um given that number um then we go back and we we figure out out okay where what are the best six trillion tokens out there. Um and the way you figure out is a combination of like what data you have access to. Um you know we want to do this with we want to eventually release the data. So we we limit ourselves to data that is um publicly available. So either um internet text or um PDF documents that you can find on the internet or code that you can find on the internet. And then among this pool you you know our initial pool was uh closer to 300 trillion tokens. You shrink it down till you reach your target number and hopefully as you shrink you only keep the best part uh of this. Um so you remove duplicates. um you have way to judge is this document better than this other document. Um you know we have a way to evaluate the capability of the model. So you pick you know if your evaluations once I don't know medical documents because there's a medical test there you figure out how do you pick documents that have good medical information it may be to the expense of some other domains. Uh but yes this delicate balancing act uh to find this data

Yeah, let's get into that. So that phase two, so mid-training. So again, a term I I personally hadn't heard of before. And uh Nathan, briefly describe what it is, but uh just um double click on that.

Um heard that some labs instead of mid training they call it tail patching. uh which I think is a much better term and and you know the term is like at the tail of training a tale of pre-training you patch the model so that the things that hasn't learned during pre-training you will learn after um you learn at that phase um and of course once you do when you do that you also need to make sure that um the model doesn't forget um stuff that during pre-training so that's why like you you mix some of the best data from pre-training you do carry over

And then you mentioned long context which is the third stage in the six stage uh pipeline. So why why the focus on long context and I guess what does long context mean in the first place?

You want these models to be able to work with very long sequence of text both as input. Imagine you want to give a I don't know a collection of documents and you also want this model to be able to generate a lot of text uh in the output. um especially now that you have this reasoning traces, right? This this thinking tokens. Why don't we train from the beginning the model to be able to do that is because the longer the input that a model is trained on, the slower is the rate at which it gets slower. It's um uh higher than the length of a context is is a quadratic slowdown. Um so we definitely don't want to do the entire pre-training at this extremely long sequence but as at some point we have to teach the model to actually work with this long sequences and we save it for the very end um so that um you know we can do it in an efficient way

All right, so that's uh the pre-training world between pre-training itself, mid-training, and long context. Uh so, now let's switch to the post training world. Nathan, if you will. So, starting with um SFT, which stands for supervised fine-tuning.

Yeah. I think one of the things that one of the things especially for a model like Mo where we're scrappy and putting everything together over time is that one of the biggest changes is that when reasoning models become popular, the like quoteunquote invogue evaluation suite of the industry shifts to add a whole bunch more new things in. So one of the things that happens at every stage is like even if a lot of the data has overlap is that you mix it in a different way. So I think like Kyle Low and May Chen as that's another researcher and intern like did this whole mixing procedure that we use across all all these stages just to like upweight the math code and and reasoning stuff to to make sure that like what happens later in postrading is much more tractable and that all this stuff is set up. So that's like the type of thing that we have to do that's kind of baked into everything. And then post- training um I think for this model everything we're doing is operating in the assumption that this is about like a 7B model. We are very narrowly focused and therefore we're going to do what many people have done which is called distilling from bigger teacher reasoning models. I think distillation is described as when you take the outputs from one model and then you fine-tune on it later. I think there's been a lot of broader discussions on this in the community. And then this supervised fine-tuning stage or SFT or instruction tuning is all about just getting the best traces from reasoning models out there or the community and then just teaching your recently trained base model to behave really really closely to what is going on there.

So that in our case we took a mix of existing data sets like open thoughts 3 and modified it which is from bespoke AI labs a startup and then we also generated a whole bunch of new data. So we ended up using a mix of teachers from like Deepseek R1, Deepseek R10528, which was their updated version, and then Quen's reasoning model, QWQ. It's like these tend to be pretty strong teachers. And then

why is that? So you you you have a pre-trained model, but you um for supervised fine-tuning, you distill using a a different model. Why is that in in simple terms?

essentially because our small model is not going to be able to output as strong of text even if so there's a kind of a fork in process where I'm talking about a small model if we had a bigger model what we would do is do a lot of reinforcement learning to start and the model then would take time to learn these interesting behaviors and have strong performance but with a smaller model the ceiling on that is fairly low like it just doesn't have the capacity to learn from these harder math problems so what the common practice is is you take the absolute best reasoning models you can get that are openly available with a good license where you can just generate new data yourself and train on it and release it to the community which is something we've been seeing a lot of this year

All right. So let's uh talk about the next uh stage in the pipeline. Stage five uh DPO and preference tuning. What what is that? What does that do?

Yeah. So this is one of the things that is um thought of as like hey let's try this. We're not sure if it'll work kind of later in the process when you spend a lot of time on other things and it it works very well. I think um DPO or direct preference optimization is not exactly new. I think it's a it's a way of optimizing for preferences. It's related to this whole ROF thing that we mentioned. Techn technically speaking in one sentence it's a analytically derived um loss function that like is essentially applying stoastic gradient descent to the RHF objective. So it becomes much easier to implement than other things. And we use this in the past with mo 2 with tulu 3 tulu 2 other. And the question was like can we apply this out of the box on top of a reasoning model and we knew that it works in many different situations because we weren't sure what would happen with these long reasoning traces being included in the loss function and so on.

So then essentially like there's a student Scott that has been working on this what he calls the delta learning hypothesis which is like a intuition for understanding DPO as being more about the contrast between your chosen and rejected examples. So the core of preference learning is that you have kind of pairs or some grouping of completions to the same prompt. So you have one question with multiple completions and his intuition and work is showing that this contrast is more important than the exact magnitude of goodness of the answer. So what he did is he spent a lot of time and trying to come up with a good pairing of reasoning models which is they're open source so that or like they're open weights and they have a permissive license and they include the reasoning traces because we kind of need this and you need them to be sufficiently well spread about. So we spent a bunch of time generating this data and doing some like normal kind of like h let's f fiddle with the learning rate and small things and and it's kind of just like yes this works I think like after we did it we saw that hugging face did something similar with small LM so they trained like a fully open like 3B model where they pre-trained it as well and the funny thing is that like we converged on using the same Quen 32B and Quen 06B. So like the Quinn the problem is that these small Quinn models and these small public reasoning models are actually so strong that getting a sufficient delta to another model to apply this preference learning technique was was kind of hard.

All right. And to to complete the the journey since we started talking about um RL so RL VR reinforcement on verifiable rewards let let's spend a little bit of time on that six stage uh in particular Nathan I understand that's your baby uh or you're one of the fathers of the of the the baby. Uh, do you want to walk us maybe a little bit through the the history quickly?

I mean, I think that I'm the person that got to bring it publicly to the world. I it's well known that people across industry have been doing this for years and then the technique started to get far more impactful. It's it's broadly taking existing reinforcement learning algorithms or downstream evolution of like proximal policy optimization PO which is like an evolution of reinforce and then deepseeek had their group group relative policy optimization. I always try to say like group robust and I think it's group relative. Um and like all these algorithms are really quite similar and it's you're you're training the models with whether or not they got the answers right or in the case of code whether or not the tests execute and don't fail.

You want to give the the plain English definition of RLVR versus RHF?

RLVR and verifiable rewards is in the name. I think essentially the reward that you get from the quote unquote environment which is like the completion or the greater is whether or not you got the problem right. RHF the reward is some is a essentially a reward model which is rating the quality of the response based on a proxy to what humans would like. So, it's described as being a much like the RLVR reward is much easier to understand because these reward models tend to have a lot of problems and you can overoptimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about where RLVR is much better matched to like performance characteristics rather than style.

the everything that you guys have have described because um in particular it sort of highlights the complexity of the systems like the multiple stages uh and I love what you mentioned Nathan a few minutes ago when you said the pre-training is scientific and uh post-training uh my words not not yours but my interpretation of your words was like it's a lot of tinkering and putting things together in a way that you hope is going to to to work and and and truly diving into how those models work on the one hand, but on the other hand, you know, each time you're uh like open a newspaper online or go on Twitter, like everybody's talking about uh AGI and how we're almost there and how it's going to change everything. There's there's a little bit of a cognitive dissonance between like the the the reality of uh trying to make those models work with all the unbelievable progress that we've seen of course but like uh you know that on the one hand and the discourse on the other hand. Uh Nathan you've you've you've had um a much more I would say tempered view of AI progress uh compared to some uh AI researchers. you had a a great blog post very recently uh that you called Thought on the Curve. Um and I'm I'm I'm curious what your latest thinking is and and look obviously feel free to jump in any time but in terms of um uh what you described in that essay in a prior one as um complexity and complexity complexity tax uh which again like in view of the pipeline you just described like it's one starts to understand the level of sheer complexity that's involved in all of this.

Yeah. Yeah. So ultimately I I definitely describe myself as lightly AGI pilled and I think you have to be to appreciate the magnitude and gravity of the situation that we're in. But also I think that the like I'm very far from believing in any sort of singularity being possible due to these things like complexity like on one hand we talked about all these things which are low hanging fruit to improve the models and I don't doubt that it I mean like you Shto commented this on the pod and other places where like at these labs they still see lowhanging fruit in improving the models in many ways I don't think that their approach feels that different they've just been refined it that relative to what we're doing but at the same time as things get complex with tools and and like adding more layers to the stack and you have to build a product to scaffold it. It's like if the requirement to get the best out of cloud is to use cloud floor cloud code which is some magic product and prompting relative to GitHub copilot like this is one thing that you're going to need to get right onto the in order to get AGI along with all these tool uses and stuff. So it's like as any system gets more complex the pace of change is slower. I think any tech company has seen this and then realistically there's going to be physical constraints on the amount of infrastructure that we could build.

So this belief is simultaneously giving us these new data centers and I personally think hopefully new power generation but there's there's a cap and for having a in order to like all these things are plateauing and then you 10x the compute and you get a big jump like you can't do this forever. So realistically there's going to be some physical constraint that kicks in at some point. But balancing that with complex systems and the lowhanging fruit results in like I think these researchers are going to grind out improvements for multiple years but never in a way that results in this kind of accelerating well that we get drawn into. So it's kind of like it's like you I don't know some ways it feels like I'm having my cake and eating it too but it seems like the likely outcome if you look at other types of technology.

the progress is also going to be incredible and normally the the growth in capability of these models but like having working on one I think it's very unlikely that we will see like a discontinuity at any point that has nothing to do with whether like we'll get to a definition of of of AGI or or super intelligence that people are happy with. we will get there. It seems unlikely that the moment like it's going to be a looking back kind of kind of exercise of like oh these were the important milestone and this is what really worked. It's building this model is so much a collection of like refinements till to unlock the next stage that it's it's going to be this smooth trajectory.

so to play it back you're both saying uh yes to AGI but no to discontinuity slash singularity and one is it fair and and and two if that's um what you're saying then uh for AGI you saying the current paradigm basically what we just described in the last hour of pre-training plus RL gets us there.

I think the AGI word is actually pretty not useful. I think that how I describe it is that big tech has all collectively realized that these language models plus scaffolding is going to unlock absolutely incredible value. And I have very high probability barring extreme geopolitical situations that big tech executes on this vision across the two to five years to build 95 to 98% of the way there of what you can do with our physical power constraints and what an LLM's ability is. And I think that that will be extreme like the transformation from that by 2030 is going to be so powerful across society.

there's a bunch of longtail like there's going to be mass societal readjustment to what the internet and media and information means and like within 5 years and that's mostly why I do this and I think like whether debating whether or not it's AGI is kind of secondary to the fact that this is coming and we want people to study and understand what is happening

The other part maybe on a more positive note is like if if the scaffolding is what really moves a lot of like from from you know raw capability model to like something that actually has meaningful impact that scaffolding is not just like oh only the labs of people trained models can do it like the the number of people can contribute to that both in terms of people with technical expertise people with and people with nontechnical expertise it's it's much larger if the scaffolding is what really moves capabilities what gets us to like this incredible technology being realized then the number of people can contribute to it is not just those who work at Frontier Labs. uh there's a tremendous amount of technical work to do uh but also non-technical as soon as you start integrating in uh this technology in the life of real people as as soon as you start working on you know highstake medical application or other highstake domains then you know large amount of like population can contribute in making this technology better and make it work for for everyone just the base model uh like I feel like the number of people can really can help making this technology really work for everyone. It's large. Everyone in society is feels like can contribute.

All right. Well, that feels like a wonderful place to leave it. Uh thank you so much both uh not just for this conversation but for all the work that you're doing uh in open source frontier AI uh which uh you know feels sorely needed and uh extremely important. So, uh, really appreciate really appreciate the time and and all the thoughts. Thank you so much.