Host: Joining us on stage is the co-founder and chief science officer at Hugging Face, Thomas Wolf.

Thomas Wolf: Hello everyone. Hello Olive, nice to have you on stage.

Olive Song: Hi, nice to meet you. Thanks for having me, yeah.

Thomas Wolf: So I think you're in for a treat today because you just saw GLM, which is current number two on the artificial intelligence leaderboard. I take Fable out because nobody can use it. And now we have number four. So basically you will have all the top models, at least the top open source models in a row. And we're very lucky to have Olive who has a pretty amazing path in life.

She came to the US, Pennsylvania. She was studying, doing a PhD at NYU in the lab of Yann LeCun working on JEPA, but we decided we won't talk about JEPA today, right? Something for another day. And then instead of joining Hugging Face, which was in New York also at that time, she decided to go join MiniMax.

So for those who maybe don't know all the neo labs around the world—and you're forgiven because I think there's like 64 neo labs right now—MiniMax is one of the top of what we call the AI dragons in China. So these are the new—there's DeepSeek which is very well known now, Moonshot who does Kimi, and GLM that you just saw, and now we have MiniMax. They're all extremely good, fighting for the first spot.

The latest release of MiniMax was M3 just earlier in June, which was the top model at the time, top open-source model. Very impressive. There's a lot of very interesting things about this model, so we'll quickly dive into them, and then talk a little bit about what's specific about MiniMax, what's great there.

So maybe Olive to start a little bit: can you give us your view of M3, what you like about this model, and how was the release?

Olive Song: Mhm. Yeah, we released M3 earlier this month, and it is a smaller model with around 400 billion total parameters and 20 billion activated.

Um, but it is very capable in terms of both coding performances, and also it understands vision. So that's what open-source models don't usually have. It's not just that the model can deal with coding, but it can also understand videos, images, and it has a super long context of 1 million tokens with our new architecture called MSA, MiniMax Sparse Attention.

So we really put these three things together because we know that they will be very important in future AI applications: coding capabilities, agentic capabilities, longer context, and multimodal understanding. Yeah, I think that's what is very interesting about the model.

Thomas Wolf: Yeah, so there's a lot to unpack in this model, and it's still, I think, the only top five open-source model that is actually multimodal, so we need to talk about that.

But maybe first about the long context, because that was also the first one that really had this real 1 million token long context that is actually functional. And you guys had also the MiniMax Sparse Attention, which is this one technique to make that efficient that you also published and share extensively. So can you talk a little bit about this? Maybe how the project went from the attention mechanism to how to make this long context?

Olive Song: Yeah, I would say the story about long context went back to even MiniMax M1 and MiniMax 01, where the model was actually able to perform tasks of 10 million token context.

Thomas Wolf: 10 million?

Olive Song: 10 million, yes. But then it was not an agentic model, right? It was just, for example, you dump in a book, it would be able to give reviews on it and stuff like that.

So what we realized was that longer context actually unlocks a lot of capabilities, especially when interacting with users and now when the agent is interacting with the whole environment and getting all the tool responses, getting multi rounds—the shorter context wouldn't be enough to perform the complex task. So for this version we said, "Oh, we have to have our longer context back."

So what we pursued was with our MiniMax Sparse Attention, which you know, was the architecture that was scalable and had a simple design. So I would say from a higher level, right? It has an index branch that selects on a higher level what matters more in the context, and then we have a sparse attention branch that performs the calculation on the selected blocks to actually perform the task.

And so yeah, like that we really designed an elegant architecture so that we can scale the length and then scale the model size in the future with that.

Thomas Wolf: That's beautiful. I like how for those who've been in the field for quite some time, we had a lot of work on attention, right? This N square and there was a lot of linear attention. And then somehow all of these disappeared at some point when FlashAttention came around. We discovered we just needed more efficient compute kernels.

Now, I like how we come back to thinking, you know, first principle: what is attention? How can we make that more efficient? So 1 million token is crazy, right? GPT-2 was 1,024 and everyone was like, "Oh, that's really big. We never need more."

Where do you see this going in the future? Like, Jeff Dean was pitching me the other day a trillion-token attention. You think we should go to a trillion-token attention?

Olive Song: So, that's definitely something we can explore towards, right? Ultra lengths of the context. Definitely, it's something that's very exciting to explore with. And something that architecture design along with hardware would require a lot of research onto that. Yeah.

Thomas Wolf: You think there's still a lot of low-hanging fruits? So, typically today we saw OpenAI really reducing—I mean, we don't know how as a firm, but reducing their inference bill by half, by probably having some more efficient processing around attentions of any type. You think there's still a lot of low-hanging fruit that can be gotten in how we can process that?

So one thing we're still very interested about M3 is how cheap it is, in particular because of this sparse attention and partly because of its small activated parameters, but it's also very efficient. You think we can go even way further?

And maybe how did you guys invent MiniMax Sparse Attention? Was it an agent coming up with the idea? Was it a human still coming up with the idea? Tell us a little bit about that.

Olive Song: Yeah. Um, so we do think there's still a lot of work that can get into architecture and inference optimization so that the model can be more efficient, especially if there are tasks that are very latency-sensitive but require very strong capabilities, right? And for those kind of tasks we really want the model to be efficient.

Um, and who came up with this part? Actually I think an intern from our team worked on that. Yeah, an intern. That doesn't usually happen in a lot of labs because I think in some labs interns don't have access to the data, the code, and stuff. But yeah, we are open to anyone who would like to contribute to our models. So, the architecture was actually designed by an intern.

Thomas Wolf: It's very good. Still some work for interns here. Good news. Um that's also a good segue to also how MiniMax is working internally. So we were discussing before coming on stage, I was saying everyone can propose a project. Can you tell us a little bit about how you are organized, how you do your research?

Olive Song: Mhm. Mhm. I think that is very different from school or even in earlier tech companies, it is pretty different.

It's that what we make sure is that we have good foundation and good infrastructure so that anyone can play with the model and can think of what they can improve with the model. And then after model releases when they are free, right? They can play with the model. They can think of their own evaluations. They can find their own weaknesses and propose a thing that they want to improve on the model.

And then other people who are interested in that would, you know, propose to join the project and they will work on it for a couple of weeks or even a couple of months. And when they work out, the final thing is shipped to our model. We use that in our final training and it's shipped out to the audience.

Thomas Wolf: Interesting. So, you can have people working for a really long time on a project. When you say a couple of months, it can be like really deep exploration.

Olive Song: Yes. Yes. I would say, for example, architecture might require longer time of investigation, research, experiments, even redoing the evaluations for pre-training. Yes, so it might require longer time.

Thomas Wolf: Very nice, yeah. And I know you're also very big on evaluation. I agree. We could talk about that.

I think one thing probably related to that is this unique specificity that M3 and your team has around multimodality. So, not just text, but the model can also understand image and video. And as I understand—but please explain better—when you read the model card on Hugging Face, it says the model was trained from the first step as a multimodal, not just have a like adapter after thought, right?

Can you tell us a little bit more about that and why you think it's important, and why starting from the first step on multimodal training and not just training text first?

Olive Song: Um so, we call it native multimodality. And so, it is somehow typical for model labs to train the multimodal, let's say, vision understanding capabilities after the text pre-training is done. They put adapters and then train that part.

But what we found out was that that would actually harm the text performance. And the vision understanding performance wouldn't converge that well because the model is kind of converged towards the text understanding. Um and it's just not the most optimal. And also not the most scalable, if you think about it. We want to scale the data, right?

And also, some labs train this capability from halfway through the pre-training. For example, continued pre-training. But what we found that this would be very, you know, recipe sensitive. It is different for—the recipe would be different for different architectures, different, you know, data mixtures, different learning rates. It's hard to control, hard to, you know, scale your experiment results and conclusions to a larger model.

And so you know, what we thought was why not just training from the very first step? That comes to the most natural. We know that a lot of labs run into problems doing that. The model would collapse after a couple of steps of training, you know, both text and vision understanding, but we managed to solve that problem.

We did a lot of work on ViT and we did a lot of work on the data that we actually training. For example, we do interleave the data, what we call interleave the data. It's actually natural data, but we keep the images and videos in instead of masking it out and we do some pretty good cleaning and masking on the data and we do very good reward modeling so that we train it from the first step and it scales up a lot. Yeah, it does not collapse.

Thomas Wolf: That's really impressive. Should we expect much larger model in the future? So this one is still fairly small, right?

Olive Song: It's 428 billion parameters, 23 active billion.

Thomas Wolf: Well, do you think you will go past the trillion?

Olive Song: Definitely. Yeah, definitely in the future. There are many tasks that wouldn't be able to—the smaller parameter model wouldn't be able to perform very good at. We are definitely going more ambitious than this.

Thomas Wolf: It's great. Looking forward. Um another interesting thing I always find fascinating about MiniMax is how you also have this whole range of apps and product, right?

So, I remember already—so MiniMax started to open source things on the Hugging Face platform in January last year. So that's 18 months ago, and we were chatting a little bit about the team to understand what you were doing, and I remember you were already having a huge usage on some of these apps.

Can you tell us a little bit how this started, right? Was it basically you had a lot of apps and then you thought "we have all this data, why not training a model" and then build up research team? How is the story there?

Olive Song: Um our story is model from the first day. So, I believe that a multimodal model that can understand all visions and outputs all modalities was the first thing that our CEO planned on the first day even before the company even started. So, that was the dream of AGI. I think that was very early, even before ChatGPT came out.

And then apps were something that comes along because you have some model capabilities, you want people to experience it well. Not many people can use it with API, right? We can't expect everyone to experience with API. So, we need good user interaction interfaces, good apps, good scenarios that people can experience model with.

I think actually those apps covered more than 300 million people around 200 countries globally. And I think over a million companies as well.

Thomas Wolf: Yeah, this was mind-blowing when I heard about the size, and we don't often realize the size of this type of usage already.

And that kind of brings me to the question around open source business model and all of that, which is the always existing question, which is right now it's nice to open source model, but you also need to have some revenue stream, right? So, I guess M3 is something you decided, for instance, to be for free, and I think it's great for the world.

How do you see this? Do you also have some specific models you use for the app? Do you think in the future you'll keep open sourcing models? How is the culture around open sourcing right now?

Olive Song: Personally, and also for the model research team, we always hope to open source the models. That is our plan.

Because we really see how the open source community together can help the model build better. For example, we receive a lot of feedbacks on the model performance from the great community, and we receive PRs on whatever we open source. And those are very, very valuable and comes to our later versions. So, definitely open sourcing is great.

Thomas Wolf: That's great. And actually, do you have some ask for the audience, people who are using M3 or MiniMax? Is there something you would love them to send back to you as feedback? Do you read when people try to modify the models or play around, you know, tweaks? Or what is the best thing you think you can take from the community for future models, for instance?

Olive Song: Mhm. I would say whatever issues that people are running into, especially with multimodality, right? This is the first time that we're combining it together. We are definitely going more ambitious on that in the future. It might have some flaws right now, but we are improving on that. So, whatever that's feedback that model is not doing that great, we will definitely improve that in future versions.

And also, whatever features that people want. Say, you know, for example, thinking effort. Right? Some people ask for that. Like everyone can ask, and we will try to accomplish that in the future models.

Thomas Wolf: Yeah. Do you see a lot of usage right now already in multimodality in terms of coding agents? I feel like it's a little bit underexplored.

Olive Song: It is, but it can actually unlocks a lot of capabilities and a lot of agent applications.

Say that for example, you want a model to read a PowerPoint, or to read some report that is not very structured. And you want it to understand a very long video—say that you dump in a long-playing video, and then you want the model to act using some tools after understanding it. And it unlocks a wide variety of agent use cases.

Thomas Wolf: So, like the agent could finally watch my YouTube tutorial and understand how to use my coding tools, how I described it? Is it something like that? Could the agent finally watch YouTube tutorials and understand things from them?

Olive Song: Yeah, yeah, yeah. I think so.

Thomas Wolf: Do you use a lot of agent coding tools internally? Is it like—I mean, coding for sure, but like is it also already in terms of research? Is it automated part or not? How does this work?

Olive Song: Yes. We have our own research harnesses. We build our own research harnesses that automate our workflows. I would say a lot of our workflows are automated.

You can see how the latest frontier models all pursues capability like kernel optimization, right? Like let the model post-train other models, let the model build data, auto data, stuff like that. You can see how more and more models are capable of doing those, including M3.

Actually, we were very good at those cases—longer horizons and kernel optimizations. And so we can use that model capability, harness it together, and help with our daily routine, and make our iterations even faster.

Thomas Wolf: Is M3 building M4 already?

Olive Song: Um building M3.1.

Thomas Wolf: M3.1, okay. [laughter] Let's hit the gym.

Olive Song: Already.

Thomas Wolf: Um I would love to finish on what you find exciting in the coming month. What do you think it can be in terms of feature or things you want to see happening in AI, or more generally in terms of whatever really is top of your mind and it's going to happen?

Olive Song: A lot of things are very exciting. But what I recently find the most exciting would be a multi-agents that I think a lot of AI applications are using, model routing, multi-agents that allows even more capabilities, even more complex tasks, and also it tells us what the models are capable and not capable of, and you can, you know, do a lot of things with that. It's pretty exciting.

Thomas Wolf: Thanks a lot, Olive. Pleasure to have you.

Olive Song: Thanks for having me.

Thomas Wolf: Thanks, everyone.