MiniMax M3 架构解密：稀疏注意力机制与原生多模态端到端训练

um all right, um so hi everyone. i'm my name is olive. i work at minimax on a reinforcement learning research. and today, i'm gona talk k little bit it about our new model minimax m three and how we trarended nativemulti model from step zero and how we trarended with MSA, which is called ininimax spar attension. um a little bit introduction of our team and how one of the few labs over the world that uh works on multime modealities, which means that we have published our um of course, LM dedels. before we worwork on our speech models, we have our own video generation models, and we have our coding agenca models. and lately we just release our minimax m three, which is open weight with frontier performance. so it is actually the first open weight model to combine the following three uh, from chair capabilities. the first one is what we call. so the uh coding and ergentic capabilities, the model is able to do a lot of the agentic task that you imagine. um it can do automatic a text composition. it can work in your coding agents. it can drop in for ninety percent of the coding works that you are currently doing. and the second pillar is that the model has one million context window. um so which is very long that enabled like unlocks pretty much basic all basic user long user interactions um of like it handles four coo bases, and we trained ded with MSA minimum sparse attention. so it is nine times faster prefiil and fifteen times faster decode versus four tension. and the third one is that the the model was trained native mutor model from step zero, which means that from from scratch, we combined visionary standing with language generation. so the model would understand both capabiliis very well from the start and uh h. so that we can build many more applications on that, say, computer use agents, right?

so first, all briefly explain what sparse attention does. our court design principal is that a simpler architecture can always lead to better scaling properties. this is a principle we have followed throughout the entire design. so our goal is to always pursue a cleaner and simpler architecture. disparse attention consists of two branches. the first one is what we call the index branch, which performs a lightwid full attention computation. it uses four quarheads and one key head, and it is trained with a care loss to match the the attention scores of the main branch. the index branch is used to identify the tokens with the highest attention scores. these selected tokens are then attended by by the main branch. in other words, the sparse branch does not attend to all of the tokens, but only to a subset of them. this subset is selected through the lightwway computation of the index branch. then the sparse branch would use the blocks selected by the index branch to determine which keys and values are needed for attention complication. all right.

and what i want to emphasize today that our design is not purely algorithm like algorithm driven and said, we took an algorithm algorithm and infrastructure code design approach. um a good good example to start with this dsa. d. sa was a very strong design, but it is also high to optimize for deep six oones architecture, especially the comonination of MQA and the large headdeam. however, what we move to a more commonly used GQA architecture directly applying the same design, becomes a lustury forward. it introduces several practical issues is especially from the infrastructure and implementation perspective. so this motivate our design choice, instead of simply borrowing and existence, spar se attion design, we rethink the architecture under the constraints of GQA and real al system efficiency. so what are the issues? um if we directly apply the existing sparse? attenararchitecture first from algorithm perspective, GKVA has multiple KV has which should ideproviprovide diverversity across different KV groups. however, if we directly reuse the same token selection strategy, multiple KV has will end up attending to exactly the same selected tokens. they essentially cancils out the diversity that jicuates is supposed to provide second from the infrastructure perspective. this design is not very friendly to GPU memory access GPU are much more efficient when reading continuous data in the MQA setting. we can syanthy KV eight layouts as one KV had with a large head dimension, for example, one by five hundred. and pulth, this is relatively continuuous and efficient to read. however, in g ua, the same total dimension may be sput acpross multiple KV hats, for example, is four by one twenty eight. these are no longer one single continuguous read. instead, the GPU needs to perform multiple independent readacross different KVVVS. this increases memory access overhead and reduces the computer memory access stratio. and another issue here is the top key computition to south to select the most important tokens. the indexture needs to compute. talk k. sorts over a large number of token webl candidates. however, talk k has inherent data dependencies, so it is difficult to fully be parallel on gpu. as a result, total level selection introduces nontribual competition emerging overhead. so how that we address these issues in m sa from the from the algorithm side, we remove the aggregation of the index branch outputs in the original design. the results from all index branch in index branch has emerged into a single set of selected tokens. but as we discuss earlier, this would remove the diversity brought by jecua. instead, we keep the selection different. the different across KV has under our sparse attention design when using GQA different KV has can attend to do different sets of tokens. this helps preserve the diversity of GQA rather than forcing all hets to share the same sports tension for the infrastructure side. we move from token level selection to block level retrival. compared with token level selection, brock block level retrieval significantly reduces the number of candidates and lower sea overhead of top key computation and result merging at the same time, block level retrival is more hardtware friendly. it improves memory access efficiency and increases the compute to memory accestrry shell. importantly, we also show that the design does not introduce a significant performance drap finally would design efficient kernels for both training and influence. i won't going to the kernel details here today, but this is the system level component that makes the sparse attention design, practical and efficient. so with this design, we were able to extend the training of our m free model to one million context very ficiently.

and as we talked about earlier, the third pillars is the native multimobality. our model minimax m three was trained from scratch with multimodel data. what does that mean? our model was string with both text and vision understand standing from steps zero is very different approach from current open weight models. our starting point is very clear as a current stage, a standalone VL model has relatively limited application scenaris. but combining it was with, you know, text understanding during the training can be challenging. so the key question and challenged there was, how can we introduce multimodel capability without hurting the cortex performance? to answer this, we actually explore several training strategies as scale. the first and most common approach is to add mottimodel training after the pretrain in decay stage, which is essentially ACPT stall approach. however, at that point, the model has already largely conversed, adding more time model data at this stage tends to interfere with a learch tax capability and causous degregation. another approach is to insure multimodel training before pretraining decaded ge, for example, after the model has already seen a number of amount of tex tokens. we also tried this direction but found that the results were highly sensitive to the dynamisure in training hypoper proramners, such as learning rate, for example, in one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run. more importantly, during a large scale, pray training. these parameters can not be ararbitarily changed just for mulormodel training, and this leled us to our final strategy. native mulormodel training from scratch in this setting. multi model capability is introduced from the very beginning rather than being added after the text model has already formed its structure under the same text token budget. we find that the disaproproach almost does not heard tax capability. we actually did many fundd. analyyis is aring this experiments. one of them missed this interesting observation from the attention maps. you can see from this picture that for CPT style multime model models, the text tokens often pay very little attention to visual tokens. the visual token regions are mostly dark in the attention map. this suggest that the multi modelities are not fully integrated, while in contrast, with native multimodel training, the attention map looks much more integrated. we do not see here a clear separation between tax tokens and visual tokens. they suggest that the model learns to fuse visual and textual information more naturally from the beginning. so overall, our training strategy is not just about adding vision capability. it is about integrating multimodelity into model while preserving its strongest text capabilities.

and of course, uh, we did many experiments on multimodeararchiteture ture choices. our all arartetetures relatively standard, the image or video is being processed by a viit, which produces visual tokens. these visual tokens are then inserted into the language model input by replacing the original image placeholders in the take sequence. after that, the visual embeddings and text ebeddings are fed together into the language model. one important finding is that applying three, the attention inside the vip is very helpful for vision understanding, and the standard viit setting attention is is applied within each individual image or frame for video inputs. this means that visual patches from different frames do not directly interacts ts at the IT stage, but with three d attention patches across different frames can attend to each other directly. this gives the model stronger temporal and cross from visual model and capability, and we found that this improves visual understanding and the benefit also transfers to some image benchmarks, suggesting that it's strength the overall visual representation, rather than only improving video specific text. and one practical detail is that we cannot simply extend three d attention to an unlimited number of frames because that would introduce additional system and low balancing chengenges in explplation using sing four frames, give us the best trade off between performance and efficiency. so overall, with our architecture design with our training strategies that would export uh throughout the stages. and with our analysis and VIT explorations, we were able to train the model from scratch uh, with more timodality understanding, and that will be the end of the presentation today. thank you. the.