Hi everyone, my name is Cassidy and I'm a researcher at Google DeepMind. Today I'm really excited to share with you some of the technical improvements and architecture that we have with Gemma 4. Last week we launched Gemma 4 which is the latest addition to our family of open source models. Gemma 4 brought incredible improvements at a scale that has not been seen before. We have a family of very small models with incredible performance setting a new precedent for what's possible with small open source models.

Gemma 4 comes in four sizes. We have two smaller effective models which are geared towards ondevice applications. These models have been adapted and improved in order to provide incredible performance at a small scale which are able to run locally on phones, iPads and laptops. We have two larger models starting with a 26B mixture of experts model which is the first ever Gemma. This model has been adapted to have incredible performance while only requiring 3.9 billion active parameters. And our largest model is our 31B dense. This has insane performance, a huge improvement upon what existed within Gemma 3 at a new precedent that hasn't been seen before.

Taking a look at our larger models, our 31B and our 26B, these models have both ranked in the top six of all open-source models on the LM, the LM arena. One of the most exciting improvements and things that we've launched alongside Demo 4 is the move to an Apache 2.0 license. This was deliberately done in order to make our models more accessible for the everyday developer. You should easily be able to integrate Gemma into your life cycle of development through initial testing all the way to deployment and building within the Gemma universe.

Now, let's take a little look at what each of these models are and some of the use cases that we've adapted this for. Starting with our 31D dense model. This is a state-of-the-art multimodal model which has been purposely built for advanced reasoning. This model ranked number three on the global arena for the AI leaderboard. This is outperforming models over 20 times its size. This is a huge improvement. The 31B has a 256k context length which has been purpose-built for autonomous workflows with native support for thinking, function calling, and structured JSON outputs.

We also have a slightly smaller 26B. This 26B is the first addition of a mixture of experts model into the Gemma family, only requiring 3.8 billion parameters during any forward pass. This model is small and efficient, utilizing a total of 128 experts while only requiring eight experts during any inference. This is efficient for running while still maintaining some of the incredible performance that we saw with our 31B.

Now, let's take a look at what's actually new in Gemma 4 and what have we done and how have we actually been able to achieve this incredible performance. Starting on the architecture side, we have our standard dense model. This is our 31B as well as our smaller effective 2B and 4B models. We have our standard decoder block. What we've done with Gemma 4 is we've made several improvements within attention. We've introduced a 5:1 ratio of interle to global layers with our smaller effective 2B having a 4:1 ratio. This means that within our local layers, we have a sliding window of how many tokens we're attending to. And lastly, with our global layers, we've now ensured that the last layer is always a global layer, meaning that our last layer is attending to all preceding tokens.

In practice, what this looks like is our global layers are attending to every token that is preceded within this, whereas our local models are only attending to a specific number of preceding tokens. In our smaller models, we have a sliding window of 512 tokens, while in our larger models, we have a sliding window of 1,024 tokens. This sliding window has provided significant improvements in the efficiency and optimizations of our local layers while still maintaining passing through information to the preceding layers.

However, our global layers remain to be quite expensive. Despite this interleing of local and global layers, all of our global layers are still required to attend to all preceding tokens, which makes it quite memory intensive and expensive to run. And this is where we've looked into introducing grouped query attention. Within our local layers, we group together two queries to share the same key and value heads. However, in our global layers, we're grouping together eight queries sharing the same key and value heads. Since reducing the number of key and value heads can have a big impact on performance, we've doubled the length of the key value heads within our global layers to have a length of 512 as opposed to 256 as used within the local layers.

These attention changes were present across all of our models. But we also introduced a new architecture with Gemma 4 and this is our MoE. Our MoE has one shared router expert with a total of 128 total experts with eight activated experts on each forward pass. All of these experts are small fed forward neural networks. In practice, what this looks like is having our constant shared expert which is activated on every pass of the model. This shared expert is three times the size of our regular experts. We then have 128 experts each of which eight of which are selected by the router during any pass of the model.

And lastly on an architecture side we get into our dense effective models. So what does it mean when we're saying our model is effectively 2B, effectively 4B? This is where we're looking at the difference in the number of parameters which are required to operate the model as opposed to the total number of representational parameters present. Our 2B is effectively 2.3 billion parameters while having a representational depth of 5.1 billion parameters. These models have been specifically designated and optimized in order to have the best ondevice performance. These are designed to run on phones, run on laptops without requiring expensive API calls to models served somewhere else in the world.

These advancements were made possible through PLE. This is our per layer embeddings where within each of our layers, we now have a dedicated embedding table. Before digging into exactly how our per layer layer embedding table looks, let's take a look at how the entire token embedding layer works. This hasn't been replaced within our effective models. We still have your standard embedding table where you're looking at the mapping of an ID for a token in toward towards its embedding vector. In our E2B, we have an embedding vector size of 1536. And in our larger E4B, we have an embedding vector size of 2560.

Now we also have a per layer embedding table. This is where similarly we have our entire vocabulary size and have an embedding representation for each of these tokens. But now we also have one of these for each of the layers. The big advancement that comes here is the fact that we store our per layer embedding table in flash memory as opposed to VRAM. VRAM is one of the largest constraints on ondevice. where you quickly run out of memory in phones and laptops. So by requiring that we no longer need to store this additional embedding table in VRAM and we can can store it in flash memory, we're able to get incredible improvements on the inference side without having an expensive cost of this additional storage and memory.

The big difference in our PLE embedding table as opposed to the standard embedding table is our embedding dimension now is only 256. So this is significantly reduced from the size of the full model. For each token, we now have an embedding for this token at each separate layer within the model. And as you progress through the layers of the models (35 for the 2B and 42 for our larger model), you'll see the progression and improvements in the embedding representation for each of these tokens at the next subsequential layer. Ultimately, these improvements allow for our E2B and our E4B to be significantly outperforming prior generations of Gemma small models.

In addition to everything that we've done on an architecture side, we've also really set a new frontier for what's possible with multimodality. In Gemma 3, we introduced vision for the first time. In the Gemma 3N model, we launched open-sourced audio vision and text model for the first time. This really paved the way for Gemma 4 being natively multimodal models. Starting on the vision side, our 31B and 26B model both use a 550 million parameter vision encoder. Our effective 2B and effective 4B have a smaller, compact, more designed encoder of 150 million parameters.

We've made big strides from Gemma 3 by the introduction of variable aspect ratios and variable resolutions. What this means in practice is you now have a choice to select the resolution and the soft token budget that you want to allocate for images. This is available in five different resolutions. What's important to understand here is this is a huge improvement from Gemma 3. In Gemma 3, we had to introduce pan and scan. And this is where you would give an image and we split it into a sequence of squares and padded whatever else was necessary. Now, we're now able to process images with varying number of patches depending upon the actual image which was provided by the user.

For the audio side, audio has been added into the E2B and the E4B with the goal of being able to support translation and speech recognition. This is made possible through the combination of an audio tokenizer and a conformer. This is a 35 million parameter conformer which processes as an audio encoder processing embeddings rather than tokens. On the audio tokenizer side, we start with raw audio. This raw audio is run through a MEL spectrogram which is able to process features out of the raw audio files. This MEL spectrogram is then split into NMEL chunks which are downsampled. This ultimately outputs soft tokens. These audio embeddings are what's passed forward into the conformer.