Host: Hello everyone, this is Best Partners. Today, we're going to talk about the story behind Google's Gemini 3 model and share deep insights from an industry titan. He is the head of pre-training for the Gemini 3 project, Sebastian Burzo. As a top global AI researcher and a member of the Metis list, this is his first podcast appearance. In our conversation, he not only reveals the underlying logic of Gemini 3's architecture but also presents several hardcore viewpoints that could shake industry perceptions. From model development to team management, from industry trends to future outlooks, every topic is profound and inspiring. So, let's dive in today.

Host: First, we must discuss the secret to Gemini 3's success. Many might assume that a model achieving such a generational leap must have a groundbreaking core technological breakthrough behind it. However, Burzo offers an answer that seems simple yet profound. He mentioned that Aurel Viny, co-lead of Gemini, stated at the Gemini 3 launch that the model's secret to success is actually higher quality pre-training and post-training, which is the result of immense team effort.

Host: Contrary to the common perception that genius inspiration drives innovation, Gemini 3's progress didn't stem from one or two pivotal breakthroughs. Instead, it was the result of over two hundred people making countless pixel-level micro-improvements across multiple dimensions, including models, data, infrastructure, and evaluation. The synergy of these improvements and innovations ultimately led to the model's massive leap forward.

Host: Burzo, as one of the heads of pre-training, is responsible not only for enhancing model performance but also for complex coordination and integration tasks. The pre-training team is quite large, with nearly two hundred people working daily on pre-training related areas, covering aspects like data filtering, optimization of model architecture, infrastructure setup, and refinement of the evaluation system. Therefore, coordinating the work of so many individuals, ensuring everyone can contribute effectively and make progress, is a highly complex and time-consuming endeavor.

Sebastian Burzo: But I think it's very important. Because in the long run, the key to success lies in integrating the work of many people, rather than relying on the leadership of a few. In the short term, outstanding contributions from a few individuals might yield some results, but only by forming a collective force can the entire team achieve true generational breakthroughs. This is precisely one of Gemini 3's core success logics: democratizing excellence through extreme collectivism. Here, 'democratizing' doesn't mean ordinary capability; rather, it emphasizes the accumulation of every small improvement, where the contributions of each team member ultimately converge into an unstoppable force of innovation.

Host: Burzo mentioned that the industry is transitioning from a data-infinite era to a data-finite paradigm. This shift has significantly altered many research directions and our ways of thinking. In the past, many believed that continuously increasing compute power and data scale would lead to perpetual improvements in model performance. However, the situation has changed. While the scaling laws haven't reached their limit, scale remains a crucial aspect of pre-training, capable of improving model performance in a relatively predictable manner. Yet, people might have previously overestimated the role of scale; it is not the sole factor.

Host: Today, the importance of architectural innovation and data innovation even surpasses mere scale expansion. The real key to success lies in how to extract more intelligence from a limited data pool through architectural optimization. This has been demonstrated in DeepMind's practices. Burzo and his team conducted the Chinchilla project, which re-examined how to adjust model and data scales under fixed training compute resources to train the best-performing model. They discovered that, compared to previous understanding, data scale should expand faster, rather than solely increasing model size. This finding has significant implications for post-training deployment and usage costs.

Host: Another related project is Retro, which focuses more on architectural innovation, exploring how to enhance model performance by enabling it to retrieve information from large text corpora. In other words, it doesn't require the model to store all knowledge within its parameters; instead, it can look up specific information during training and inference. This approach remains highly relevant and practical today.

Host: In this industry context, research taste becomes a critical factor determining a model's fate, a point Burzo strongly emphasizes. He believes that research taste is difficult to quantify but has several key elements. Firstly, research cannot be isolated; it must be able to cooperate and integrate with others' work. For example, if a model improvement boosts a single metric but increases the difficulty for others to use the model by five percent, it's likely not a good trade-off, as it would slow down the entire team's research progress and consequently impact overall advancement.

Sebastian Burzo: Secondly, one must be wary of complexity. The complexity that research can tolerate is limited, and research risks need to be controlled. Therefore, teams often abandon solutions that are optimally performant but excessively complex, making some concessions on performance to choose less complex versions, as this facilitates future progress. Furthermore, research taste includes intuition: how to judge, under limited computational resources, which research directions are feasible and which are not. After all, most research ideas do not succeed. Researchers need to determine how much effort to invest in a direction before pivoting or persisting. In deep learning, a negative result doesn't mean a method is unworkable; it often means the way to make it work hasn't been found yet. Recognizing this is crucial.

Sebastian Burzo: During the extremely expensive pre-training phase, the hardest part isn't deciding what to do, but what not to do. This wisdom of trade-offs is the core manifestation of research taste.

Host: In the research process, there's another important balance: the trade-off between short-term and long-term goals. Burzo states that there are always critical path items that need completion, such as improving a specific part of the model or identifying an under-optimized component. Significant effort is then dedicated to addressing these immediate issues. This is done partly because these improvements are certain to enhance model performance, making them relatively safe bets. On the other hand, seemingly imperfect parts can lead to more problems as the model scales up or its capabilities grow in the future. Therefore, addressing these issues diligently is crucial.

Sebastian Burzo: On the other hand, there is more exploratory research. These ideas might be applied in the next version, or the version after that, of Gemini. They could have a greater impact on model performance but are not yet fully validated. This balance doesn't have a fixed answer and is somewhat cyclical. For instance, during the model scaling phase, exploratory research tends to be more prevalent as there are fewer urgent issues to address concurrently. However, leading up to the release of a new architecture or model, the focus shifts to risk reduction, concentrating more on execution.

Host: It's worth noting that at DeepMind, the tension between research and product is relatively low because all leadership has a research background. They understand clearly that while specific benchmarks and goals can be accelerated to some extent, ultimately, the progress of research work is paramount. This allows research teams to focus more on creating long-term value.

Host: Speaking of teams, DeepMind's organizational structure also provides crucial support for smooth research progression. At the highest level, there are pre-training and post-training teams. Within the pre-training team, there are specialists responsible for models, data, infrastructure, and evaluation. Burzo considers evaluation a critical but often underestimated component. Doing evaluation well is indeed very difficult. Additionally, large teams are dedicated to infrastructure and deployment. Close collaboration among these teams forms an efficient R&D system. This architectural design ensures deep specialization in professional domains while enabling cross-disciplinary synergy, providing robust support for continuous model iteration.

Host: From a macro architectural perspective, Gemini 3 is a Transformer-based Mixture-of-Experts architecture. Compared to its predecessor, the architecture hasn't undergone drastic changes. However, a key feature of Gemini 3 is native multimodality. This means there aren't separate models for processing images, audio, or text; instead, a single neural network handles all these different modalities simultaneously. While this design offers significant performance advantages, it also involves cost considerations.

Sebastian Burzo: The costs of native multimodality are primarily reflected in two aspects: first, complexity and research costs, as it involves handling more tasks, especially the interactions between different modalities, which impacts multiple research facets and increases complexity, thus requiring more time for thought and study. Second, computational costs. Compared to pure text, image inputs are typically larger. If processed in a simple, direct manner, the actual computational cost would be higher. However, related research can improve the efficiency of these processes, and the benefits largely outweigh the costs. This is why DeepMind persists in training such models.

Host: Regarding pre-training data, Gemini 3's pre-training data is a mix of various sources, inherently multimodal, containing information from many different modalities. A widely discussed issue in the industry is data scarcity. People often debate whether we will face a shortage of data in the future. Burzo believes this won't happen. While the industry is shifting from a data-infinite to a data-finite paradigm, it doesn't mean the amount of data is decreasing; rather, data is finite. This paradigm shift drives adjustments in research directions. Before the advent of large language models, many conducted research based on benchmarks like ImageNet, leading to technologies suitable for that stage. Today, new innovations will similarly emerge under the data-finite paradigm.

Sebastian Burzo: Synthetic data is also a hot topic in the industry right now. I think synthetic data is an interesting area, but it must be used with extreme caution, as it's easy to misuse. Typically, people use a powerful model to generate synthetic data and then validate its effectiveness through small-scale ablation studies. However, a key question here is: can synthetic data be generated to train a future model, and can this new model's performance surpass the original model that generated the synthetic data? DeepMind has invested significant time in contemplating and researching this.

Host: Regarding future directions in pre-training, Burzo highlights the capability of long context. He believes this is a highly promising area. In Gemini 1.5, DeepMind achieved a significant leap in long-context capabilities, enabling current models and Agents to handle large tasks like codebases. He predicts that in the next one to two years, there will be more innovations in this area, not only improving the efficiency of long-context processing but also further extending the model's context length.

Host: Furthermore, attention mechanisms are another important research direction. Recent interesting findings will shape many research avenues in the coming months. Another aspect, often overlooked but critically important, is evaluation. Burzo emphasizes that evaluation is very difficult in pre-training because it needs to bridge two gaps. Firstly, models used for daily training and evaluation are typically smaller and less performant than the final scaled-up models. Therefore, evaluation methods must be able to predict the performance of large models and still guide them in the right direction; in other words, it must be a good indicator.

Sebastian Burzo: On the other hand, there's the post-training gap. Models are not directly deployed after pre-training; they undergo further training. Therefore, evaluations conducted during the pre-training phase or on pre-trained models must accurately reflect their performance in subsequent training. Consequently, advancements in evaluation significantly drive progress in model and data improvements, as they allow researchers to accurately measure the actual impact of model or data enhancements.

Host: To ensure evaluation accuracy, DeepMind's evaluation system is largely built internally, and this trend is growing. While external benchmarks can be used in the short term, they quickly become contaminated. The content of these benchmarks spreads online in various forms. If training data includes this content, it's difficult to detect. Therefore, to avoid self-deception and truly understand a model's actual performance, the only way is to create independent evaluation sets and keep them strictly confidential.

Host: Regarding the issue of alignment, Burzo states that most alignment work is done during the post-training phase, though some related work occurs during pre-training. When asked if the primary principle of alignment, given core datasets come from the internet with much harmful information, is to exclude this harmful information from model training, he offered a thought-provoking answer.

Sebastian Burzo: He said that models need to understand harmful information to know how to avoid it. Therefore, it's at least necessary for the model to be exposed to a portion of this information so it can identify harmful content and avoid generating related outputs. Otherwise, when users mention certain harmful information, the model might not know what the user is talking about and thus cannot identify it as harmful.

Host: During the conversation, the host also mentioned DeepThink, a reasoning model launched a few days after Gemini 3's release. Although Burzo didn't reveal many specifics, he explained how these reasoning models work: unlike computations performed solely within the model, these models compute at the sequence length level, giving the model more room for thought. The model starts by forming hypotheses, testing them, calling tools to verify assumptions, conducting searches, and so on, finally reviewing the entire thought process to provide a clear answer to the user.

Host: As for Google's Gravity project, Burzo believes Agents can have the greatest impact in execution-level tasks, such as monitoring experiment progress. Visual perception is also crucial for Agents, as we are now beginning to require models to interact with computer screens. Therefore, excellent screen comprehension ability is very important. This is also a significant aspect of the pre-training phase.

Host: Furthermore, continuous learning is another important research direction. Continuous learning essentially means constantly updating models with new knowledge as it is discovered. For example, if a new scientific breakthrough occurs tomorrow, and the base model trained yesterday is unaware of it, how to enable the model to quickly absorb this new knowledge is the problem continuous learning aims to solve.

Sebastian Burzo: In this regard, I believe significant progress has been made over the past few years, primarily in post-training and search. By making search calls using search tools, models can acquire new information. This is precisely what the Retro project does: by retrieving data, it attempts to decouple the knowledge corpus from the reasoning part. On the other hand, continuous learning is also related to long-context capabilities. One way to achieve this is by continuously expanding user context, allowing the model to obtain more information within the context, thereby enabling continuous learning. However, this might require a larger paradigm shift, such as changing training algorithms to allow models to continuously learn from real-world data streams.

Host: When asked if there is over-investment in the AI field, Burzo stated that the situation is much better now. About two years ago, many were trying to create specialized models for tasks that general models could solve within six months to a year. But now, people are increasingly realizing that for general tasks or tasks not requiring extremely specialized models, general models might suffice. This means research into how to use models and how to build model application frameworks is becoming increasingly important. Concurrently, improving the robustness of models and these application frameworks, enabling them to reduce errors and recover from them, is also a significant research direction.

Host: During the interview, we can also learn about Burzo's personal growth trajectory. The experiences of this top AI researcher are quite inspiring. His background is quite diverse; he grew up in various parts of Europe and moved frequently. Born in the Netherlands, he moved to Switzerland at age seven. His father is Swiss, and his mother is German. He completed most of his schooling and early high school in Switzerland, primarily using French with some German courses. At fifteen, he moved to Italy, where he finished high school until around nineteen years old.

Host: As a child, Burzo developed a strong interest in programming. His father had a technical background, so Burzo started learning programming with his father at the age of ten or eleven and has always enjoyed this skill. In school, he also excelled in mathematics and science. He consistently achieved good grades in math exams without needing to study specifically. He originally planned to pursue further studies at ETH Zurich, but he happened to see a university ranking that placed Cambridge University at the top, so he decided to apply. To his surprise, he received an acceptance letter a few months later. He subsequently completed his undergraduate and master's degrees at the Cambridge University Computer Laboratory.

Host: Joining DeepMind after graduation was a fortunate opportunity for Burzo. A lecturer during his master's program, who was also a researcher at DeepMind, was approached by Burzo at the end of the last class to request a recommendation. The lecturer readily agreed, helping him secure an interview opportunity at DeepMind. In 2018, when DeepMind had not yet merged with Google, he joined the company as a Research Engineer. Initially, he worked on projects related to reinforcement learning, training unsupervised networks to learn key points in Atari game environments and attempting to have Agents play Atari games. However, he disliked the synthetic nature of it and always wanted to work with real-world data on tasks that could have a practical impact.

Host: He then shifted towards representation learning, training neural networks capable of good representation for various tasks. His first related project was named 'Representation Learning with Real-World Data.' The qualifier 'with Real-World Data' was added to the project name at the time because people defaulted to assuming projects used synthetic environments or data. Since then, the situation has completely changed: real-world data has become central to large model training. Subsequently, he participated in the Gopher project, DeepMind's first paper on large language models. The team then consisted of about ten to twelve people. From that point on, he deeply realized that such research requires team collaboration and cannot be accomplished by individuals alone. It was also from then that he began focusing on pre-training, conducting large-scale pre-training. This not only cultivated his research interest but also led him to find his passion.

Host: From an initial Research Engineer to now leading a pre-training team of over a hundred people, Burzo's career trajectory reflects his persistent pursuit of technology and his outstanding abilities in team management and research engineering.

Host: Finally, regarding his future career development, Burzo states that he greatly enjoys collaborating with and learning from many excellent people in his daily work. This is a significant driving force for him. Every day at work, he interacts with very intelligent individuals who teach him a lot of new knowledge, which he finds very enjoyable. Moreover, many different factors are currently at play, and there are still many areas for improvement in the AI field. He is full of curiosity about the future because, from the current perspective, the progress in this type of work seems endless. He feels it's truly fascinating to witness this process and see how far the industry can go. He also believes that the rapid development trend in the AI industry will not slow down for at least the next year or so.

Host: Reflecting on this in-depth conversation, Burzo has unveiled the mystery of Gemini 3 and given us a clearer, deeper understanding of the current state and future of the AI industry. The second half of AI has arrived. As Burzo emphasized, it will belong to full-stack researchers – those who can integrate technology stacks, value teamwork, and possess both research taste and engineering capabilities. Gemini 3's success is merely the opening scene of a much larger future. We have reason to expect that with the collective efforts of numerous researchers, AI will bring us even more surprises. Thank you for watching this episode; see you next time.