Hi, I'm Matt. Welcome to the Matt podcast. My guest today is Lucas Kaiser, one of the key architects of modern AI who has quite literally shaped the history of the field. Luc was one of the co-authors of the attention is all you need paper, meaning he's one of the inventors of the transformer architecture that powers almost all the AI that we use today. He's now a leading research scientist at OpenAI, helping drive the second major paradigm shift towards reasoning models like the ones behind GPD 5.1.

This episode is a deep exploration of the AI frontier. Why the AI slowdown narrative is wrong, the logic puzzles that still stump the world's smartest models, how scaling is being redefined, and what all of that tells us about where AI is heading next. Please enjoy this fantastic conversation with Luc.

Lush, welcome.

Thank you very much.

There was a narrative at least in some circles maybe outside of San Francisco throughout the year that AI progress was slowing down, that we had maxed out pre-training, that scaling laws were hitting a wall. Yet we are recording this at the end of a huge week or couple of weeks with release of GBD 5.1, GPD 5.1 Codex max, GVD51 Pro as well as Gemini Banana Pro for Gro 4.1 Almo 3. So this feels like a major violation of that narrative. What is it that people in Frontier AI labs know about AI progress that at least parts of the rest of the world seem to not understand?

I think there is a lot to unpack there. So I want to go a little slower. There is this thing that's happening in AI and in AI every week now a lot is happening. you know, new model, coding, doing slides, self-driving cars, images, videos, you know, it's a nice field that doesn't make you be bored for a long time.

But through all of this, it's sometimes hard to see the fundamental things that are happening. And fundamentally, if you look at AI progress, it's been a very smooth exponential increase in capabilities. This is the overarching trend and there has never been much to make me at least and I think my colleagues in the labs believe that this trend is not happening.

It's a little bit like Moore's law, right? Moore's law happened through decades and decades and arguably you would say it's still very much going on if not speeding up with the GPUs. But of course it did not happen as like one technology was bringing you there for 40 years. there were was one technology and then another and another and another and another and this went on for decades, right? So, from the outside you see a smooth trend but from the inside of course you know progress is made through new developments in addition to the increase of computer power and better engineering.

So all of these things come together and in terms of language models I think there was a big pivotal point. I mean one point was of course the transformers when it started but the other point was reasoning models and that happened I think one preview was a bit a year and a month ago or something like that. So we started working on it maybe 3 years ago but it's very recent if you think of it as a paradigm that that's a very recent thing.

So, it's always like these S-curves, right? It starts then it gives you amazing growth and then it flatlines a little bit. I feel pre-training in some sense it's on the upper part of the S, but it's not like scaling for pre-training don't work. They totally work. What scaling clause says is that your loss will log linearly decrease with your compute. We totally see that and clearly Google sees that and all other labs. The problem is, you know, how much money do you need to put into that versus the gains you get and it's just a lot of money and people are putting it.

But with the new paradigm of reasoning, you can get much more gains for the same amount of money because it's on this lower part of the S-curve. There are just discoveries to be made and these discoveries unlock insane capabilities. So, it's not like pre-training fizzled out. It's just we found out a new paradigm that at the same price gives us much more amazing development. And this paradigm is still very new.

It happens so fast. I think if you blink you may miss it because it was basically you had chat 3.5 right GPD 3.5 in chat and it would give you answers and it used no tools, no reasoning. It would answer you something. And now you have chat and you know if you were not into it you may have blinked and it also gives you answers and you may say okay it's more or less the same except the chat now will you know go look on some websites reason about it and give you the right answer instead of something it memorized in its weights.

I very much used to like this example of when what time does the SF zoo open tomorrow? The old chat would tell you right totally hallucinate from its memory an hour that it read zoo opens on probably the zoo's website from 5 years ago and it didn't know what's today or tomorrow so it would just assume it's a weekday. Chat now knows what's today because it's in the system prompt it goes to the zoo website reads it extracts the information if it's ambiguous probably checks three other websites just to confirm and then gives you the answer. But if you blink, you may think it's the same, but no, it's dramatically better.

And as a consequence, since it can read all the websites in the world, it can give you answers in stuff that it wouldn't be able to even touch before. So, there is tremendous progress, right? And it happened so fast it may even be missed. I think one of the biggest things that I would say people kind of know on the inside and others don't is that already right now it's not about the progress. There are so many things Chad or Gemini any LLM can do for you that people just don't realize. You can take a photo of something broken ask how to repair it. It may tell you you can give it a college level homework and it will do it for you. That's absolutely amazing.

So there is an education gap to some extent.

Well, it just happened. I mean if you think you said codeex, right? You know programmers are conservative a little bit. I still use Emacs from time to time. All the coding tools like okay it will complete one line for me but people are very like this is my editor. I write code here. Now people are like no this is codeex. I ask it to do stuff. I will fix it later. Right? But I think it's the recent few months when the transition happened from people using it sometimes but rarely to now basically this being how a lot of people work in coding. That's quite big. I'm not sure everyone's aware of it but it's also like if you don't do programming why would you be aware of it? I do believe though that this will come to more and more domains.

To the point of all of this being very new and somewhat sudden. Something that you or I hear from time to time when talking to people is that part of the reason why people are so optimistic is that there is a lot of lowhanging fruit, very obvious things to improve for those models in the next few months. First of all, do you agree? And second, can you give us some examples like obvious thing that you need to fix next and that the industry will fix?

Yes, there is a ton of extremely obvious things to fix. A larger part of this ton is just hard to talk about on a podcast because it is in the engineering part. You know, every lab has their own infra and their own bugs in the code. Machine learning is beautifully forgiving in some sense in contrast to old software engineering which would just yell at you when you made a mistake. You know, our Python code generally probably run except much slower and give you worse results if you run it wrong. So you realize, oh no, it was wrong. And you improve it and the results get better. These are huge distributed computing systems. They're very complex to run. So there is a huge amount to improve and fix and understand in the process about just how to train your model and how to do RL because RL is more finicky than pre-training. It's harder to do really, right?

On top of that, there is data. You know, we used to train on just like common crawl basically. It's a big repository of the internet that people just scraped without regard of what and and some things came in, some didn't. It it was a mess. So now, of course, every larger company has a team that tries to filter this and improve the quality. But it's a lot of work to really extract better data. Now synthetic data is becoming a thing but when you generate synthetic data it really matters how you do it with what model what you the whole how the in the engineering aspects of everything.

It's such a new domain that you know it was done somehow it works it's beautiful but there is just so much to do better that people I don't think have any doubts that there is a lot there. And on top of that there are the big things like multimodal. I mean language models they are now as I'm sure you know and most people realize actually vision language models because and they can also do audio. So they're multimodal models to some extent but the multimodal part still lags behind the text part to a large extent. So that's one big area where obviously you need to do better and it's not a huge secret how you can do better. You know there are some methods that maybe will make it even amazingly better but there are some very simple methods how you can do just better but you know this maybe requires retraining your whole base model from scratch and that takes few months and it's a huge investment so we need to organize it. So there is a lot of just work that will undoubtedly make things better. I think the big question that people have in their mind is how much better will it make them?

So I'd love to do a little bit of a deep dive slash educational part on the whole reasoning model aspect because as you just mentioned since it's so new some people truly understand how those work many people don't. At a very simplistic level, what is a reasoning model and how is that different from your sort of base LLM?

So, a reasoning model is like your base LLM, but before giving you the answer, it thinks what people call in the chain of thought, meaning it generates some tokens, some text that's meant not for you to read, but for the model to give you the better answer. And while it does this these days, it is also allowed to use tools. So it can for example in its so-called thinking process go and browse the web and to give you a better answer.

So that's the superficial part of the thinking models. Now the deep part is that you start treating this thinking process as part of the model basically. So it's not something the model generates and it's an output for you. It's something you want to train right. you want to tell the model you should think well you should think so that the answer after this is good in whatever way and this leads you to a very different way of training the model.

Because models were in usually trained with just gradient descent the way deep neural networks are trained meaning you say predict the next word and you do a gradient you differentiate your function from the model they're not fully differentiable but you approximate it and you train your weights to do that and that it was quite amazing that doing just that you could make a chat. But with a reasoning model you can do that because there is this reasoning part you can differentiate through that so we train this with reinforcement learning and reinforcement learning basically you okay there's just this reward and you need to do a bit of tries and reinforce meaning push the model towards doing more of the things that lead to better answers.

And this kind of training is a bit more like it has more restrictions than the training we used before. The training we used before, you took all of the internet, put it in, even if you didn't filter it very well, it would mostly work. Reinforcement learning, you need to be careful like you need to tune a lot of things, but you also need to prepare your data very carefully. So currently and for at least the most basic ways we use it currently, it needs to be fairly verifiable. So there is your answer correct or not? You prepare data for that. You can do that in mathematics, coding very well. You can do this in science to some extent right you can have test questions they're correct or not but you know if it comes to like writing poems is this poem good or not it's for now the reasoning models are really shine in in domains like science.

Great. Thank you for this. All right. As a quick detour and we'll go back to more Frontier AI topics. I'd love to talk a little bit about your story. I mean you have the incredible distinction of having been at the forefront of this industry both the transformers paper which was the birth of one paradigm and now you're very much leading the charge on the reasoning model part which is another paradigm so this just incredible story how did you become an AI researcher?

I was a mathematician and a computer scientist but in theoretical computer science. Yes I was definitely very into math in high school and into computers also later in high school. Yes, I did my studies in Poland. I went for a PhD in Germany. It was the theoretical computer science and mathematics PhD. So I very much I'm a mathematician. I was always fascinated by you know how is this thinking going? What is intelligence? As a child I always wanted to like emulate the brain.

Then there was this opportunity to join Google just as the deep learning was starting off. I already had my tenure position in France and the French system has this beautiful thing that you can take a leave of 10 years. And you can still return anytime you want. So it's a no risk.

So at some point when you when you've solved AGI you may return back to France and be a professor.

Well, if you solve AGI, they may take you anyway. The nice part about the leave is that they will take you back even if you don't. But it's actually very important. A number of I think there's a number of Nobel Prize winners who who took this leave to just try something more risky and you know, sometimes it works, sometimes it doesn't. There's a lot of luck in in science and research but it's very good to have this opportunity to take it. So I came to Google that was Google brain at the time. I came to Ray Kurzweil's group he was my first manager he interviewed me and was very inspiring.

I have to ask you at this point about the Transformer paper story, how it all came about. The eight of you, right? Seven or eight of you. How did you all get together?

Well, we never got together. I recently got a photo on Twitter of a photo session of all eight of us and it was saying it was fake but I knew it was fake because I don't think all eight of us were ever in the same physical room. These ideas developed from many sites before and after.

And maybe one minute for the broad public on what attention actually means since it's such a fundamental concept.

So, attention is the mechanism that tells the model as you're doing the next thing, look into your past and find the most similar things that you see in the past to what you are seeing right now. It came from the machine translation times where people wanted to align words in one language with words in another. The main novelty of Transformer was self attention. But Transformer is more than just this idea. I think that's the beauty of these eight weird people somehow came together even though not physically to do it is that we all approached it from different sides. So there was people working on the attention idea. There is how do you store knowledge in neural networks. And then you know in deep learning people laugh that ideas are cheap making them work is the hard part.

And I remember distinctly that people were like, "So you want to use the same model for a few different tasks? Like why do you even do that?" Like if you have a different task, like if you do translation, you train one model, if you do parsing, you train another. You never train the same model for three different tasks. And I was like, "No, no, we're going to do all tasks in one model." And people were like, "No, no."

Talk about the transition from Google to open AI and perhaps how those two cultures are different.

So, Ilia was my manager at Brain. Then he went on to found OpenAI. He asked me a number of times whether I would like to join in the years. I found it a little bit too edgy at the time. Then Transformers came. So, we had a lot of work with that. And then COVID came and COVID was a tough time for the whole world, right? But Google totally closed. Google was reopening extremely slowly. So one part of me was I find it very hard to do remote work. I much prefer to work with people directly. That was one reason.

But the other was also Google Brain when I joined it was a few dozen people, maybe 40, something like that. When I left it was 4,000, 3,000 people spread across multiple offices. It's very different to work in a small group and to work in a huge company. So with all this, I was like, you know, OpenAI though is in a much stabler state. We're doing language models. You know, something about this that may look like a good match. And I was like, okay, let me try. I've never worked in any company other than Google before, other than the university. So it was quite a change to the small startup group, but I like working in smaller groups.

Great. What is next for pre-training?

Pre-training as I said I think it has reached this upper level of the S-curve in terms of science but it can scale smoothly meaning if you put more compute you will get better losses if you do things right which is extremely hard and that's valuable.

I think what people underestimate a little bit in the big narrative is you know open AI three four years ago was a small research lab with a product called API but it was not such a big there was no GPU constraint on the product side for example all GPUs were just used for training so it was very easy as a decision for the people to say you know we're going to train GPT4 this will be the smartest and largest model ever. And what do we care about small models? I mean, we care of them as to make like to debug the training of the big model, but that's it. So, GPT4 was the smartest model and it was great, right?

But then it turned out, oh, there is this chat and now we have a billion users and you know, people want to chat with it a lot every day and you need GPUs. So you train the next like huge model and it turns out you cannot satisfy this like people will not want to pay you enough to chat with the bigger model. So you just economically need the smaller model. So and this happened of course to all the labs because like the moment the economy arrived and it became a product you had to start thinking about price much more carefully than before.

So I think this caused the fact that instead of just training the largest and largest thing you can for the money you have we said well no we we're going to train like the same thing but same quality but smaller cheaper. The pressure to give the same quality for less money is very large in some senses as a researcher almost makes me a little sad I have a big love for these huge models.

On the side people kind of rediscovered how amazing distillation is. Distillation means you can train a big model and then put the same knowledge from the big the big is a teacher to the little one. People knew about distillation. It's a paper a long time ago but somehow people kind of rediscovered how important that is for the economics but now it also means that oh training this huge model is actually good because you distill all the little ones from it. So now maybe there is a bit more of a return to it.

Walk us maybe through the evolution of from GD4 to 5 to 5.1. What has actually changed?

That that's a very tough question. I think less than you think. From GPD4 to 5 I think the biggest thing that changed is reasoning meaning RL and synthetic data as I told you the pre-training part in that time frame was mostly about making things cheaper not making things better. The main improvements from four to five is adding reasoning with reinforcement learning and this allowed to generate synthetic data which also improves the model. So that's the big picture.

In addition to that, ChatGPT is now a product used by a lot of people. So the post- training team has learned a tremendous number of lessons. There was these things called hallucinations. It's still with us to some extent, but dramatically less than than two years ago. Some part of that is because reinforcement learning can now use tools and gather data and it also encourages the model to know you know verify what it's doing. The 5 to 5.1 is mostly this kind like it's mostly a post-training improvement.

The fundamental training method of reasoning is very limited to science data. So, it's not as broad as the pre-training. Pre-training models felt kind of almost uniformly good or bad at things. But the reasoning models are even more people call it jagged, right? They have amazing abilities somewhere and then close by not so much. And that can be very confusing.

It's something I always love that it's weird because you can say the model is amazing at mathematical olympiad. At the same time, I have a math book for my first grader daughter. She's 5 years old. I took one exercise from this math book and none of the frontier models is able to solve it. and you would be able to solve it in 10 seconds. So that's something to keep in mind. Models are both amazing and there are tasks that they cannot do very well.

I can show you this as an example. It has two groups of dots on both sides and the question is is the number of dots even or odd? If you look at it you see oh they're like two identical things. So that would be even. That's what the 5-year-old is supposed to learn. But there is one dot that's shared. So now that must be odd. For this simple one which has like 20 dots or so. Gemini 3 actually does it right? And then you have another puzzle which is very similar except now there are two mountains of dots and there's also one dot shared at the bottom now. And right in context right after that you ask okay how about this one and then it does some thinking and it just totally misses that there is a shared dot and it says the number is even.

Here is the exact same prompt for GPT 5.1. It also solves the first, it sees the dot. It says it's odd and then it sees the mountains and somehow it doesn't see the dot and it says it's even. The nice thing is if you let it think longer or if you just let it think again it will see it. So if you use GPT5 Pro it takes 15 minutes. The human 5-year-old takes 15 seconds. The GPD51 Pro will run Python code to extract these dots from an image and then it will count them in a loop. So that's not quite...

And why is that? What trips up the model?

I think this is mostly multimodal part. The models are just they're starting. They have not yet learned to do good reasoning in multimodal domains and they have not yet learned to use one reasoning in context to do the next reasoning. All of these though are things that are very well known and like the models are just not trained enough to do this. it's just something we know we need to add into training.

I think the big question in all of AI is is reasoning enough to increase generalization or do you need like more general methods? I think the first step is to make reasoning more general. That's my passion. That's also what I work on.

They still have limitations because they don't live in the physical world because they're not very good at multimodal because reasoning is very young and there's a lot of bugs in how we do it yet. But once we fix that there will be this big question is that enough or is there like something other big to to make models generalize better so we don't need to teach it every particular thing in the training data that it just learns and generalizes. I think that's the most fascinating question.

Do you see promising fundamental architectural changes outside of transformers that have caught your attention and feel like they could be a serious path to explore in the future?

I think there is a lot of beautiful work that people are trying out. Yann LeCun has been pushing for for other methods. So, I feel like his approach is more towards the multimodal part. But maybe if you solve multimodal, it also helps your other understanding. There is a lot of people pushing fundamental science still. The engineering part is the biggest bottleneck. I mean GPUs are a bottleneck too when you scale really up but implementing something that's larger than one machine it's an experimental research project so you don't have a team to do that I think that's still harder than it should be.

One theme that people like me, VCs and founders and startup think about a lot as we see all the progress at OpenAI is as the models keep getting more general with more agent capabilities the ability to run for a very long time going to areas like science and math... is there a world where basically models or maybe just one model does everything? What's left for people that build products that sit on top of models?

I just showed you a 5-year-old exercise that the model doesn't do. I think we need to keep that in mind.

So you're saying there's hope.

There is hope. The next model will do it right.

Transformer paper started with translation. I recently went to a translation industry conference. The translation industry has grown considerably since then. It has not shrunk. There's more translations to be done. Translators are paid more. The question is why would you even want a translator if the model's so good in most of the cases. The answer is sometimes imagine you do a listing for a newspaper but in a language you don't know and GPD5 will almost certainly translate it correctly for you. Would you still publish it without having a human who speaks that language look at it? It's a question of trust probably, right?

This is in an industry that fundamentally is totally automated, right? There is still the question of trust and I think that's a question we will grapple with for a long time. There are also just things you want a person to do. I don't think we will have no things to do but that doesn't mean that some things we do may not dramatically change.

And maybe as a last question to help us get a sense for what people at the frontier of AI are currently thinking about or working on, what do you personally find really interesting as a research area?

I always find this general data reinforcement learning is my pet peeve and what I work on. Luckily that for example, robotics is probably just an illustration that we are not doing that well in multimodel and that we're not doing that well in general RL in general reasoning yet. The moment we do really well in multimodel and we manage to generalize reasoning to the physical domains that the robot needs, I think it will see amazing progress.

When it does, I have a feeling given that a lot of companies are launching hardware that's kind of teleoperated or glove operated or something. So my suspicion is by the moment we make this progress, the hardware will may be there by then and having a robot in a home may be like a big visible change. More visible than chat. It's stunning to me how quickly we get used to these things, right? The self-driving cars in San Francisco are something people got used to so quickly. So, maybe this will happen for robots, too. Nevertheless, I do think it will be quite dramatic in our perception of the world when it happens.

Lucas, it's been absolutely wonderful. Thank you so much for spending time with us today.

Thank you very much Matt. Thank you for the invitation. Great to talk to you.