Alessio: Hey everyone, welcome to the Leading Space podcast. This is Alessio from Google Kernel Labs, and I'm joined by Squix, editor of Leading Space. Hello, hello. We're here in the studio with Jeff Dean, chief AI scientist at Google. Welcome. Thank you for having me. It's a bit surreal to have you in the studio. I've watched so many of your talks, and obviously, your career has been super legendary. So, I mean, congrats. I think the first thing must be said: congrats on owning the Pareto frontier.

Jeff Dean: Thank you, thank you. Pareto frontiers are good, and it's good to be out there. Yeah, I mean, I think as you say, it's not just one thing; it's like a whole bunch of things up and down the stack, and you know, all of those really combine to help make you and us able to make highly capable large models, as well as you know, software techniques to get those large model capabilities into much smaller, lighter weight models that are you know much more cost effective and lower latency, but still you know quite capable for their size.

Alessio: So, yeah, how how much pressure do you have on like having the lower bound of the Pareto frontier too? ... How do you prioritize frontier versus like we actually need to deploy it if we build it?

Jeff Dean: Yeah, I mean, I think we always want to have models that are at the frontier or pushing the frontier because I think that's where you see what capabilities now exist... At the same time, you know, we know those are going to be really useful for a bunch of use cases, but they're going to be a bit slower and a bit more expensive... So I think we like to do both, and also, you know, through distillation, which is a key technique for making the smaller models more capable... for multiple Gemini generations now we've been able to make the sort of flash version of the next generation as good or even substantially better than the previous generations pro.

Alessio: Are there any benchmarks or like test sets they use internally because it's almost like the same benchmarks get reported every time... Like, how do you have to keep pushing the team internally to to like this is what we're building towards?

Jeff Dean: Yeah, I mean, I think benchmarks... often kind of have a lifespan of utility... Once it hits kind of 95 percent or something, you get very diminishing returns... We have a bunch of held out internal benchmarks that we really look at where we know that wasn't represented in the training data at all... long context is useful, but it's way too short today... I think what you would really want is can I attend to the internet while I answer my question? ... But that's not going to be solved by purely scaling the existing solutions, which are quadratic.

Alessio: What you know, this you know, mention of latency and and saving things to disk reminds me of one of your classics... latency numbers every programmer should know. ... I've been trying to make numbers every AI programmer should know.

Jeff Dean: Yeah, I mean, it's all going to be about energy and how do you make the most energy efficient system? And then moving data from the SRAM on the other side of the chip... can be you know a thousand picojoules. ... So you better make use of that that thing that you moved many many times. So that's where the batch dimension comes in because all of a sudden, you know, if you have a batch of two fifty six or something, that's not so bad, but if you have a batch of one, that's really not good.

Alessio: I'm still not over the fact that a year ago we had AlphaProof and AlphaGeometry... and then this year we were like, "Screw that, we'll just chuck it into Gemini."

Jeff Dean: Yeah, I mean, I think it makes a lot of sense to me because you know humans manipulate symbols, but we probably don't have like a symbolic representation in our heads... I think general models will win out over specialized ones in most cases.

Alessio: We had a previous guest, David Wan... he kind of like blames almost the brain marketplace as like the reason that Google didn't invest enough in language models.

Jeff Dean: Yeah, I mean, I think I actually wrote a one page memo saying we were being stupid by fragmenting our resources... I said, this is just stupid. Why don't we combine things and have one effort to train? ... and that was the origin of the Gemini effort. ... I kind of liked that, and then there's also the NASA interpretation...

Jeff Dean: Yeah, I mean, first, I think the coding tools are you know getting vastly better... Imagine if you had a team of 50 interns, how would you manage that if they were people? ... I think that is probably within the realm of possibilities that lots of people could have 50 interns.

Jeff Dean: The better you get at interacting with these models... they will get really good at crisply specifying things rather than leaving things to ambiguity. ... being able to crisply specify what it is you want is going to be really important. Yeah, my my joke is you know good prompting is indistinguishable from sufficiently advanced executive communication.

Alessio: Should we ask him for predictions to to go? ... What's something that you're you're not quite happy with yet that you think will get done soon?

Jeff Dean: Let me make two predictions... So I think a personalized model that knows you and knows all your state... is going to be incredibly useful. ... When you say much lower latency... we're at let's say 100 now. We can go to the thousands. Is it meaningful to go 10,000s? Yes. ... it may be a thousand tokens of code that with nine thousand tokens of reasoning behind it.