Unknown Speaker: In the model of the intelligence explosion, what happens is you replace the AI researchers, and then there are a bunch of automated AI researchers who can speed up progress, make more AI researchers, make further progress. We should just ask the AI researchers about whether they think this is plausible.

Unknown Speaker: So, let me just ask you: like, if I have a thousand AI researchers (or AI research units), do you think that you get an intelligence explosion?

Unknown Speaker: I think we are less, at the moment, bound by the sheer engineering work of making these things, than we are by compute to run and get signal, and, uh, and taste, in terms of what the actual right thing to do is. And that, like, making those difficult inferences on imperfect information. So, compute and taste—that's interesting to think about.

Unknown Speaker: Because at least the compute part is not a bottleneck on more intelligence, it just bottlenecks on, say, 7 trillion parameters or whatever, right? So if I gave you 10x the H100s to run your experiments, how much more effective a researcher are you? Please, how much more effective a researcher are you? Uh, I think the Gemini program would probably be like, maybe five times faster with 10 times more compute or something like that. So that's pretty good elasticity? Like 0.5? Wait, that's insane. Yeah, I think like more compute would just like directly convert into progress.

Unknown Speaker: So you have some, um, some fixed size of compute, and some of it goes to inference, some of it goes to running the experiments for the full model. Yeah, that's right. Shouldn't then the fraction that goes to experiments be higher, given that you would just be like, like, the bottleneck is research, and research is bottlenecked by compute?

Unknown Speaker: One of the strategic decisions that every pre-training team has to make is like, exactly what amount of compute do you allocate to your different training runs? Uh, like, to your research program versus like scaling the last best, I like, you know, thing that you landed on. One of the reasons why you need to still keep training big models is that you get information there that you don't get otherwise. Um, so scale has all these emergent properties, uh, which you want to understand better. And if you, like, are always doing research, you're not sure what's going to like fall off the curve, right? Yeah. Um, if you like, keep doing research in this regime, yeah, uh, and like, keep on getting more and more computer efficient, you may never, you, you may have actually like gone off the path that actually eventually scales. So you need to constantly be investing in doing big runs too, at the frontier of what you sort of expect to work.

Unknown Speaker: Okay, so then tell me, what does it look like to be in the world where AI has significantly sped up AI research? Because from this, it doesn't really sound like the AIs are going off and writing the code from scratch and that's leading to faster output. It sounds like they're really augmenting the top researchers in some way. Like, yeah, tell me concretely: are they doing the experiments? Are they coming up with the ideas? Are they just like evaluating the outputs of the experiments? What's happening?

Unknown Speaker: So I think there are like two walls you need to consider here. One is where AI has meaningfully sped up our ability to make algorithmic progress, right? And one is where the output of the AI itself is the thing that's like the crucial ingredient towards, uh, like model capability progress. And like, specifically what I mean there is synthetic data, right? Um, and in the first world where it's meaningfully speeding up algorithmic progress, I think a necessary component of that is more compute.

Unknown Speaker: And, and you probably like reach this elasticity point. Like, AI maybe at some point, are easier to speed up and get on context than yourself, than other people. Um, and so AI meaningfully speeds up your work because they're like a fantastic co-pilot, basically, that helps you code like multiple times faster. Um, and that seems like actually quite reasonable. Um, super long context, super smart model. Um, it's onboarded immediately, and you can like send them off, uh, to like, complete subtasks and sub goals for you.

Unknown Speaker: Walk me through, like, a day in the life of, show, like, you're working on an experiment or project that's going to make the model, quote unquote, better, right? Like, what is happening from observation to experiment to theory to like writing the code? What is happening?

Unknown Speaker: I think the most important, like, part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale. Um, and, uh, the, and like, interpreting, understanding what goes wrong. And I think most people would be surprised to learn just how much, much goes into interpret- like, interpreting and understanding what goes, what goes wrong. Because the ideas people have, long lists of ideas that they want to try. Not every idea that you think should work, will work. And trying to understand why that is, is quite difficult. And like, working out what exactly you need to do to interrogate it. Um, so, so much of it is like introspection about what's going on. It's not pumping out thousands and thousands and thousands of lines of code. It's not, um, like, the difficulty in coming up with ideas. Even, I think many people have a long list of ideas that they want to try. Pairing that down and shot calling under very imperfect information, what the right ideas to explore further is really hard.

Unknown Speaker: Tell me more about what do you mean by imperfect information. Are these early experiments? Are these like, what is the information that you're, like, scaling in low increments? You can see like in the GPT-4 paper, they have like a bunch of like dots, right? Where they say we can estimate the performance of our final model, like, using all of these dots, and there's a nice curve that like flows through them. Concretely, why do, why is that imperfect information?

Unknown Speaker: is you never know if the trend will hold for certain architectures. The trend has held really well, and for certain changes, it's held really well. But that isn't always the case. And things which can help at smaller scales can actually hurt at larger scales. Um, so making guesses based on what the trend lines look like and based on like your intuitive feeling of, okay, this, this is actually something that's going to matter. That's interesting to consider that for every chart you see in a release paper, there's a graveyard of like first runs, and then like, there's all these other lines that go in like different directions.

Unknown Speaker: Yeah, it's crazy. Both like as a grad student, and then also here, like, the number of experiments that you have to run, and then intuiting what went wrong is actually really hard. Like, working out what, like, this is, in many respects, the problem that the team, the Trenton is on, is trying to better understand: is like, what is going on inside these models? We have inferences and understanding, and like, headcanon for why certain things work, but it's not an exact science. Um, and so you have to constantly be making guesses about why something might have happened, what experiment might reveal whether that is or isn't true. And that's probably the most complex part.

Unknown Speaker: Yeah, I I agree with a lot of that. But even on the interpretability team, I mean, especially with Chris Ola leading it, there are just so many ideas that we want to test. And it's really just having the engineering skill, but I'll put Engineering in quotes because a lot of it is research to like very quickly iterate on an experiment, look at the results, interpret it, try the next thing, communicate them. Um, and then just ruthlessly prioritizing what the highest priority things to do are. And this is really important, like the ruthless prioritization is something which I think separates, uh, a lot of like quality research from, um, research that doesn't necessarily succeed as much.

Unknown Speaker: We're in this funny field where, uh, so many of our theoretical, initial theoretical understanding is like, broken down, basically. Um, and so you need to have this Simplicity bias, and like, ruthless prioritization over what's actually going wrong. And I think that's one of the things that separates the most effective people: is they don't necessarily get, like, too attached to solving using a given, like, a solution that they're necessarily familiar with. Um, but rather, they attack the problem directly. Um, you see this a lot, uh, in, like, maybe people come in with, like, specific academic background, they try and solve problems with that toolbox. Um, and the best people are people who expand the toolbox dramatically.

Unknown Speaker: they're you know they're running around in there and taking ideas from reinforcement learning but also from optimization Theory and also they have a great understanding of systems and so they know what the sort of constraints that bound the problem are and they're good Engineers they can iterate and try ideas fast like by far the best researchers I've seen uh they all have the ability to try experiments really really really really really fast um and that is that cycle time and at smaller scales cycle time separates people I mean machine learning research is just so empirical.

Unknown Speaker: yeah and and this is honestly one reason why I think uh our solutions might end up looking more brain-like than otherwise uh it's like even though we wouldn't want to admit it the whole Community is kind of doing like a greedy evolutionary optimization over the landscape of like possible Ai architectures and everything else uh it's like no better than evolution.