Speaker: You're one of the only people outside of OpenAI who in 2020 had this detailed empirical model of scaling. And I'm curious what processes you were using at the time which allowed you to see the picture that you painted in the scaling hypothesis post that you wrote at the time.

Speaker: So I think if I had to give an intellectual history of that, for me, I think it would probably start in the mid-2000s when I was reading more of Ray Kurzweil. Um, at the time, they were making this kind of fundamental connectionist argument that if you had enough computing power, um, that that could result in discovering the neural network architecture that matches the human brain. And until that happens, until that, that amount of computing power is available, AI just seemed basically futile. And to me, I think I found this argument very unlikely, um, because it's very much a kind of "build it and they will come" view of progress, which I just didn't think was correct. Um, I thought that it just seemed ludicrous to suggest that, you know, just because you'd have some like really big supercomputer out there, um, which matches the human brain, then that would kind of just summon out of non-existence the correct algorithm. Algorithms are really complex, they're hard. Um, they required deep insight, or at least I thought they did. Um, and it seemed like really difficult mathematics. You can't just like buy a bunch of computers and then expect to get this advanced AI out of it. Um, it just seemed like totally magical thinking. So I knew the argument, um, but I was super skeptical and I didn't pay too much attention. Um, but then Shane Legg and some others were very big on this, um, in the years following. And as part of my interest in transhumanism and LessWrong and AI risk, I was paying close attention to Legg's blog posts in particular, um, where he's extrapolating, kind of, out the trend with updated numbers from Kurile and Morac. And he's giving these kind of very precise predictions about how, you know, we're going to get the first generalist, uh, system around 2019, as Moore's Law keeps going. And then by 2025, would have kind of humanish Agents with generalist capabilities. And that by 2030, he said we should have AGI.

Speaker: And then Dan Neff and Alex Neff came out. And when those came out, I was like, wow, um, this seems like a very impressive success story for the connectionism view. Um, but is it just an isolated success story, or, you know, is this what Kurile and Morac and Shane Legg had been predicting, that we would get GPUs and then get better algorithms would just kind of show up? Um, so I started thinking to myself that, you know, this is something, it's a trend to keep an eye on. Um, and maybe it's not quite as stupid as an idea, um, as I originally thought. And I just keep reading deep learning literature, notice again and again that the data set size just kept getting bigger, the models seem to keep getting bigger. The GPUs slowly CPT up from one GPU, you know, the cheapest consumer GPUs to two, and then eventually they were trading on eight. And you can just see the fact that the neural network just kept expanding from these incredibly niche individual use cases, which you next to nothing. Um, the youth just kept getting broader and broader and broader. I'd say to myself, wow, is there anything that CNN's can't do? As I just see people applying CNN to something else, you know, every individual day on arXiv. This gradual trickle of drops kind of just kept hitting me in the background as I was going on, um, with my life. You know, every few days, like another one would drop and I'd go like, huh. Um, you know, maybe intelligence really is just like a lot of compute applied to a lot of data, applied to a lot of parameters. Um, maybe Morac and Legg and Kur were right. And I just note that and kind of continue on thinking to myself like, huh, if that was true, it would have a lot of implications. So I think there wasn't really like a Eureka moment there. It was just continuously watching this trend that no one else seemed to see, except possibly a handful of people like Ilya Sutskever, um, Schmidhuber. Um, and I would just pay attention and notice that the world over time looked more like their world than it looked like my world, um, where algorithms are super important and you need like deep insight to do stuff, you know. Um, their world just kept happening.

Speaker: And then GPT-1 came out. And I was like, wow, this unsupervised sentiment neuron is just learning on its own, right? Um, that seemed pretty amazing. Um, it also was a very compute-centric view. You just build the Transformer and the intelligence will come. And then GPT-2 came out. And I had this holy moment. You look at the prompting and the summarization, like, holy, do we live in their world? And then GPT-3 comes out. And that was really the crucial test. It was a huge, huge scale-up, one of the biggest scale-ups in all of neural network history, going from GPT-2 to GPT-3. And it wasn't like it was a super narrow specific task. It really seemed like it was the crucial test. If scaling was bogus, then the GPT-3 paper should have just been totally unimpressive and wouldn't show anything that important. Whereas if scaling were true, you would just automatically be guaranteed to get so much more impressive results out of it than you had seen with GPT-2. So I opened up the first page, maybe the second page, and I saw the few-shot learning chart. And I'm like, holy, we are living in the scaling world. Legg and Morac and Kur were right. Then I turned to Twitter, and everyone else was like, "Oh, you know, this shows that scaling works so badly, why, it's not even state-of-the-art." And that, I was, that made me really angry. I had to write all this stuff up. Um, someone was wrong on the internet.