[Host]: Today, I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You've been on sabbatical for the last few months. One of the things you've been doing is rebuilding, improving, and hacking on AlphaGo. Today, you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang: I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning. That was quite mysterious to me, and I've always wanted to understand that phenomenon a little better. My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It's always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree. If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

[Host]: We should first discuss how Go works. How does the game work?

Eric Jang: Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. Black always goes first. The way you capture an opponent's stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it's cut off from oxygen, if you will, and it's a dead stone. There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you're actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it's actually fine. You put it down, and it immediately resolves to death, so the outcome is the same. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

[Host]: How does the game end?

Eric Jang: The game ends when either a player chooses to resign or both players pass consecutively. Those are the rules. We should talk about how you resolve scores between humans and how you resolve scores in computer Go, because there's some ambiguity in how humans evaluate this. Most humans would look at this board configuration and conclude that black has totally surrounded white, and white has no chance of life. This will map later to how we think about the deep neural network. You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates p(win). The human glances at the board and knows, "I'm probably going to lose." They're essentially running a neural network that looks at a board and implicitly amortizes a huge number of possible game playouts. This gives us a hint that in games like Go, there are ways to radically speed up the search process. You can train a value function to look at a board and quickly resolve the game without playing out all of these trees to a very deep search depth.

[Host]: Now help me crack this with AI. Let's understand how AlphaGo actually works.

Eric Jang: The number of steps in the game can be somewhere from 250 to 300 moves. If you keep on expanding possible moves, you end up with an enormous explosion in the possible game outcomes. This is something on the order of 361^300, which is far more than the number of atoms in the universe. AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable. We use Monte Carlo Tree Search (MCTS). Every simulation involves this four-step process: selection, expansion, evaluation, and backup. In AlphaGo, they use a slightly different action-selection criterion called PUCT, short for Predicted Upper Confidence with Trees. Q(s,a) is the mean action value, how good a given child is on average. In both UCB and PUCT, there's this term that basically rewards taking actions you haven't taken before.

[Host]: Let's talk about the neural network part of this.

Eric Jang: There are two networks. There is the value network, which takes in a state and predicts, am I going to win or lose? Then we have a policy network, which induces a distribution over good actions to take. Transformers work, ResNets work. For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. They provide the inductive bias of local convolutions. One interesting finding from the KataGo paper was that they found it quite useful to pool together and aggregate global features throughout the network. The original AlphaGo paper, called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later, they removed this restriction by having the model teach itself how to play well. I find it super nice for implementation to always initialize your experiments to something easy and get the problem working before trying to bite off the whole thing and learn tabula rasa.

[Host]: We now talk about the RL part of how this thing gets stronger by playing itself.

Eric Jang: The beauty of how AlphaGo trains itself is that it can actually take this final search process and tell the policy network, "Hey, instead of having MCTS do all this legwork to arrive here, why don't you just predict that from the get-go?" It's almost as if you could amortize the first 1,000 steps into the policy network instead of the search process. This is very related to DAgger in robotics and imitation learning. The inefficiency of policy gradient RL is that you have to roll out a whole trajectory to get any learning signal at all. In LLMs, they treat this entire sequence as a single action. AlphaGo is doing something very fundamentally different. It's not trying to do credit assignment on wins. It's trying to improve the label for any given action you took.

[Host]: If you understand it more deeply, AlphaGo seems less impressive in retrospect the more you understand it. You're building this very explicit tree search for it. I don't know if you share that intuition.

Eric Jang: I personally disagree. I think they're profound for different reasons. I think the most profound thing here is that a 10-layer neural network pass is able to amortize and approximate to very high fidelity a nearly intractable search problem. This was a breakthrough that I think most people don't even fully comprehend today. It actually makes me wonder if our understanding of problems like P=NP, or these fundamental computational hardness problems, is incomplete. I wouldn't say this solves Go, but in practice, it is extremely useful. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute.

[Host]: One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this automated LLM coding assistant loop. What are your observations about what the AI is good at?

Eric Jang: I think automated scientific research is one of the most exciting skills. What works is that the models can do a very good job of hyperparameter optimization. It is also fantastic now at basically executing any experiment. But they don't seem to be that great at stepping back and do the lateral thinking. Often I had to catch infra bugs myself. Go captures a lot of very interesting research problems. You can check the outcome of a Go game quite easily. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics. Or automating AI research.

[Host]: Awesome, Eric. Thanks for doing this. Where do people go next?

Eric Jang: My website is evjang.com. On my GitHub, there's an AutoGo repo. I also highly recommend people check out this blog post, "As Rocks May Think." Thank you. It's an honor to be on the podcast.