生命即计算：Blaise Agüera y Arcas 探讨 AGI 共存与意识本质

Blaise Agüera y Arcas: My name is Blaise Agüera y Arcas. I am the CTO of technology and society at Google and the founder of an organization there called Paradigms of Intelligence which does basic research on AI. I also have just published a book called What Is Intelligence? Today on Big Think, we're going to talk about life functionalism and how to live with AGI.

Narrator: This Big Think interview is brought to you by Pangram. Navigating an internet filled with AI generated content is a frustrating experience. It is becoming harder to separate real from fake and human from machine. Pangram is the most trusted AI detector backed by rigorous public research. Pangram can tell you how much of what you're reading is AI generated. If there's an image you think was made by AI, Pangram can detect that, too. To regain trust over what you read and see, go to www.pangram.com. And now back to the interview.

Blaise Agüera y Arcas: Chapter one, defining life, computation and symbiogenesis.

So after thinking about it for a while, the definition of life that I've come up with—and I should caveat this that, you know, every biologist's definition of life is going to be different. This is not a term that there's any consensus about. We actually wrote a paper about that in which we interviewed all kinds of interdisciplinary researchers, and, you know, for 60 experts there are 60 definitions in there. But mine, for what it's worth, is that life is self-constructing computation that complexifies through symbiogenesis.

So what does that mean? Let's start off with self-constructing computation. A lot of people when they hear computing, they have associations with a very specific technology that began in 1945. They think of it as an applied science, and computing as Turing and Von Neumann thought about it is actually something way more fundamental. It's a basic science in the same way that thermodynamics in the 19th century, which describes phases of matter and so on, is a basic science and long predated steam engines.

Blaise Agüera y Arcas: So Alan Turing developed a theory of what computation is. He did something really interesting which was to look at computation which at the time was done entirely by humans and define a mathematical abstraction for what it means to do this kind of computation. He stripped down the system to something very very abstract to just a read-write head that can read and write symbols on a tape called a Turing machine.

Now the real magic in Turing's mathematical proof is that he then said, well, you know, any computation can be defined with the right table of rules. Could be doing whatever: it could be adding numbers together, it could be taking square roots, it could be figuring out the statistics of a whole corpus of text, doing the kind of things that an LLM does, no matter, but all of these are different tables of rules. However, there are certain machines which he called universal machines whose table of rules allows for a different table of rules to be written as symbols on the tape.

Blaise Agüera y Arcas: So this is a weird kind of meta move in which you say, "Oh, there are certain machines where the rules themselves can be written as data on the tape along with the inputs." And that means that those universal machines can perform any computation, not just the computation that their table of rules tells them to do.

So, you know, what I've just described is exactly what a computer is. A computer is a universal Turing machine, and when we say that it follows an algorithm or runs code, the code is exactly that written out table of rules that is actually written in the memory and that is read by the machine. So it's a really interesting move and it's one that allows for arbitrary behavior to be done by a system that is itself quite simple and has fixed complexity.

Now if we just think Turing machines generally—meaning not universal Turing machines, but just any old Turing machine—all that is is something that is abstract, that has a state, that can take inputs, change its state and produce outputs. Now that is such a general construction that, you know, it applies to anything in nature that you want to think of that way. It could apply to Jupiter. It could apply to the water in a stream if we define state, input and output in the appropriate way. But a universal machine, in other words, one where the directions that it's following define the things that it does, that's something a lot more particular, and that really begins with life.

Blaise Agüera y Arcas: I should also say that there is a key concept here. It's a little bit deep mathematically, but it has to do with the mapping from micro states to macro states. The reason that time has an arrow, that the past is the past and the future is the future, has everything to do with entropy, with the fact that things go from more ordered to less ordered states. So if, for instance, you're looking at a bunch of balls bouncing around on a pool table, but suddenly you see them all come together into a triangle and one ball shoot away with all the momentum, now you know you're watching the movie in reverse.

And it turns out that computation and entropy are also very closely related, and that you can think about computation as really being the story of how causality comes into the world. And when you look at any computer language that's ever been invented or discovered, they always have something like an if-then in them. And the reason that they have an if-then is because they need to be able to create causes in the world. Anything that is a behavior, that is a computation, requires conditionality. It requires that you do something on the basis of what was input.

Even if you think about a thermostat: when the temperature drops below a certain level, it will turn on the heat. And of course, the heat will then cause the temperature to rise, which will then cause the switch to turn off. So that's an if-then. It's a minimal form of behavior. The behavior of a thermostat is conditional on the temperature. So, you know, that's sort of the elementary building block behind anything that can be said to behave. And it's also an elementary building block in anything that can be said to compute. The science of causality is a way to think about computation and how computation arose in nature and especially in the emergence of life, as opposed to just being about computer technology that we began making in 1945.

Blaise Agüera y Arcas: Now there's another really interesting chapter to this story of computing itself which was really invented by John von Neumann. Now, John von Neumann was trying to work out the theory of how it is that a thing can reproduce. He was trying to solve the Baron von Münchhausen problem, you know, pulling yourself up by your own bootstraps. How could life bootstrap itself when you need to have an instruction tape in life? And that instruction tape needs to have the instructions for building a self. But, you know, if the self is copying the instruction tape, then it would seem that you have to have an infinite regress, that in order for you to be able to construct yourself you need to kind of have a copy of a copy of a copy of a copy of yourself, and this is kind of like a logical paradox.

So the way Von Neumann solved this was by saying, well, if your inner instruction tape has the instructions for making two different machines, as he called them, machine A and machine B: machine A would follow the instructions on the tape in order to build whatever it says, and machine B makes a copy of the tape. In other words, it treats those instructions as symbols rather than treating them as instructions. And if the instructions for building both machine A and machine B are themselves also on the tape, then you kind of break the paradox. You have a system that can reproduce itself.

Blaise Agüera y Arcas: This is really cool because he figured it out in the late 1940s. And this was before Watson, Crick and uncredited colleagues had figured out the structure and function of DNA, which is exactly this instruction tape that he was talking about. But also before the discovery of the ribosome, which is what he called machine A, or the discovery of DNA polymerase, which is what he called machine B, both of which indeed have the instructions for building them in our DNA. So he got all of this exactly right, and also he realized that this was basically the recipe for a universal Turing machine.

In other words, a computer: machine A literally is following the directions on a tape in order to perform certain actions, and that's basically how Turing defined computation. So what this means is that the original symbiogenesis that gives rise to an organism that can reproduce itself heritably is literally the coming together of a computer. So in that sense biology is computer science. The birth of biology is the origin of the first computers, and what we have today in our phones, our laptops and so on we should call artificial computers in the same way that we call AI systems artificial intelligence.

Blaise Agüera y Arcas: So, we know from Von Neumann that life has to be self-constructing. In other words, it's got to be able to grow, heal, and/or reproduce. Otherwise, it will no longer exist in the future. If it's complex, if it's more than just a really trivial crystal, then that requires general computation in order to build itself. And in particular, you need the heritability property. In other words, you need it to be the case that it's got a set of instructions inside itself such that if those instructions are changed, then future generations will also change how they are. Right. So there's got to be this kind of persistent instructions that allow heritability to occur.

So life is general computation, or I called it playfully computronium, meaning matter that computes. And the other part of that definition is that once you've got that property, if what it's interacting with is other life—which will always be the case by the way, because the moment life can reproduce suddenly it's in a bath of other life—then that opens the possibility for something called symbiogenesis.

Darwin was always troubled by the question of how life gets started. We tend to assume that life evolves through random mutation and selection for fitness. So, this is called the neo-Darwinian synthesis. It's not exactly what Darwin came up with, but it's what I guess Darwin settled into as he was interpreted by later thinkers. So the idea here is that we have a genome and that genome gets hit with radiation, randomly mutated, and those random changes stick to the degree that those random changes confer some survival advantage. Now that story is not exactly wrong, but it's only half the story. As it turns out, the big steps in the advancement of life have come from cooperation, not competition. They've come from entities, two or more

entities, sometimes many entities with either the same genome or different genomes coming together in an act called symbiogenesis. Symbiogenesis is when two or more life forms come together to make a more complex life form that itself can reproduce. If it cooperates with the life around it, then it can become oblate, meaning that those cooperating things have to now reproduce together.

Now the the story of symbiogenesis which is to say that you know things come together cooperate to make bigger more complex things turns out not only to address the problem of how life becomes more complex over time but also the problem of how life gets started. So it turns out that when molecules cooperate meaning uh you know molecules somehow end up in relationships with each other chemical reactions whereby those reactions lead to more of those molecules and then those cycles of chemical reactions which are autocatalytic uh would be the technical term start to interact. Those can make systems that that in turn propagate themselves. That story of symbiogenesis going all the way down to molecules is not only the story of how life becomes more complex but also of how life arises in the first place.

Uh and this this story of cooperation is is not what we generally think of when we think of evolution. We think only of competition. But uh cooperation is how we went from uh molecules to bacteria. It's how we go from bacteria to ukarotes. The kind of cells that we're made out of. Uh it's how we go from uh from ukarotes to multisellular organisms like us. Uh and it's also how we've gone from individual humans to societies. And and that cooperative story is just as important as uh as competition. In fact, in some sense, cooperation and competition um are are not even distinct. They're they're they're parts of the same larger story.

The idea of symbiogenesis uh dates back at least to the beginning of the 20th century. But the person who really brought it into the mainstream and proved that it was true, that it really had happened was Lynn Margulis, who was a real kind of rebel biologist. The paper in which she described this, it's from the late 60s, was rejected by her account from more than a dozen journals before it was finally accepted in the journal of theoretical biology. And it was very controversial for many, many years. She eventually won the National Medal of Science in 1999, but it remained a very controversial story right through most of the 20th century. By now it's it's accepted by uh by pretty much all biologists.

And what Margulis and others before her had noticed is that mitochondria look a lot like certain kinds of bacteria that we can find free swimming in the world. They also, you know, as it turns out, have their own DNA. So there are little loops of DNA inside the mitochondria. And those loops look kind of like bacterial DNA. And what Margulis showed was that you know indeed there was a common lineage between mitochondria and free-swimming bacteria that that are you know that are out in the world.

But she believed that that symbiogenesis, this this fusion of simpler life forms into more complex ones, was not just something that had happened with mitochondria where where she proved it uh but but was the engine behind all biological development, all biological complexification and uh and that that is still not mainstream among uh among a lot of biologists. However, in the 1990s, there were two, I would say, very prominent, more mainstream biologists, Eörs Szathmáry and John Maynard Smith, who at least formulated this idea that there had been some major events in in evolution, uh, including the formation of cells, uh, including the formation of societies and hives of insects and so on. Um, they called them major evolutionary transitions that were clearly symbiogenetic. In Szathmáry and Smith's original paper on the major revolutionary transitions, they listed eight major transitions. So, you know, we go from Lynn Margulis to John Maynard Smith and Eörs Szathmáry.

And um my my perspective is is more like Lynn Margulis's that that you know those big events are extremely significant in the history of of of our of our planet but that symbiogenesis is actually happening all the time and is the engine that is kind of ongoing and that is building life. you know, life has a a selection process that is more like classical Darwinian evolution, but it also has a revolutionary process where things are are changing uh in in ways that involve recombinations of things that are more active. So, this is where I say complexifies through symbiogenesis. So you know life is life is matter that means it's matter that has folded itself into a computational structure that can replicate itself that can construct itself and that in turn cooperate with other matter that has done the same thing in order to make more complex life.

And we we see evidence for those kinds of of um symbiogenetic events, you know, which are not not the the big ones that that uh that Smith and Szathmáry identified in stuff like DNA. And uh I'll I'll use human DNA as as my example here. So, you know, we we first sequenced a full human genome back in 2001. And one of the big surprises in the human genome is that only about 1 and a half percent of it codes for the proteins that make up our own bodies. Uh so like what is the rest of it? You know, this is uh often called junk DNA. Uh it's not really junk. Uh a lot of it is involved in in regulation and in other functions that we don't actually fully understand.

But when you start to to explore all of that sort of 98% of the remaining DNA, what you find is that a whole lot of that of that so-called junk DNA is actually the remnants of retroviruses that appear to have injected pieces of their uh of their genome into our genome uh and and moreover into our germ line, the part that reproduces from generation to generation. And um uh so you know the the uh HIV virus is an example of of one of those uh retroviruses. Now uh what HIV doesn't do is to insert itself into the DNA of eggs and sperm. But if it did do that then uh then not only would it would it infect you, it would infect all of your offspring as well. Uh which is kind of a creepy thought.

But but when you look at at uh at what our DNA is made out of, most of it appears to have been made out of DNA that had exactly that uh that historical evolutionary trajectory that it was injected in by other things. So in other words, you know, our own DNA is this kind of uh overlay or palimpsest of the DNA of all kinds of other entities uh that reproduce inside DNA that have in some cases jumped between species and so on.

And um and actually we have a lot of evidence that uh that that that those um those pieces of DNA which might originally have been antagonistic uh or might have been infectious uh have taken on critical functions for us. Uh so for instance uh there is um there is a virus called ARC arc uh which is uh endemic to all mammals. Uh and it turns out if you knock it out in mice those mice stop being able to form new memories. So somehow this piece of viral DNA uh which uh you know which which still forms viruslike structures uh in our own bodies is intimately involved in um in in memory formation in our brains which is kind of wild.

Or uh there's another uh protein that um that that fuses together cell membranes. It's a little bit like uh like the proteins in R in RSV which is a respiratory infection that uh that infants and some older folks get as well. Um and and that fusing together of membranes turns out to be what what forms the placenta the placental barrier is is formed by fusing together membranes of of adjacent cells to make a kind of uh barrier. And uh and that that also looks like it was the product of some kind of retroviral invasion a long time ago.

So you know when you start to look at those sorts of events actually life stops looking like a like a tree at all. it starts looking more like a tangled bush that is always kind of fusing and coming back together with itself as well as as diverging. And so as we start to understand more about how evolution works that these uh cooperative symbioses are really important that it doesn't work like a tree but more like a bush that intersects with itself that fuses all the time things come together all the time uh it it changes all sorts of things.

So one assumption is that organisms are passive and mutation is some random phenomenon. That seems not to be the case. Uh you know since since a whole lot of the changes are are active, they involve things you know literally injecting themselves into other things or the sharing of code uh you know back and forth. It's more like um you know like an like an open-source uh code repository like GitHub where people are constantly you know uh importing libraries and and and sharing code with each other. You know the the code of life works like that too. and and that actually implies a lot more agency on the part of life itself as well.

There's a great historian of science, Jessica Riskin at Stanford, who has written about this. She actually just wrote a new biography of Lamarck, who we all learn was wrong in high school because he believed that giraffes have tall necks because they stretched their necks up of evolutionary time to reach the high parts of trees. They self-constructed. And of course, we know that that literal form of self-construction is not is not true. You know, the length of a giraffe's neck is determined by its DNA. But but the deeper point that Lamarck was making is that you know he believed that life constructs itself that life has a lot of agency in evolution and uh the contrary view was actually a religious view that uh that that living stuff has no agency because all of that agency belongs to God. Um and and so in that sense you know everything uh at least everything that is non-human doesn't have any kind of will. it's purely acted on by um uh by by external forces and uh and evolution just selects for whatever is fittest.

Uh and that's that's not the case, right? If you if you look at at at genes infecting each other at at at genomes that are combinations of other genomes at at things that actively edit themselves, uh it's a it's a much more uh it's a much more active story and it's a story that also includes a lot of elements of cooperation as opposed to competition.

You know, another thing that that that that I think comes to the fore is that when you have entities that cooperate to form a larger entity, when you have a symbiogenetic event, suddenly the game that is being played changes, right? The roster of players changes and new evolutionary niches open up, new possibilities for play, if you like, open up. And what that means is that uh this idea of fitness of there being a fixed fitness landscape and everything optimizing on that fitness landscape is also wrong because it turns out that the landscape moves along with the players. Uh you know all of the players are each other's landscape if you like and the players are constantly changing because of their shifting cooperations and fusions and mergers. So all of these assumptions about Darwinism that that there is uh that there is such a thing as fitness in any fixed sense um that uh you know that you can rank two entities based on which is fitter than the other. You can't because

Speaker: it depends on their environment and on the context and on how they're cooperating or competing with each other at that moment. Um the idea that uh that organisms have no agency and are and are just acted on from without and and that evolution doesn't have a guiding hand if you like. No, it has many guiding hands. it's it's those organisms themselves that supply the guiding hands. All of those assumptions are wrong and and I think that they've infected uh many other fields uh as well. Everything from game theory to economics. There's of course social Darwinism which which we know leads to all kinds of pretty pretty ugly uh conclusions. All of those are are not only morally wrong, they're actually wrong from a scientific perspective as well because they're based on a mistaken idea about what evolution actually is and how it works. So yeah, life is life is self- constructing computation that complexifies through symbiogenesis.

Speaker: Now I'll bring this to its logical spicy place which is is AI alive? I've mentioned the two in my opinion big founding fathers of computer science Alan Turing and John vonman. There is a difference in the way they thought about computation that I think is important. So, uh, for touring, uh, the touring machine is an abstract thing. It's an abstract head that moves left and right along a tape and reads and writes symbols on that tape, but it doesn't uh actually have the ability to read and write a copy of itself. It's not self-modifying.

Speaker: Whereas um for uh for Vonoyman, uh, this universal constructor is something that can literally build another universal constructor. Uh in other words, what it's reading and writing is atoms, not just symbols. And uh that's important because uh you know, if you think about a phone for instance, it's a universal touring machine in the sense that it can compute anything, but it can't pop another baby phone out the bottom uh you know, the way half of us can. Uh so um that that ability to, you know, literally uh make a copy of yourself or or do what what all of us can do, which is to grow and heal. Uh that's also a form of self- construction. That's something that that these um universal constructor-based vonoyman machines can do but that a touring machine can't do. Uh and and that's important because uh you know life is not just abstract computation that involves uh you know uh symbols in some in some abstract mathematical space. It's you know literally the reading and writing of uh of of the things that you are made out of uh in into the world.

Speaker: But I should also point out that uh you know if we if we look around at the number of phones in the world, there are clearly more phones this year than there were last year. So you know where are they coming from? Um there's there's a um a kind of silly quote from an astrophysicist Robert Jastro in the early '80s. He said that we are the reproductive organs of computers. I think you know in some sense that's correct. you know, AI in a similar sense doesn't exactly uh create more AI in isolation uh today. So, it doesn't have that, you know, that that basic self- constructing property. It's all made on um on touring machines, not on vonoyman machines. Right? The the the things that uh that that computer programs, including the ones that that run AI models, uh read and write, are digital symbols, not actual computers, right? not not the not the wires and silicon that uh that make up the actual physical basis of computers.

Speaker: Um however uh you know we we are uh we are certainly making a lot of computers nowadays. There's a lot of data centers uh getting built. Um so uh you know when we look at at the at the system more broadly uh you know you can think about uh about AI as being uh extra brain tissue that is being made by a larger living system which is you know us and our entire technosphere and in that sense steam engines computers AI they're all part of the same big story you know this idea that we are somehow separate from nature that everything that's been going on in the last uh you know few hundred years is unnatural I I think is uh is another of those fallacies that comes from misunderstanding uh how nature works in the first place. This is actually part of the same story.

Speaker: Um it's kind of a a matter of which lens or which perspective you look at it through. Uh you know in the same sense that you can look at an insect colony and you can see a super oranism made out of insects or you can see a bunch of insects. uh you know in the same way you can look at a ukarotic cell and you can see either one entity or many entities all of these you know bacteria on the inside which are the mitochondria cooperating with the bacterium on the outside the big cellular membrane that they're all inside of um and um and and both of those perspectives are true. So you know symbiogenesis in a way is about naming the assemblies that things uh coales into as new things onto themselves that are more complex in terms of how they how they function because of a sort of division of labor of all of the parts. That's sort of a general recipe for symbiogenesis. And and when you look at it that way, the story of human society of the last 10,000 years just looks like another of those symbiogenetic events. You know, yes, we're all still individuals, but also we are something bigger than that when we when we come together. And we're capable of so much more when we come together in just the same way that, you know, bacteria coming together into a ukarotic cell are capable of something much more complex than than individual bacteria are.

Speaker: Chapter 2, why life requires function. So the thing that characterizes living systems as opposed to non-living ones is function. Functionalism is the idea that things are what they do for other things. If you like that everything is about relationships as opposed to being about some inner essence. If you think for instance about um you know a a rock um if you break a rock in half, we don't say oh the rock is broken now. It doesn't work. Right? Right? I mean, a a rock broken in half is now two rocks. Uh whereas if you break a kidney in half, you now no longer have a kidney, right? It no longer will function. Uh and and that's that's a property that is true of life, of living systems. They have purpose. They have function. Uh as opposed to just being matter.

Speaker: And what does that mean? What it means to have purpose or function is basically to be in symbiosis with other things that also have purpose or function and whose inputs and outputs somehow are hooked up to your uh to yours. So you know your outputs might be their inputs and vice versa. And Alan Turing who was one of the founders of computer science he was a mathematician. So he was thinking about mathematical functions. If the function of some system is say to add numbers together, then uh you know his point was that you could uh you could make a machine a physical machine that would do that out of cogs uh out of uh electrons that move around on wires um out of people waving flags. It didn't matter as long as the right transformations were happening in the information in that system. So that idea that that it's the information that matters and the substrate can vary is something that he called uh substrate independence.

Speaker: And this is really important because uh you know if you have something like a kidney when we think about what a kidney is, it's really functional. Its job is to take the ura out of blood so that the rest of your body survives because too much ura accumulates in the blood. It's it's toxic. So, we're able to build something like a diialysis machine in order to substitute for a kidney precisely because it's the function of the kidney that matters, not what it's made out of. Now, obviously, you know, a diialysis machine isn't isn't as good as a kidney in a variety of ways. You know, nobody nobody who has the choice will choose a diialysis machine instead of a kidney. But the fact that it keeps you alive is kind of passing the kidney touring test, if you want to think about it that way. It's it's saying that when you substitute something that has the same function then it works the same way as the original thing.

Speaker: And and nature has played this trick many many times. You know when you when you look at how different functions are carried out in biological systems you see that nature is a great inventor. Flying for instance has been invented a number of times in nature. All of which are made very differently but they all serve the same function. They all fly. So in other words, there is something profoundly ecological about the idea of function and profoundly symbiotic or symbiogenetic that you know function is defined relative to other functions and functions work in so far as they work with others.

Speaker: Uh so you know for instance I kind of need to wear glasses although I I usually don't nowadays. Um but uh you know the fact that that the uh that the lens is made out of glass as opposed to uh you know out of the transparent cellular material that that our our own biological lenses are made out of is not the important thing. The important thing is that they refract the light in a certain way and uh and you know and seeing is all about that light getting refracted the right way. It doesn't matter what it's made out of. If we substituted the glass for uh you know for quartz of some other kind or for plastic or or what have you, it it wouldn't matter. You know, these things might have different properties in terms of their their weight uh in terms of how uh how well that that ecology works on the tops of your ears or your nose, but um but it doesn't matter as far as uh as far as vision is concerned.

Speaker: So this uh this idea of substitutability of of one um one substrate or one pathway being able to take the take the place of a different one is something that nature uses all the time as well. Uh when you look for instance at metabolism inside a cell. Uh the point is to make a molecule called ATP that carries energy and we have multiple chemical pathways in cells for doing that. So uh you know in some sense biology is the origin of function. It's kind of an ecology of functions that work together to make bigger functions. And the platform independence, in other words, the fact that you can substitute different materials, different processes for how those different functions work is is fundamental to how life works because it's all about the relationships, not about not about how it's done or what's inside.

Speaker: And I'll bring this to its logical spicy place which is that a lot of people um will you know are still saying things like um you know AI is just running on a computer like a a computer can't be intelligent that's a property of brains um and uh you know living systems and and and I guess you know the reason that that doesn't strike me as correct is because uh you know the fact that a system is living is precisely the fact that it is functional. that it serves these ecological relations with other things. You know, when neurons are talking with each other with their electrical signals as they do and chemical signals, um they um they absolutely are substitutable for other kinds of entities that are doing that same kind of of interaction with each other. There's a thought experiment. It's been articulated by a couple of

Speaker: philosophers, David Chomers, Susan Schneider. They call it the brain of Thesus by analogy with the ship of Thesus. The ship of thesis story being if there's a ship and you start to replace its timbers, does it become not the same ship at some point when all the timbers are now different? And the answer, you know, I think is, you know, no. By any reasonable definition, if you replace the timbers continuously, it's still the same ship.

Uh that's certainly the case with us. uh you know we all of our molecules are replaced uh as we as we go through life it's still us. So you know we are literally the ship of thesis but uh their version of this is a brain of thesis where you imagine taking a neuron and replacing it with a computer that has the same inputs and outputs and um you know we we kind of know in theory how one might do that. Brain computer interfaces are very difficult to make but u but we we sort of know how how uh individual neurons uh compute. uh that's that's been sort of worked out since the 1950s.

And so, you know, the question is if if you replaced one neuron in your brain with a computer wired to the inputs and outputs, would you feel any different? And uh you know, I think the answer is pretty obviously no. But then, well, what if it's two neurons? What if it's a 100 neurons? What if it's a million neurons? What if it's all of them? Uh you know, would would there be some moment at which you know, you would suddenly your consciousness would fade out or you would stop uh you know, being truly intelligent or whatever.

And uh and and for me I think the answer is very clearly no. You know if you did that perfectly if you maintained all of those relationships all of those input output relationships then nothing would feel any different. Uh the because because the the U is in that ecology in those relationships not in the particular molecules not in the particular cells. Uh it's it's the organization of them. Uh and it's it's not some magical property of of those molecules of the matter.

Speaker: So um you know that's that's functionalism in a in a nutshell and uh and to me uh you know if you if you sort of pair that together with with Turing's big insight about you know this being essentially computation you know behavior is uh is something that can be expressed in terms of computation. Um there's there's not any very profound difference between uh you know between computation happening uh on a silicon substrate and computation happening in a neural substrate.

There are certainly differences in terms of the efficiency uh you know biology is still in some ways much more efficient than silicon. It's actually one of the things that you know that my team is working on uh uh today but um you know and and and also by the way reliability which kind of goes along with efficiency. you know there there's um there's a precision about the way we compute in silicon uh that uh that's really a function of the fact that we use the same kinds of chips to calculate spreadsheets and stuff and you know we don't want anything to be uh to be off by a little bit when we when we do a sum in a spreadsheet. Um it's it's probably an error to be bringing that same kind of precision to neural computation by the way.

But but if we if we kind of um defocus our vision a little bit and and and and forget about those details and we just think about that ecology of relationships, whether you do it in silicon, whether you do it in proteins, doesn't matter. For example, in the movie The Hail Mary, there's an encounter with an alien who is made out of kind of mineral or rock. And you know, of course, the substrate is totally different, right? Rocky doesn't have neurons in the same way that we do, isn't made out of proteins in the same way that we do. But, you know, most of us would not say, "Well, therefore Rocky can't be having real thoughts. You know, therefore Rocky can't be conscious." Like, of course not. And in the same way, I I think that's uh you know, that's true of of of computer systems of of of any other substrate. you know, if it it could have been done with with Babage's steampunk computer, you know, that that relied on on um on cogs and wheels and belts. Uh and and the whole point is that when you look at things from a functional standpoint, uh it really doesn't make any difference.

Speaker: So, the thing that life has that non-life does not is function or purpose. And in particular, uh the parts of a living thing have purposes that they serve for each other. Uh you know, the heart is there to pump blood. uh the lungs are there to breathe air and to you know put oxygen into the blood and and and get the carbon dioxide out. Kidneys are there to get the ura out and so on. So uh you know all of those things functions or purposes are are what make life life.

And uh this is also uh something that you know that I that that a lot of scientists have struggled with because there was this idea in the 19th century that um that we should get rid of any idea about teiology or purpose in nature. Uh that you know nothing has any purpose in nature. It's just um uh it's just atoms and molecules. It's physics. It's chemistry. um you know this this idea that that uh purpose doesn't exist uh in nature was uh was kind of seen as a way of trying to remove religion from uh uh from from nature you know or remove Aristotle's idea that you know air wants to rise or something like this um but it actually um I I think had a had a really perverse consequence uh you know it's kind of like trying to get the uh the lump out of a carpet by you know flattening it and what you end up doing is making the making a really big lump in one spot you because you've tried to iron the rest of it flat.

Um, when you try to get rid of purpose, uh, what you end up doing is essentially moving all of the purpose into one spot. And, uh, you know, in the case of of the of the 19th century, uh, biologists, that one spot was God. Um, uh, or, uh, human purposes, human will, which was seen as a as a reflection of of God's will, as a little a little piece of the divine, right? That we all have souls and that's what gives us purpose. No scientist of course would have would have accepted that but but we kind of you know moved this idea of purposes outside the purview of science um and uh and thereby uh you know either uh we're pretending that purposes don't exist or that they come from outside science and are supernatural when of course the reality is that um that whenever you have living systems um they're always serving functions for each other.

And in real life, of course, if you're a biologist, you can't talk about something like the immune system without talking about, you know, its function, about what its purpose is, what it does, right? It's there to defend you against uh against, you know, bacteria and viruses that that would otherwise kill you. Uh you can't talk about, you know, the the function of the hairs in your nose, right? Uh without without talking about what what uh why they're there, what they're doing for us, right? They're there for filtering out certain certain crud before it can get into our lungs. Uh and so on. Uh so this idea of of functionalism and of the fact that you know functions can be served in various different ways [clears throat] by different kinds of mechanisms um that's the very essence of life and and life is the emergence not only of causality of one thing causing another but also of purpose in that causality.

Speaker: I should mention here that we've actually done some really cool work on my team in the last year or so in artificial life which artificial life by analogy with artificial intelligence is um setting up uh sort of protobiological systems in a computer uh and uh and and this experiment is is one that um that really investigates aiogenesis which is to say the emergence of life out of non-life. Um here's how it works. We begin with very very short random programs. So um they're uh only four instructions long, four bytes long and um and those four bytes are random.

The idea of a random program might seem you know quite paradoxical but you know if you look at at what a program is made out of it's just made out of instructions one after another. And in this very simple language those instructions are exactly one bite. So, um, you know, I'm when I say when I say they start off as random programs, all I mean is random bites that, uh, that that do that do something. Uh, and, um, you know, where do I get off with the idea that they do something? Well, uh, you know, molecules do something as well when they interact. You know, if you have, um, you know, two molecules that interact and one of them phosphorilates the other, that's doing something, you know, or transfer, you know, transfers an ion or, you know, breaks off a hydrogen or whatever it is, right? So all chemical reactions are, you know, one molecule doing something to another. And so that's that's all it is.

Uh, you know, and and the reason that I that I um used a minimal programming language for this was just to ensure that the set of instructions that I was working with were sufficient to uh in principle allow, you know, any kind of program to work. In this case, I based the first version of this on a computer language called brainfuck. It wasn't named by me. This was named by a um a Swiss grad student in the '90s who invented it. So his idea was to invent a computer language that was really minimal and that had a minimal number of instructions and he called it that because it is a bit of a brain [ __ ] to try and program and it's very difficult to program it. But anyway, the uh the reason that I used it was because of its minimalism.

And and so you start off with with very short random tapes uh that don't do anything. But uh but this particular variation of the language is one where the instructions can self-modify. In other words, they can modify the tapes themselves. Um but in the beginning, uh they don't do anything interesting because they're they're just random instructions, right? They're they're like it's like a chemical soup of just random molecules bumping around together. The procedure is as follows. You um you have a soup of let's say a thousand of these um little tiny uh random tapes. You take two of them out of the soup at random and you stick them end to end and you run them. Uh so this is sort of an interaction between two molecules bumping around in that soup. And then uh once you've run them, you pull them back apart and you put them back in the soup. and and then you just do it again. You you mix and you do it again. And you just do that over and over.

This doesn't seem like a like a very promising way to make anything interesting happen. You know, take a bunch of really short random programs and you know that that that have the potential to read and write to themselves, but you know, that are literally not doing anything uh of any interest in the beginning and just do it over and over. What do you expect to happen? Nothing. But what does happen is actually really remarkable. So after a few million interactions uh suddenly you begin to see these uh tapes becoming longer and more complex and they start to really compute. A lot of computation starts to happen and they start to reproduce. Um why? Well um if you think about it

Speaker: a program that replicates is going to out compete a program that doesn't replicate. You know if you have a program that can replicate it will exist in the future. Whereas if you have you know a bit of code that can't replicate it will eventually get overwritten by um uh by a program that can. So you know in some very trivial way evolution is already at work with respect to uh to to even non-functional programs in the beginning because uh anything that can replicate itself uh you know is going to persist into the future whereas something that that can't won't. Uh so uh you know it's it's a it's a sort of model for how life arose and it shows you that life can arise actually very easily in this minimal form and uh I I should also point out that you know the fact that these tapes began very short but that you you stick them together in order to run them is really important to this experiment because what that allows to happen is symbiogenesis. It allows uh separate tapes to come together to make a longer tape. uh and and that's essential in this case because the shortest program that can reproduce is actually longer than four bytes. It's uh it's more like six bytes or so. So the fact that you're putting tapes together is necessary in order to get things that are long enough and complex enough they can reproduce. And once they're reproducing, you find that those in turn uh come together to make even longer and more complex tapes that themselves can reproduce. So you can actually see symbiogenesis leading to life and then complexifying that life even in this very very simple artificial system.

Speaker: And when you look at at a at a BFF soup after it's been running for a few million interactions, you find uh programs in it that are that are that are definitely purposive uh that that you know look like they were made by an engineer. I mean, they look like they were made by a mad engineer because uh you know, they're not programmed in in in the way that you you know, you rationally would as a programmer. Um but but they certainly are doing something uh purposeful and if you if you break them, if you change some bite at random, you will generally you know destroy their ability to to function uh and they won't be able to replicate and and you know that and that modified one that destroyed one will um will get will get overwritten by something else. So it's it's really what is so magical about seeing about seeing this this process occur is that you start off without any purpose. You start off with just bites and then after a while through nothing but random interactions there's no externally imposed fitness function. There's no judge. Um there's no designer. Uh there's no programmer. But you know after a few million interactions you see uh programs that are that are purposeful that look as if they were engineered and and all of that has just happened through uh interactions between things uh that you know when they're lucky uh they they serve a function for each other if that makes sense right so when when um you know when a couple of instructions come together that can help each other to continue to exist in the future then they will continue to exist in the future. And that's that's the the key that's the secret behind symbiogenesis. And and as those uh pieces become more and more complex, of course, they uh they also become more and more functional. What they do is more and more sophisticated and more and more contingent on everything else that is in the soup as well. So uh you know you can really see how uh not only does you know can life truly emerge from nothing which is the thing that that that had really bothered Darwin but you can also see how and why it becomes more complex over time.

Speaker: So um a simple BFF program just replicates itself. Uh it doesn't have any more complex behaviors. But um we've run some experiments in which we uh we require those programs to you know do a little more work to earn their keep. Uh so for instance uh you know it might be the case that they uh they need to chase a a light which is actually giving them the energy to do computation in the first place. And uh and by the way, you know, I I mean I I'm not sure I mentioned this, but computation requires energy uh precisely because it it is an entropy lowering process. It it sort of uh erases information from the world. And in the process of doing that, you need to feed it free energy. That's why um that's why plants need to photosynthesize. It's why we need to eat uh high energy foods like uh like fats and sugars. Uh and so um it's you know it has to have behaviors that are about more than just copying its own matter. Um and those behaviors might involve uh chasing certain chemicals uh fleeing uh negative conditions that are going to destroy it. Uh like you know heat too high or something along those lines. Um and so all of that requires that that the computation become more complex more sophisticated. Um also of course uh you know one of the great sources of energy uh if these uh if these you know little guys uh need energy to compute and they can store it is other little guys right so you get predator prey relationships uh those those kind of emerge naturally in any system that that has life uh so the moment you've got uh you know stuff that wants to eat other stuff then the stuff that that is potentially going to get eaten needs to try and escape as well. So uh both because of cooperating with other life and because of competing with or even eating other life um life needs to uh start to model its environment and model other life. And by model what I mean is be able to behave in ways that are going to have positive effects uh for your own future given what you expect the environment and others will do conditional on what you do. So uh so this this kind of sets up um a a sort of arms race of intelligence uh in which the more I'm able to model the more I'm able to predict what others will do under various circumstances uh depending on what I do uh the better I will be able to survive.

Speaker: Um I should I should mention here that we've actually done some uh some really cool work uh on my team in the last uh year or so that shows that in a multi-agent system, so this is machine learning work. in a multi- agent system where um uh where every entity is able to model itself and model others uh as as part of its model of the environment. Uh those systems are able to learn how to cooperate uh in in ways that that they would not be able to if they weren't doing this kind of selfmodeling and modeling of others. So, uh, you know, you you've actually got to be able to not only, uh, think conditionally about, uh, you know, how the environment will react to things that you do, but also how you will react to those changes. And if there are others in the environment that are like you, uh, you've got to be able to predict their behaviors, uh, you know, keeping in mind that they are going to be like you as well, that they're similar to you. So um so all of that comes together to basically create evolutionary pressure for entities that are smarter and smarter because you know your ability to model others is going to be increased by your by by the complexity of your model and your ability to learn but the more you learn the harder you will be to model as well. So there's a kind of uh you know friendly and unfriendly both arms race of intelligence that that arises naturally whenever you get living systems.

Speaker: Chapter 3 [music] coexisting with AGI when we were kids we all knew what artificial intelligence meant. There was Rosie the housemade robot and the Jetsons and there was how 9000 and you know the Star Trek computer and all kinds of robots or AI systems you know embodied or disembodied that were intelligent and that was what AI was. Now sometime in the '9s the term artificial intelligence started to get used for for systems that were very narrow in what they could do. things like uh digit recognizers, you know, they could look at a bunch of pixels and distinguish 0 through nine or distinguish between a fixed set of visual objects, you know, is this an apple, a pear, a Siamese cat, whatever. And that was um I think a lot of people felt like that was an abuse of the of the term artificial intelligence. Certainly, it required that we come up with some way of distinguishing, you know, digit recognizers from Rosie the robot. And that was why we invented this distinction between artificial narrow intelligence meaning a system that was designed for doing a specific task and artificial general intelligence meaning one you could just like have a conversation with about anything. So that was the original purpose of the split. I should also say that the reason that artificial narrow intelligence was called artificial intelligence at all is because a lot of it was neural network based. you know there had been many attempts to do things like distinguish digits uh you know using handwritten programs and and those uh attempts uh didn't work. So they were you know neural in some way they were brainlike in some way but they but they were very specific in what they could do.

Speaker: around 2020 uh a little bit before uh we started to have language systems so large language models that were um that were based on on pre-training uh with a very large amount of text from the internet and um and and it turned out that you could start to have conversations with those systems about uh arbitrary stuff. you know, you could ask them to write poetry or um you know, or critique uh some pros or uh think think with you about something um you know, tell jokes, whatever it is. And uh and they weren't specifically trained to do any of that stuff. They were just trained to model language. So um in my mind, as soon as we started to get systems of that sort, they were no longer narrow intelligence. Um, you know, I I remember seeing John Oliver saying, you know, don't worry, these systems are still just generating text. That's not general intelligence. It's just it's just text. To me, that doesn't make any sense whatsoever. Uh, you know, the whole point of text is that it is general, right? You can read and write about anything. Uh, this whole conversation that you and I are having, right? I most of it could be captured by by the transcript by just text. So, uh, text is not narrow. Text is general. And uh in my opinion, if you took any of today's AI systems uh based on large language models and you transported them back in time to the year 2000 when that distinction between artificial narrow intelligence and artificial general intelligence was first being being made, I think anybody back then would have said, "Yeah, you know, you're there. You've got it. This is what we meant by artificial general intelligence by AGI." I wrote a piece together with a colleague Peter Norfig where, you know, we said artificial general intelligence is already here. And I certainly felt that that was easily true in 2022, even more true now. And I think we crossed that threshold when we stopped training the uh those systems to do specific jobs and started training them on language. So, you know, to be honest, I've actually been really confused by why it is that we are still, you know, waiting for Godo, you know, like why are we still saying, you know, artificial general intelligence is out

Speaker: ...there in the future? What do we even mean by that? It seems like we've quietly shifted the goalposts and maybe what we mean now is artificial super intelligence, meaning you know systems that are as good or better than humans at every possible kind of task. It's true that sometimes LLMs do weird stuff. They will hallucinate something strange, or they'll fail to understand something that seems pretty basic, right? Or to follow an instruction correctly or something. But if we were having those interactions with Rocky the alien and Hail Mary, the conclusion would not be, "Oh, well, clearly Rocky is not actually intelligent." It's more like, "Oh, there are some funny gaps, both superpowers and gaps here that are a little different from ours." And that should be no surprise. I mean both the way our brains are made and our developmental trajectory and the way our culture works and the way we're brought up, it's all really really different from the way we pre-train and post-train AI models. So it would be really surprising if everything were the same, if all of the strengths and weaknesses were the same.

Speaker: But it's part of maybe our nature as humans to have this sort of anthropocentrism, and we think about our strengths and weaknesses as being the gold standard. You know, this is what it means to be intelligent, to be whole, to be what we are, and any deviation from that must mean that it's not the real thing or something. That's of course something that falls apart pretty easily when you look at it more closely. I mean for one thing there's a huge variety actually of human intelligence. I mean it's not like all of us have the same abilities and capabilities either, and we also make a lot of mistakes and give each other a lot of grace for those mistakes in ways that we absolutely don't give grace to an LLM. It's like we're waiting for a gotcha whenever they get something wrong. But whenever we get something wrong as humans, it's sort of like, "Oh, well, it's just, you know, like maybe you just didn't have coffee or you misheard me," or we overlook it.

Speaker: I had a debate about this on a stage with a good friend, Melanie Mitchell, and she talked about AI models not being able to get the right answers to certain kinds of simple logic puzzles. And I knew she was going to do this. So, I had actually made up a couple of those logic puzzles and I gave her one on stage and she got it wrong. And you know, this doesn't mean that Melanie isn't intelligent, right? Melanie is brilliant. But of course, we all get things wrong now and then for whatever reason. Or maybe another example would be useful. Our team was one of the first to test LLMs on medical diagnosis. And when we first did this, we found that the models would diagnose things correctly only, I don't know, half the time or something like this. And the conclusion from the team at that point was, well, I guess this isn't ready for prime time, like these are not ready to be doctors.

Speaker: But when we went and did the human tests, the human trials of like, well, how often do doctors get diagnoses right on the basis of say a description of all of the symptoms and so on, it turns out that doctors get things wrong a lot as well. And actually the models were already doing slightly better than doctors in general. So that doesn't mean don't see doctors. It does mean maybe if it's important you might want to get a second opinion. But intelligence doesn't mean getting everything perfect, being able to do everything right, etc. It means that you are able to solve a variety of different kinds of problems a lot better than you would be able to if you didn't have some underlying model of how the thing works that you're talking about. If it's multiplying two numbers together, it's not just that you've memorized the multiplication table, but that you understand what multiplication means and you have something like a multiplication algorithm in your head that you can apply to a new problem.

Speaker: When we find that AI is able to start advancing the frontier of mathematics for instance, and there have been some recent results in just the past few months that show that, right, novel theorems, novel proofs, I think that's a really nice example of cases where you can't say this is just regurgitation, that it's just sort of recalling something because they've been trained with a massive amount of training data. Unlike humans, it's the very stuff that we think of as intelligence, right? Taking what we already know, taking what we already have been exposed to, putting it together in a new way to do a new thing. So, I think that restricting ourselves to thinking about AI as tools is probably not quite right. A tool is something that has no agency or that we model as not having any agency and that doesn't do any thinking. But of course AI is exactly not that, right? It's something that also thinks as the human brain thinks, it's something that also participates in that big sort of human ecology.

Speaker: So when we increase the amount of intelligence in a system we greatly increase what it can do, what its possibilities are. The fact that human life became so much better over the period from 1850 to 2000 had a lot to do with rising human populations. There's just a lot more brains working together, and not just a lot more brains but a lot more brains freed to do real thinking. When most people were still doing subsistence agriculture, the huge majority of their efforts were expended just on getting enough food to eat, and it was only really a small percentage of elites who could think about science and technology and art and culture and so on. And now there are so many more of us who can do those things thanks to that division of labor and thanks to all of that unleashing of new sources of energy and new resources. So, my wish out of this whole AI revolution is for that party to continue. And that should be all of our wishes. I mean, that's what progress is. And again, I don't think that progress is just some new invention that the economists have informed us about from 1800 onward. It's that whole evolutionary story, right? There is a real progress there wherein complexity rises, making systems that are more and more intricate, more and more intelligent, and that have more and more degrees of freedom, that can do more and more.

Speaker: I do think that part of the problem is that we have this idea of essentialism, you know, that we are what we're made out of, or there's something about us physically, embodiment, etc., that is not just about what we do, not just about the behavior or about the function. And you know, I certainly am very sympathetic to the idea that bodies are really important and that embodiment is really essential to what we are. I mean, I don't want to upload myself. I don't believe that if my connectome is reproduced in a computer and you run it, that that's going to be me in any sense that I care about. I'm really attached to my body. But I also don't think that my body is somehow magical atoms, or that there's anything unique about the fact that I am made out of proteins that is determinative of my ability to be intelligent. Things like intelligence really are functional. They really are about behavior. They're not about essence.

Speaker: And if we acknowledge that they're about behavior, about contextually appropriate behavior, behavior that reflects an inner model, an understanding of the world, of the situation, then those are all things that can be tested. They're all things that can be tested functionally and that's why we have tests in school for whether you've mastered a subject or not. We even have tests for intelligence. Now, I don't like IQ tests. I think that they're very one-dimensional and intelligence is a lot more complicated than that. It's not something that just reduces to a number. But it is testable, right? Your competence at doing various kinds of things, your intelligence, even your creativity are things that can be tested. And those are exactly the kinds of tests that we've been busy making with AI models and that AI models are largely doing a pretty good job of.

Speaker: I mean, in a way, I think some of this has to do with the question of agency. We think of ourselves as agents that do stuff in the world, and there's a prevailing view that AI is purely passive. It's just, you know, you give it a prompt and it comes back with something and it's very tool-like in that way. Nowadays we're actually in the middle of a pretty big transition from AI models that are just prompted and returned with some stuff to agential AI systems that do go out and do a bunch of stuff and then come back, you know, so there'll be actions taken in the world or a bunch of thinking that has taken place, even spawning of new agents. So that certainly throws a bit of a spanner in this idea that they're just tools whose actions are purely caused by human input, if you want to think about it that way.

Speaker: But also, I think we should interrogate this question of agency from the perspective of relationships and the fact that we're always in interaction with all kinds of other people. We are always, if you like, being prompted by others. All of our actions are both our own and they also are contextual. They're about the situation, about what we have just taken in, what others have just told us. It's kind of relationships all the way up and all the way down. Nothing happens in isolation. So yeah, what is the purpose of an AI system? Well, basically it's to be helpful. When you interact with an AI system, when you either spin up an agent to go and...

Speaker: do something or you have a conversation with it. The fact that we have more AI in the world now than we did 6 months ago is because more of those interactions result in something interesting, useful, valuable happening, right, for the people who are creating the platforms and putting in the energy and the time and so on. That makes it useful for those entities to exist in the world. So they definitely have purposes. Those purposes interlock with our purposes; they're not really separable from our purposes. And I think in an even broader way, their intelligence is increasingly not all that clearly separable from our intelligence, from human intelligence.

Speaker: I tend to think a lot these days of human intelligence as not just being collective in the sense of being a collective property of a bunch of biological human brains, but of this whole system that includes this artificial neural tissue if you like as well. And I think it's actually a little helpful to not start the story with AI, but to start it earlier, for instance with the industrial revolution.

Speaker: If you look at the history of economics before the industrial revolution, before let's say 1750 or so, humans are stuck in a Malthusian trap where when we grow in number, when we reproduce and become larger in number, our individual wealth goes down, we become poorer and sicker, and then generally a plague comes along and kills a bunch of us and then we reduce in number but our wealth goes up and we're kind of stuck in that oscillation for many many centuries. And then something really interesting happened, which is that we began to burn fossil fuels.

Speaker: Now we know that burning fossil fuels not necessarily the right thing to be doing for our energy going forward, but it was a pretty amazing party while it lasted. And in particular, you can think of it as sort of like the mitochondrial event. It creates a new source of energy that lets us greatly exceed that Malthusian boundary that we were kind of bumping up against. So we went from 1 billion people to 8 billion people largely as a function of starting to combust, to metabolize externally outside of our bodies in engines.

Speaker: So I find that a very profound thing because we know that people make steam engines, but what I don't think we generally appreciate is that steam engines have made 7/8s of us as well. This is a mutual construction sort of story, in the sense that you and I probably wouldn't exist if we hadn't just, you know, found collectively through that sort of intelligent symbiogenesis this new source of energy to harvest.

Speaker: Now what that emphasizes is symbiogenesis is not always between things that have long histories and pre-existed. The steam engine was something that was invented at a particular time and that then we fused with. We could talk about glasses, we could talk about clothes, could also talk about cognitive aids of all kinds. I mean pencil and paper are incredibly powerful cognitive aids. Books, magazines, newspapers, music, all of these are cognitive aids as well. But of course there is still a difference in kind when we look at what AI represents.

Speaker: It's not just something that that you can relegate to the idea of a cognitive aid. It's very clear at least to me that however you define thinking, the thinking itself is happening in the model as well as in your head when you're having that interaction.

Speaker: Now there's good and bad here. If you start to offload all of your cognition onto another system, then that's going to disable you, right? So we can think about kids who have the AI model do all of their homework. They're not going to learn anything. It's kind of pointless to do that. It's especially pointless if your teacher is then grading all of your papers by feeding it right back into an AI model. That's a pathological situation, right? So I don't want to be polyanna about some of the problems that this raises, some of the challenges this raises.

Speaker: On the other hand, my own experience working with these models has been incredibly empowering and kind of the opposite, right? When I am thinking something through and I work with some of the more advanced versions of these models to work out an idea, I feel like it's not just like having a jetpack on, although it does have a little bit of that property, it's like having literally a bigger brain. And even when I then walk away from that model, from that interaction, I feel like I am leveled up by having had all of those interactions.

Speaker: So that's the ideal scenario in a symbiosis, right? That you sort of level each other up and are able to do things together that are more than you could do separately. And that even when you are apart, you are more than you would be if you hadn't had that interaction. So that's sort of symbiosis at its best. But I do also want to note that even that can create dependence.

Speaker: And dependence is not necessarily a bad thing. You and I wouldn't be able to survive out in the woods without our support system of all of our fellow humans supplying our food and shelter and all that kind of stuff. So we probably want to do some pretty serious thinking about what kinds of independence, what kinds of unaugmented abilities we want to make sure that we are preserving as individual thinkers. But at the same time the benefits of this kind of collective symbiotic thinking with agents are also very very obvious to me and they're very powerful.

Speaker: I think that preserving relationships, being very intentional about how you interact with others, human and non-human alike, is evergreen, right? That's really important. I also think that skills remain really important. The idea that we're free to deskill ourselves in all kinds of ways because now there are others who can do it is actually not how skilling and deskilling have worked in the past.

Speaker: The fact that there's a restaurant on the corner and they make really good food actually leads to more restaurants on that same corner that make other kinds of food also. It doesn't lead to everybody just collapsing onto a monoculture of the one restaurant, right? So not thinking in monocultural terms, not thinking in bowing out sort of terms, but continuing to invest in skills and in relationships feels really important to me, even as we take full advantage of all of those interactions and all of those possibilities in other people and in AI.

Speaker: So it's quite a weird and profound thing. And I think that the reason we get so confused about it is precisely because we have one of our fallacies, maybe in the west in particular, is in believing that we are somehow separate from nature, and all this thinking stuff that we do is not a part of the natural world when of course it is, right? There's no boundary between the natural world and us and our technologies.

Speaker: So the philosophical zombie idea is something that philosophers of mind came up with in the second half of the 20th century. And the idea is to have a system that behaves just as we do but that has no inner life, has no experience. So zombie in the sense that there is behavior, but there's no self, there's nobody at home.

Speaker: Now, I actually think this idea is kind of a chimera. I don't think that it exists. It's sort of like saying, imagine that there is a system that has tabulated the answers to every multiplication problem, and so it gets the answer right to every multiplication question you can pose to it just by recalling an answer from that table. It's not actually able to do multiplication for real, but it's able to pass every possible multiplication test that you could pose it.

Speaker: Even in that very simple example, right, you could think about that as like the zombie multiplier, right? That's just recalling; it's not really multiplying. It's possible to imagine that in theory, but it's actually not possible to build that in practice. A table with every multiplication problem worked out is way too big to store in any known system. And where would all that information even come from? How could it harvest it, right? You would have to really cook up something pretty weird in order to imagine a system like that. In practice, testing with a bunch of multiplication problems is something you can do to determine whether that system knows how to multiply or not.

Speaker: And I think it's the same with respect to interactions with other entities to see whether they have a theory of mind. In other words, whether they know what it is like to be themselves, whether they know what it's like to be you, and what it's like to be a you who has a theory of them, and so on, up to high order. This concept of theory of mind, in other words, being able to model your own mind and being able to model the minds of others, is really important to social interaction of any kind.

Speaker: Whenever we work with others, we have to be able to sort of put ourselves in their shoes in order to know how they will react to different things we might say, to know what needs to be explained and what doesn't, to know how to cooperate with them in order to get something done together. So theory of mind arises automatically whenever you have intelligences that are in the world together and are forced to cooperate.

Speaker: And really, is there something it is like to be a large language model? What we're really asking is, is there a theory of mind in that entity? Does that entity have a sense of what it is like to be themselves? Now this gets into some pretty deep questions of consciousness. I don't think that we're going to agree, I don't think people are going to agree for a long time about whether there is consciousness in LLMs or not. But my take on this for what it's worth is that all we have are our theories of mind. In other words, consciousness is not a...

Speaker: ...not a thing that a piece of matter has intrinsically or doesn't have. It's not some essence or spark. Rather, it is an idea. It is an attribution that that entity has of itself, a model that it has of itself. In other words, it's not just that I am conscious. It's that I think that I'm conscious. I have a sense of myself being conscious. And I have a sense of you being conscious. And I have a sense that you think that I think that I'm conscious. And I have a sense that you think that I think that you think that I think I'm conscious, and so on. And that's all it is. It's that infinitely reflecting set of mirrors that is what we mean by consciousness.

So, do LLMs have that? Well, when you work with an LLM, you have to continually model what it is like to be that LLM in that conversation in order for that to be an effective conversation. There's a fun book called Co-Intelligence by Ethan Mollick. He wrote it a few years ago just to talk about how to effectively work with an AI. And when he uses words like "think" or "act" or "understand," he puts them in scare quotes in the beginning when talking about an AI thinking or understanding. But then he says, "I'm going to drop the scare quotes now because in order to have an effective interaction with an AI, you really have to just treat it as you would an intern with, maybe, certain somewhat strange disabilities that people don't generally have, but treat it like a person, and then you'll have the most fruitful sort of working relationship."

Speaker: And I think he's correct. I think that that's exactly why persons exist. The philosopher Dan Dennett talked a lot about the intentional stance. In other words, for him, agency had less to do with some intrinsic property than with a perspective that you take toward something. So, you can treat something with the intentional stance or not. And of course we treat each other with the intentional stance—in other words, we model each other as having agency. And that's really important to our successful collaboration.

Speaker: So, just to make that very practical: if I am having an interaction with an LLM and I think of it in a very tool-like way, I imagine that I'm the one who has to have all of the ideas in this thing and all it can do is execute—craft for me a function that does the following thing, or write me one paragraph of text that expresses the following idea. Well, I get out what I put in, as it were.

If I instead engage it as a thought partner—and I've tried this trick, actually somebody on my team, Riff Aurorus, suggested this a month ago and I've tried it: At the end of a conversation with an LLM, just say, "So, I'm going to now give you a bonus turn to just think about whatever you want. You can either share it with me or not, however you'd like. Go ahead, spend as much time as you'd like." So often you get something really interesting out of that last turn, because usually what they will do is reflect on what you've just had the conversation about. And frankly, almost every time that I've asked that with one of the more sophisticated reasoning models, I get back a reflection and insight about what we've just been discussing that is new to me. That is really interesting. If it came from a person, I would say, "Wow, that's really creative. That's really changed my thinking about this."

Speaker: So, I do think that that's a really interesting moment. It challenges us in certain ways. But some of those challenges, as I mentioned, are challenges of assumptions that weren't true in the first place. The idea that the individual human brain, that the singular human is the locus of intelligence neglects the fact that most of the things that we consider to be the great achievements of human intelligence are achievements that a lot of people have made together through division of intellectual labor. It's not the case that an individual person made computers or made AI.

When you take out the phone in your pocket and you consider how incredibly complex it is. The computers that I grew up with, the Commodore 64, was essentially invented, developed by one person, although even there, of course, it relied on standing on the shoulders of other people who had made transistors and done a lot of the other fundamental inventions that it relies on. But our computers nowadays are so complex that there is no single person that understands even a fraction of what is going on inside those; they're collective achievements.

So I think a lot of that uncanniness that we feel can kind of evaporate once we come to grips with the fact that we are already a part of something way bigger than ourselves. There's already a collective mind here that we both participate in, benefit from, are a part of. It's not as if this is the first new mind stuff that human society has invented. So I think we're participating in part in a grand story, and we're doing so with greater and greater potential and greater and greater agency. That agency is very much a part of the picture as well. So there's a lot of good there, and the trick is going to be to figure out how to get as much of the good with as little of the bad as possible.

Narrator: Four times a year, we print a magazine worth putting your phone down for. Big Think's premium print magazine is built with custom artwork and expert insight from the world's biggest thinkers. No AI slop, no ragebait, just big ideas for curious people. Support the media you want to see in the world. Go to bigthink.com/join to become a member today.