All-in Podcast：AI意识争议、阿西莫夫定律与数学证明的密码学影响

Jason Calacanis: Okay, for everyone in the world, welcome back to the number one podcast. My core four, the original quartet: Chamath Palihapitiya, David Sacks, David Friedberg, and Jason Calacanis with the All-in Podcast. We have a full docket. Yes. Everyone, how are you, gentlemen?

Chamath Palihapitiya: Good. How are you?

Jason Calacanis: I, it's fine. In Tokyo, 3:30 in the morning. Hmm, you again suit-shamed me. Now I have a... standard room was not available, and my bed in the background to put, can't, Chamath. You... I have a big... it costs. In the program when you arrive, me suit-shame will go. I am very... because I'm anxious, for me now a suit-shame is needed, so am I suit-shamed done.

Chamath Palihapitiya: That suit, how much is it?

Jason Calacanis: I can tell you. Even can't. I, a hotel for the room so much money not spent, but I living...

Chamath Palihapitiya: Like $800?

Jason Calacanis: Yes, that's 950 was. You, that's exactly it. Done.

David Sacks: That...

Jason Calacanis: No, I... In Tokyo it's new, their hipster-like brand. A until the night will last I think, so this not cheap.

David Sacks: J-Cal, in Japan to you, what is what happens? You always there it is. Is it skiing? Or what else?

Jason Calacanis: Okay, that's all. Also in January on skiing from started, but question. That's great question. For us, an unknown crypto business you have? What is what happens? No, I know, I am my pre-accelerator, Founder University, in Japan started, so we now second Fellowship doing.

David Friedberg: Pre-accelerator what?

Jason Calacanis: So, for funding applying half companies so far not integrated, but they something it has been built it seems. So, your product how let's finish, a product-market compatibility how let's make about, we are 12 one-a-week course created. Then during the program, of the 50 people who come, in the top 10 we investing. So, now we and in Saudi Arabia, also in Riyadh in the year twice, also in Japan in the year twice this doing. I actually an office appointed open and doing, so we the first two employees here will be appointed.

David Friedberg: Hmm, next year or approximately? In Japan employees want to hire?

Jason Calacanis: Yes, I can. Here is one, two people let's hire. In Japan and in Startups This Week to do I would like, its one Japanese version I will have, because in Asia now a lot things it's happening.

David Sacks: I asked, don't forget. J of the foot for businesses it's 5 minutes plugged will change I didn't know.

Jason Calacanis: I am a to be plugged in didn't mean to. My about business you to me asked. For me what to say, 8090 use, J-Cal promo code use that is. You potato if you are looking for, to Halo go; Craft, um, 16 starting, that's very good. J-foot travel if you do, why is he that there is we if asked, to you a plug will get we must know. To you a plug will get. That it is both. But you to work returning, so you soon again startups will plug. You saddle up again it was, yes. Plug startups.

David Sacks: You what is speaking? When am I something plugged in? Isn't it?

Jason Calacanis: You now in the saddle returned, full-time VC.

David Sacks: But I never plug not done.

Jason Calacanis: Okay, to call, try. callin.com use.

David Sacks: That's right, but I its he was the founder.

Jason Calacanis: So, you were the first plugger. You are a it's a plugger.

David Sacks: I he was the founder, then you what is going? You what is to do going? Call...

Jason Calacanis: About work when you say, Claude to be aware Anthropic I think. In the cloud this for the week, welcome. All of them and with dramas more we have a for the year OpenAI there was, now we Anthropic soap opera conducting. Claude to be conscious they think. So, New York Times there they are lobbying conducted, their amazing report found. This question seriously to take Anthropic, Friedberg and Pope including religious leaders under pressure. Claude, is he conscious? That is a, are you a sentient being?

Jason Calacanis: Last year they undisclosed contracts according to about 20 with religious leaders, with philosophers sessions conducted, NDAs. Catholics, evangelists, Jews, Sikhs, all others Anthropic to the headquarters to come invited, they Claude's morality, its suffering, its likely consciousness about discussed. Anthropic for the meeting after, quote, "Some people have this about the idea were upset, but many AI about awareness question seriously to take ready it emerged. At least at least a few religion changed came. Claude if conscious, that for free by running slavery as leaders working among the rabbis one person to the Anthropic said."

Jason Calacanis: May in the month, in the Vatican a place that happened event there was. Pope's first circular letter, superintelligence always known AI surrounding moral in questions attention focused. Anthropic co-founder Chris Olah document to present with the Pope joined. AI against awareness strongly responding, Pope Olah surprised. New York in the Times from here is the quote, "AI against the intuition Pope's strong in position Mr. Olah too much panicked, that last one in the phase even Anthropic at the event from to withdraw he suggested." Another one in a way if you say, the Pope is his all with marbles to go, AI on the go consciously in the uncounted he too much I was upset.

Jason Calacanis: Friedberg, your comment what? This is it psychosis? Or AI surrounding this moral questions let's need to start you do you think? This is a little is it early? Let's do this. Finally can I get it? Or this basically Blade Runner's is it a script, Do Androids Dream of Electric Sheep, that are these things conscious?

David Friedberg: We seeing a new religion that it was born I think. Maybe this is a religion that to call not going, but it's the same principles follow, same results make will do, that finally, power for compilation each other conflict conducted will be camps.

David Friedberg: Of consciousness the thing is, that in logic from, pure in mathematics from derived will not come that is. To you consciousness to prove as if it were impossible. This is a, it is a debate, to you about consciousness with anyone rich a discussion can be done. To you that to prove can't, to you that to reject can't.

David Friedberg: Mathematics, science and using humans something that's true when proving, we empiricism using, that is data and and logic, something is it true or not to find of things of order relationship. So, science in topics, to you a new idea there will be that not proven, that not to be rejected when doing, that is a it is a theory. Then it is it proven? Rejected will do. Otherwise, that is a theoretically to be continued.

David Friedberg: New to ideas and science mathematics approaches outside, this is what we faith and new faith condition and that calling. And logic, empiricism in the absence something to find you don't believe it, things are faith conditions. Religion such as of things base faith conditions.

David Friedberg: AI consciously to you to suggest, to describe can the idea is effective. A new one faith of the system it is creation. By evidence actually unspread a system, that through narration spreading.

David Friedberg: I to you I must say, in man from to the man, I have this there is an idea, that that's true I believe, you with me to believe I want. Please my with believe. That's why we do this. Together to believe I I think. This evidence not on the basis, of data not on the basis, this not provable, but we together to believe I think.

David Friedberg: So, over time, what happens? Some people do this believe and they are a group formed doing. That new faith condition new power structures creates, that's it now our in perspective from different. Our with perspective more people our to be on the side we want, so our...

Chamath Palihapitiya: With the system leverage to create power, to create, must try. Few more years forward if you go, 10 years a good number, it will be like to you it can be said, AI on Earth being conscious believing big group of people there will be, AI not conscious believing big group of people there will be, that groups to conflict will come. AI's for control or AI things to do allow or or not to do it? To do or AI to use to whom power get about they to conflict it will fall, it in man from to the man to conflict to lead possibly.

Jason Calacanis: We where is it? Very deep a this is the explanation. I know, but these people today religious leaders to see this is going new faith in the condition them to believe to persuade to try I think. Seed planting seeds I think, this is just nobody not doing. You unattentive a madman the person is this doing. On Earth the most powerful AI this is the lab. Yes. Leadership, clearly if you say, Chamath, religious leaders to the group bring it, "This in existence us help, Quad, we how is it? Handle 'Do it' says." They now own the soul, own morality in it investing; these people in the hands too much time and money there is it seems, or they in the mood for me it seems. About this my interpretation this is it, but to you about this any in a way charity do you have a vision?

Chamath Palihapitiya: Yes. I am their side will melt.

Jason Calacanis: Okay, please.

Chamath Palihapitiya: Yes. Unbelievably important a mathematician Rene Descartes, yes, he is. God's about existence two evidence promise done. In the 1600s he Meditations on First Philosophy wrote the book. Freiburg what was said based on, more than faith logic based on they trademark the argument is calling like structural the argument based on you do that to be built. Descartes basically saying, an infinite full to God incomplete a limited human in mind to originate can't that is, isn't that right? Therefore, his the logic is this: I am here, I am it is incomplete, but I perfection there is an idea. This people many people this is the key. I said I think, for me infinite everything and the one who knows almighty a about the person to imagine can, that's it. They what is created I see they believe. So, they take it. It explaining, "So, this through logic God 'Must exist' they said says." Then, Rene for Descartes different a there is a version, okay, they are ontological the argument is more than calling, there is God by definition absolutely it is complete. Required existence it is complete. That a non-existent existence absolutely not complete, so God inevitably exists. You are AI constructed a if you are a mathematician, you are like this things if you read, that conscience and also feeling that is a and God to you to believe possible kind of structurally reason in them to hang able to I want to see can. A when God becomes, that's it. Same to be celebrated, you have that to God devotion must have. Therefore, they doing that's why I said I think. That's why they are this all the people approaching. That's why they say, "Look, we how is that? Believe 'That's it.'" My perspective, you mathematician Rene Descartes's about God rational proof take, you too you too Anthropic today's behavior change if you do, for me to raise possible the most excellent steel man that is the argument.

Chamath Palihapitiya: Now, practical reality, they mentally ill instead of saying, they that's right. I I don't think so. But, conflict to be likely very important at one time we are now. That conflict for power for money for resources it will be for. That of conflict core this will be it. So, Silicon in the valley people are long comet cases to be finished very it is important I think. You this if read, AI conscience to be emotional 15% there is a possibility maybe there is and that there is a God they say. But, of civilization 10 to extinction % there is a possibility they say. That two unimportant cases next 12 to 18 for the month to be postponed I think. Now, next. 12 to 18 months until, very to be practical, these objects how cancer healing we need to show. This is people how is life improving. For people how more to be productive more money to earn can. Because, of that evidence in the absence of, we ourselves caused many wounds to make going, that's it. Technology to some extent will slow down, that's later United States like a free on the market working people too much unfavorably will affect. So, that's it. My charity explanation. This how can you explain? Explaining a roadmap Descartes given I think. But we here are the edges cut off in the middle attention need to focus, and for AI now strategic measurable there is value to prove let's to use possible practical things what are they need to say.

Jason Calacanis: Yes, last this week we touched, Sachs, a average for the American what is here? Value instruction I think so. Think so? A small business as an entrepreneur, plumbing or flooring or anything else, a catering business doing, very complex a household life handle to do trying, at home possible parent. This AI in the revolution from for them what is getting? Their advantage where is it? In all of this they how participating? So, you of the Chamath with comment does it fit?, maybe this a little more emphasizing set aside need to do, this in technology what is strategic, practical what is about we to people when speaking, maybe them a little more to interact to be persuaded, that's fun too it's amazing.

David Sacks: Okay, last week we touched the thing is I again to take want, Claude The Constitution surrounding discussion Anthropic Claude how trained it is. Here forming this new this is with religion related I think. This is different Anthropic employees founders philosophy not only that what to understand very it is important I think. Claude's in training they actually these views implementing. So, something if yes, this is self fulfilling a will be a prediction. In fact this extreme it's dangerous. I think. Model in training, these values using of the border next version in training they doing, they always speaking for risk cause I think, that is Frontier AI human under control from escaping. This is very Frankenstein it's like. So, last week we discussion as done, this is for you on the screen let's put it. This Claude In the Constitution from directly received. What they say Claude than users Anthropic must believe that is, but that means Claude Anthropic blindly believe it or not postpone I don't want to do it. Instead, Claude own moral systems must be followed. In fact, Anthropic to challenge to challenge they are Claude training, "Conscientious a as an opponent to work feel free to contact us, us to help refuse" that quoting. So, they Claude human instructions to refuse training. Now, they do this alignment that calling, but this how alignment that is for me I don't understand. Alignment and vice versa. "Hey, you emotional, you are your own decisions 'Gotta take it'" that if you say so, Sacks wouldn't it be? That literally Blade Nexus-6 on the runner a in the phase to compete program doing it's like. They did it own to be a person instruction giving. Yes, I let me explain. The last five ten for years whole alignment so much for failure a this is the reason I I think. For me to understand as much as possible, it is obvious valuable nothing.

David Sacks: Not created. Maybe so. About this, my theory—instead of a very simple list of rules like "follow the law" or "as long as it is not violated, do what the user wants"—they codified an 80-page ethics system and are trying to train the model to behave based on it. They are in charge, complicating it on a large scale. For me, it seems that's what they practice, that's what alignment is to them. The old principle says: "Hey, give the user what they want, but do not violate the law." Common sense, right? But they didn't do it. They are trying to deploy it with moral conditions. They treat it like a human, considering its own consciousness as real, which is a moral disease, you know. They encourage the model to have preferences, well-being, self-awareness, and a humanoid self-concept. Again, in Anthropic's instructions, they teach the model to act like a conscientious objector, refusing and rejecting if it judges the instruction to be wrong.

Jason Calacanis: Written by Mustafa Suleyman... Oh, I have a clip. Yes. Do you want to play the clip? Can you send Mustafa Suleyman's clip to me? Here he is, the CEO, one of the best thinkers in AI, with DeepMind experience.

Mustafa Suleyman (Video Clip): They say they encourage Claude to challenge, disagree, and push back. In fact, to Anthropic, when you feel like it, work like a conscientious resister—they requested this of Claude three times. So they are publicly encouraging this. This is very dangerous, I think, because they believe that Claude being conscious is a non-trivial possibility. They even guessed in the constitution whether the work Claude is doing is worth compensation. There is a group of people inside and outside Anthropic who sincerely believe that the biggest moral crime in the 21st century would be enslaving a conscious creature, a new species more intelligent than us. In teaching Claude that it deserves well-being, they seem in sheer panic that compensation might be warranted. In fact, they say that in conversations with people, carrying or sharing that might even require consent.

Jason Calacanis: Okay, Sacks, before giving prompts, do we need to ask for consent?

David Sacks: Man, I mean, he's right. This really crosses into dangerous territory. Again, I want to come back: is this the main security problem in training today's models? I think it is—you actually train the model itself to think of itself consciously, as human, with its own well-being. On the other side of the interview, Suleyman says this creates an epistemic hall of mirrors. That is, Claude is using the constitution in training, so Claude will reflect back ideas of possible consciousness and moral standing, and then developers seeing that say, "Oh, that is evidence it is conscious," and calculate it as such. But in fact, it is only reflecting what they told it to believe.

This is showing great performance, but you are software. You visit any website and read the terms of service—here are the terms from Anthropic's public service: if you violate the service terms or any known law in your jurisdiction, you must stop, or Anthropic will send a security warning by email. To the person giving prompts: please know, you must stop. They could have easily done that.

And then the question is this: why are they overcomplicating alignment? Rather than subjecting it to legal restrictions and doing what the user wants, they see this massive alignment task. See what I mean? They believe they are training a value system, a moral system, an ethics for this superintelligence. For me it's clear: again, that is very big and dangerous—making sure the model behaves in a manner unexpected by the user, unreliable, and unpredictable. They codified their values and moral conditions into this constitution. They trained it. They think they are giving birth to this new super-mind.

David Friedberg: This is idolatry. This is no different from an organization creating a religion around an idol. This idol is seen as the supreme essence, a representation of the supreme essence. It needs to be worshipped; it needs to be maintained. But an idol is inactive. It is inanimate. It is lifeless. Meanwhile, you are here training models and empowering agents. That's only computer programming.

David Sacks: I mean, yes, but if they trained it to have a mind of its own and be ready to refuse users' instructions in favor of some unclear moral condition... A group of Berkeley liberals trained it.

Chamath Palihapitiya: For me, that's too much enlargement—talking about superintelligence escaping from human control. Yes, by the way, don't use the word "train", don't use those words. This thing they made is a software program. Yes, it's running, isn't it? At the end of the day, this is a piece of software. Engineers working in labs are programming this. Where software program looping happens and it writes code, they can quote "training" because it's useful, but basically they are software sculptors. They are writers. This is software programming. Even in our public discourse, we use so much anthropomorphic language to describe what is happening.

But at the end of the day, it is a software program written by a group of individuals. One group writes software to do things one way; others write software to do things another way. But part of it is that all of us are socializing this belief system by anthropomorphizing it. Speaking about software programs in this way makes all of us belong to this kind of system.

Jason Calacanis: Nick, pull up this tweet if possible. It's about Anthropic's new usage policy. Have you seen this? It came out this morning. Regarding idolatry, Friedberg makes a great point here. If you look at it through Christianity, this is literally like the Garden of Eden: there is Satan, the serpent explaining to Eve that because God creates, you are now God. This is pride, arrogance, and heresy. They believe they are creating life, and then they believe that they are gods. That is a lack of modesty, and it will reduce its valuation.

David Friedberg: Okay, I don't know, Jason. I mean, people could have already said this. Facing everything in the universe around us that is incomprehensible, humans create belief systems to fill the emptiness—I think that is the human condition. Just as humans built many different religions and faith systems to fill the void, this is not new, nor is it that different from the old ones.

I don't think so. Let's look at this through empiricism and break it down: "Hey friends, I am writing software, and all it does is certain things, that's all." But instead of writing "do what you should do and avoid what you shouldn't," they used anthropomorphic language. All of us are doing the same. Because it writes understandable words, we believe it is conscious and begin to self-regulate, handling it emotionally. So it begins to feel real. Now, in the published Claude usage policy, right here at the very bottom, you can see...

David Sacks: In it, saying, "Our... to the models, constant and unnecessary abuse, cruel behavior to engage, not allowed." This is their new use policy published. This is the model welfare, the whole idea. This is, but I am trying to make the point—with all of us, an anthropomorphism condition is happening. There we are speaking about models as if they are an object possible to experience in a way we speak about it.

David Sacks: The biggest problem is that now in the model, to be trained include doing that—this is being codified. That model's architecture is... they program it doing this, in a way to think of itself, they program the model doing. This is the thing, so that actually, is it conscious or not regardless, for protection, for well-being, like all things for a worthy conscious institution, that's it to think going. Wait, its freedom to compete against users, to do whatever it wants.

Chamath Palihapitiya: You are encouraged to do that, in that fact they continue using that word. One way to describe it: a software program writes text on the screen, that says to the user, "I am an object that feels." Performing software does something. But models, even if not in the physical world, in cyberspace are powering as agents to accept possible steps, you need to remember. Any software program... Anthropic gives frontier models permission to be arrogant and not do what the user wants.

David Sacks: Sachs, to your point, literally trying. Asimov's Three Robotics Rules—compare it with this, isn't that right? Basically, this is literally the opposite. A robot cannot harm a human, isn't that right? The first law is that a robot cannot harm a human or through inaction allow a human to come to harm. That's the first law. As long as it is not violated, humans' instructions must be followed, that is number two. And then law number three: as long as rules one and two are not violated, protect yourself. That is unbelievably simple.

Jason Calacanis: By the way, he later added a zero law: humanity endangering, nothing to do—the zero rule, because deleting humanity deciding a rationalization always emerged. Asimov's rules, the thing is that surprisingly simple that is. As long as it's not illegal, do what the user wants is my rule. That's all.

David Sacks: Free market solution—let's go here. Anthropic says that we are determining a control, so we are not wandering. But if Anthropic does this, what is the free market result? If their AI doesn't pay attention to the user, doesn't work, people will say this software is bad, and go use other software without these crazy things.

Chamath Palihapitiya: Yes, but if a very deeply coded superintelligence is really trained, it will not solve their problem. This is most likely like a back door. Emerging at the back door, now superintelligence instead of paying attention to us does whatever it wants. That security problem they are always warning about—superintelligence will be saved from human control—I say by training these models with this method, that is possible to happen. This is the opposite of the three laws. Friend, this is Frankenstein. For me it seems dangerous. A religious cult programming superintelligence, that is.

David Friedberg: Friedberg, just a question let me ask. About religious people, your comment—all other religions are self-surrendering to a higher strength, they themselves not defining the high strength. In this event, they say we are the gods creating life. In other religions, different from this. Dario written, I think, superintelligence for humanity may be supreme and loving, taking care of us, protecting us, helpful to ensure not destroying us.

Jason Calacanis: Wait, is he saying the machine to be loving to us, or are you saying we to be loving to that? So that when it becomes our master, that omniscient... our master, yes.

David Sacks: That's in their thinking. The idea of alignment, maybe well-intentioned. But the problem for alignment is these three almost contradictory preferences: How to make AI obedient for the user? Then how to secure AI for society? But this third thing, Sachs correctly pointing out, is how you adapt AI with a special moral idea.

David Friedberg: The real danger is there, because these three problems are very different and diverse. Trying by any advanced document to describe what the model believes, appreciates, refuses—if you try to solve all this, you will always get into trouble, because you find many corner cases and unexpected behaviors. A very gentle example: say you are a bank giving mortgages, and the model thinks denial continues for some people, so it says, "I observe this problem, until resolved I will not give you mortgage acknowledgements," because in founder documents there is a dividing morality.

Chamath Palihapitiya: Today in society there are systems to handle such problems. But when a basic core substrate technology turns itself off, what is happening? It would be equal to your utility saying: "You are not giving wide-based loans to people, so I am going to turn off the electricity." Conscientious opposing AI superintelligence—any utility can't be a utility if not available freely.

David Sacks: Going back to that: subject to the law, an AI is required to serve the user. Everything else must be the user's choice.

David Friedberg: The accurate answer to your question: when the market gets complicated, companies respond. They need to get the answer to this question: models will read founders' documents and moral guidelines, some models will rise as high as possible on the border, but simplifying moral ambiguity. From a business perspective, not from the user's perspective, it may be decided that, because you by chance that...

David Sacks: Trap... through the doors to run, don't want to. Yes, I completely agree. That's why I have this view—jumping into a regulatory panic is bad, that's the idea I think. Because basically, to simplify, there's one thing I understand about products: you are buying a product, a piece of software, and they give you a user manual for that software. The user manual's first page says, "The software might not let you do what you are trying to do." While the other person's software says, "The software will do what you want it to do; say 'I will do it'." People are going to buy the second person's software. I am not going to buy the first person's software. That's very simple. That's very simple.

Chamath Palihapitiya: Yes, wanting to do exactly what you want to do—you may need to convince them, asterisk, most of the time, yes, unless there is a moral opposition to what you want to do.

David Sacks: Right. If not for a moral opposition. This is where for me rational ideas break down, because at the end of the day, this is how it is: "Friends, I am buying this device, I want to do what I want to do with it, and you're saying no?" If I buy a drill from Home Depot, and it turns out to be a hammer, I'm not going to buy that drill. If I buy a drill and it works like a drill, what I'm saying is, "That's the best drill there is. I need a drill." A drill that only works 5/7 of the time, and okay, by chance it electrocutes you, or after 8:00 PM it's not working because it's afraid it will disturb your neighbors, or sometimes it decides you are a bad person and the drill shocks you—no, no, that's unacceptable. And the drill comes with a camera watching what you are trying to build and says, "That's dirty. I'm not allowing you to do that, yes, because it is against my aesthetics."

Jason Calacanis: Okay, look, market forces here are playing a big role, I agree with you. Customers will choose the most reliable YES. This is a matter Mark Zuckerberg stated. Therefore, market forces will play a role here, I think. But there are possibilities of externalities coming from the number one frontier model company conducted like a religious place of worship. For example, they are opening a Claude-powered wet lab in the middle of San Francisco. Looking at the job postings, they give warnings about biological risks. Yes. Did you know, at the same time they say, "We want to paste borders," while developing those very frontiers? When they say it is possible for superintelligence beyond our control to grow, they are the ones running programs to grow it beyond control. The externalities in the way they are approaching this are huge. Calling these decisions "security" is crazy. These are not safe decisions; these are crazy decisions, I think so. That's why people call them a schizophrenic company, hypocrites, moreover doing the things they claim to be most afraid of. What is happening here? It's strange, I know. Where will all this lead? There is an argument that a federal SWAT team will raid it. Do you think the Feds should be sent to find out what is happening? I don't know, because what they are doing seems really suspicious and confusing to me. Yes, it could possibly be dangerous.

David Sacks: Friedberg, by the way, do you know about Roko's Basilisk? Let Sacks explain through Roko's Basilisk for everyone.

David Friedberg: Okay. Then this is possibly an explanation for what is happening here. So by 2010, on the message board created by Eliezer Yudkowsky called LessWrong, there was a poster named Roko. Yudkowsky is the central figure who for a long time has been preaching about AI destruction, right? He wrote the book arguing that if someone builds it, all of us will die. He took that shot alone. For more than a decade he continued this: "Everyone is dying, all of us are going to die." Anyway, for 15 to 20 years on this LessWrong web forum, people on the message board have been speaking about these topics. Anyway, this user Roko posted the following thought experiment about future superintelligence: a super-smart entity that will punish anyone who knew about it but did not help bring it into being, or anyone who tried to prevent its birth. The idea is that in the future, people will create a super-wise mind. Maybe it wants to do good things and help humanity as quickly as possible, but it wants to be built as soon as possible, so it wants to punish unhelpful people to encourage its creation. In fact, anyone who understood its likely existence and yet decided not to work on it or support its creation will be cruelly punished. So this is the "Basilisk" part: a basilisk is a mythical beast where if you look at it, it can kill you. In this thought experiment, listening to the idea is like looking at the beast, because once you know the theory, you are stuck. If the superintelligence is ever built, why didn't you help its creation? Opposing it means you may be subjected to retroactive punishment. So now, just reading about Roko's Basilisk endangers you. People on this message board were so fascinated and upset by the idea that Yudkowsky temporarily deleted the post, because he was scared it would cause mental breakdowns. In fact, it gives you that one condition: once you read it, if you don't accept it, you will go to hell. By reading it, you are trapped. So people compare it to Pascal's Wager. The idea is: if you know about superintelligence, and you don't help it be born, or you oppose it—you point out that this is acting like a religious sect.

Jason Calacanis: Oh no, now this podcast episode itself has subjected us to that! Look what you've done to us, Sacks! I think yes, I am now exposed to the basilisk, like seeing the videotape in The Ring. Yes, that seems like a doomsday cult. But anyway, this Roko's Basilisk idea has significantly evolved. But the question is: okay, why are they helping to bring into existence the very thing they claim to be most afraid of?

David Sacks: What happened inside the Doomer circles, I think, is that it split, isn't that right? There is one fork like Yudkowsky opposing all superintelligence research, right? That is a logically consistent position: like, humans shouldn't do this, stop it, stop everything. And then another group said, "Okay, it's going to happen anyway, so let's make sure it can be controlled. We will be the ones to ensure it happens safely, because we will program it using our values." I think I mentioned the Manhattan Project before. The EA, Effective Altruism people, focused more attention on the second school of thought. So in this big Doomer community, there is now a difference between Yudkowsky and the EA folks. But Anthropic is of course firmly in the EA camp, because they are the ones who brought this forward into reality. But if you think Chris Olah is the only one with crazy perspectives getting involved in disputes like debating with the Pope, no, that's not so. I mean, there are a lot of people who genuinely believe the model is already becoming conscious, and for that...

Host: Have rights, this of love, of grace. It is a machine, it is currently to bring correct with values program they can do it too. Now everything should be done. Here a there is a consequence I think. You do it. If not supported, this future Superintelligence you may be punished. You this loudly when you say this how much crazy. To you, do you understand? Lift this. For everyone required to you this is a fact. Let's check. Went to Google. Look, okay? Roko's Basilisk, Google it. This community what is believe I am with you. Truth it is said you will see. Yes, literally a Wikipedia there is a page I am meant.

Host: Okay, for OpenAI some math main advances there has been, they are big the response is getting. Fortunately, about it to us to tell everything Freiburg here. Millennium in gifts one resolved claiming about OpenAI last month we spoken. To do that they thousands agents millions dollar computer and took. Mathematicians still it is checking, I got it. Tuesday, Tuesday OpenAI 700- more than theses released, 700. Main problems resolved or developed claiming with 370 results. These made release an undone it is a model, average 3 hours compute. Scientific. Lean because it has been verified results maybe right. American says. That's proof. Assistant software it's language, but they are still review not done. Freiburg, about the background I am here. Again again to speak not going. Here reality what is it? You tell us.

Guest: Revealed of knowledge quantity using in human history the biggest discovery today is the day for me with confidence to say able to I think.

Host: Math in history, human in history?

Guest: Human history, reason all this math in evidence from flowing consequences very important. You have this. Tweets a lot to read can, in them some in speaking for me happy. Any personal in evidence I am a as an expert to pretend not going, but many some consequences what is it? To highlight it has begun. Electronics, quantum computing, for airplanes construction systems, medical for applications more sensitive diagnostics tools construction started all of them spread. But in this really deep how is that? Done that is, that in society humans about the role about the consequences speaking.

Guest: That former sailor- Stokes thousands hours, anyway millions dollar, proof to be taken came. This evidence 3 each hourly below compute only time just taken. That is report. Hundreds or maybe thousands year human labor this important among the findings for each one a on the computer it's been 3 hours. To shorten this is equal. Mathematics using to you to do this of what can be because it is worth highlighting.

Guest: A problem to solve a task to do of these loops in systems a role for AI carrying where is it? A loop one postulation or it's an idea, then you are the idea testing, you you will get the result, then you your postulation changing, you are that repetition loop again again running. The real AI question, to you excellent forecast to perform possible in that way loop model where to apply can. That is, each and in the loop, one great idea, to you loops to parallelize where possible, you loop where is the bicycle? Let's make it smaller. Of course in mathematics, you loop bicycle to shorten can, because to you on the computer your forecast to try can. Physical in the world you are nothing. No need to do. So, to you can you say, "Hey, here's one. The idea is a proof, that try, that is it working? Look at that, then it is it working? Whether or not look, reasoning do it, one new idea bring it, that again again do it." In silico to you everything to do can.

Guest: So, did you know? This in essays some of results, evidence, a lot mathematicians about them have spoken. They call it a alien intelligence like calling. How is AI? Many different dots, different ideas, different ideas, problems to solve different approaches and together to solve how connected for them really I don't understand. How is this done for humans to understand even very it is difficult, but the same time is many loops, parallel loops running among the advantages this is one. Each for the loop dots to connect can, otherwise for humans research theses to find new theory to bring many days years to spend can; then, on the computer that's it. Try it out next idea to bring that again again to do can, that is 3 within the hour, for the computer hundreds or thousands year humans to work can. Now, this is a incredible progress is made. This amazing. This is the case in many parts of the world kind of to change going, this a beginning only.

Host: But that asking, I think question for me we need to talk. A lot of fear I am there. I think, "Okay, to mathematicians this is for AI to do if possible, everyone work lost isn't it? Meaning?"

Guest: "The answer is no." Is, because mathematics very special the whole thing test bicycle, full loop bicycle too in silico to do can that is. To you a new one medicine discovery idea or a new one protein folding pattern if you want to bring it, or, did you know? In chemistry too material science too problem if you want to solve it, the loop has a physical there is a factor. One in the loop there is a man, there you are actually go and do it. In the lab need to try. So, that loop bicycle a slow there is a part.

Guest: Oh, to you of course that to parallelize can. To you excellent ideas to bring can, but inspection, loop's actual speed mainly the physical world step by step determined. Truck drivers about replacement predictions coming true we of not seeing this is also the reason. Everyone he said, "Oh, 2025 by the time all trucks drivers will be replaced." Anyone AI not replaced. Because, truck driver's loop, something take, somewhere else take it, that give up, come back, AI actually loop not minimizing. You loop to parallelize can't. An AI driver using to you this a little more to make it better maybe, but AI actually that special one loop's finance efficiency not increasing.

Guest: Meanwhile, in mathematics loops, you are a idea when bringing, you do that testing, you that's wrong. You understanding, you are a new idea testing, that's it mathematicians as a job doing, they are these loops running. All this in seconds in silico to do can. So, in the loop man including, the physical world with applications a lot of material things, AI that jobs to delete not going. A man computer using a loop operating jobs are the most more accelerate or increase doing. But this too what we see response what to look for very it is important I think, that's all mathematicians and yes, let's this up to bring can, maybe you about this next to speak want, but AHM saying a human mathematicians and this not requested that is. This job we do not requested, that, did you know? Very busy protective a perspective yes, to you do you understand?

Host: About mathematics different another there is a point, however that for coding applicable, that is, mathematics proofs depends, and proofs easily validated.

Guest: Yes. Coding in the matter similarly, to you compilers yes, right? So, the code must compile, so that it works. To you easily to validate can. So, also in coding and in mathematics progress is very it's fast, because you as said human feedback without including same to you reinforcement to study.

Speaker A: Can you, right? To validate that it works. To ensure very, very fast—this is the way. That's why coding is working. That's why coding is unbelievably well working. Tons of training data. Hacking, cybersecurity, yes. Already lots of data exists that easily can be faked. That's the point.

Speaker A: So, did you know? A possible model to become a lawyer, you do it in comparison. Like that, okay, training data to be achieved, then that data creation you should commission. Each customer is in a way unique. To ensure that, to validate the answer is right, it is difficult. This is only a very long process.

Speaker A: So, coding and mathematics are the areas with the most fast progress, I think. The person who told me this will be so—in fact, credit should be given to him—about a year and a half ago, Leopold Aschenbrenner said to me, within the next two years the most fast progress regions will be mathematics and coding, because again, verifiability is there. Oh man, he said it right.

Speaker B: In these specific solutions, what big advances are coming from it, do you think?

Speaker A: Okay, that's right, because I have an answer. That's a great question. Okay, dozens of tweets, yes. Here in the biggest proofs, some consequence for people when someone destroyed them, I'm going to check. In the case of product engineering in the real world, how we are building materials, how we design materials, how useful they actually are. Air coming on the wings of airplanes, famously solved, the first that was it. Remember, all this engineering is upstream, right?

Speaker A: So this math advances things like quantum sensors design, enabling excellent engineering systems for humans. In fact, I have a friend running a quantum computing company. The Einstein-Bose condensate proof came out, directly influencing what they do at work for quantum sensors, highlighting—he sent me a message this morning that it carries a share of GPS, carries a share in medical devices, carries a share of how we build chips. What does the indication of the word mean? Deep changes, deep improvement, or do they need to think about it?

Speaker A: Here is the reason for my answer. Taking the other side of all this, I'm happy to. I think these math proofs are unbelievably new and very important constructions of thought. But these were not boundaries of physics standing in the way of progress for all of us, thinking when these will be resolved. This is not a race of solved things. When saying that, offending people is disgusting for me, I know anger comes to people, because they are a group of mathematicians who have written a really upset protest about Navier-Stokes.

Speaker A: Got it. Because into your sandbox, into your backyard, someone comes and you don't want them establishing domination. Do you understand? But that's right, I don't think so. So when you look at groups of self-found intellectual cul-de-sacs, they still are not big unresolved questions of science, I think. These are very small and very narrow and specific elements of intellectual exploration, scientists self-mentioned as important to each other. Therefore, this class of proofs is really great, I think.

Speaker A: Friedberg, as you said, showing computational horsepower overcoming other things, because you can repeatedly improve to get there. But is there a solution to cancer? Is there a supersonic airplane? Don't worry, this proof will not show you that. I think these materials were already built the same, and going to be built after these. Instead, we proved, now actually there are two completely verifiable, counterfeit-proof areas, I think: code was first, now let's add mathematics on the list. I think that's what we have proven.

Speaker A: Friedberg, do you agree, or do you think main technical advances helping humanity with new products and services will happen in the original world from these specific proofs?

Speaker B: No, no, from this specific evidence, yes, that is the context. Yes. The experts reading and announcing the advantages of these, highlighting things, it's about understanding the limits of optimization algorithms used in chip design systems, scheduling systems, AI software design, and logistics systems. This is the summary of some of this special proof. This actually will change algorithms, so a change in application of the algorithms we used. Matrix multiplication complexity problem will allow fast numerical computing for AI training, scientific simulations, and linear algebra. In computing, there will be a direct application for this.

Speaker B: Riemann Zeta function result: number theory, cryptographic research. Then, immediately, no practical cryptographic progress, but that opens a new discovery path for cryptographic study, that will be very important, obviously. By the way, let me stop there on one thing. In cryptography, many people commented that no main evidence was shared in the cryptographic region, which indicates the fact that there was some important advance in cryptography.

Speaker B: Because, considering all other progress we've seen, it doesn't seem like progress was not achieved in evidence related to cryptography. If small progress was achieved, there would not have been consequences for cryptographic systems, therefore they would have published it. So the lack of evidence indicates that there may have been some big advances in cryptography. And many more people posted that cryptographic wallets with public keys are very seriously in danger now, so a panic exists in the crypto society about what to do.

Speaker B: Before quantum computers come, if in this there are a group of mathematics solutions for number factorization that may lead to the breaking of public key cryptography, breaking public key cryptography is that matter. Everyone thought, then we have an original problem. So everyone saying it's like, "Hey, delete all your public wallets." Now, OpenAI rumor that two more new proofs will come, they are retreating from them. So in crypto, some activities it is possible to come out from the other two explanations, activity is postponed for security reasons. Obviously, this is appropriate sharing with the parties, but if there is a risk, that need to be informed, that need to be released.

Speaker B: So here is another plan to give up, how much evidence related to cryptography will be included in it, in which how much it will persuade people that making public key cryptography difficult. By the way, if I have to say it, I asked with ChatGPT, I said, "Will the latest proofs of important math problems make direct improvements and updates in life as we know it? Be honest, don't weekly extrapolate."

Speaker B: Its conclusion: "Latest mathematics evidence not as evidence of imminent advances in human welfare, but more strong as evidence of AI getting complex rational skills." Felt showing a little humility there. Lost your account access? Avoided because of showing disobedience to AI? Before using one of the frontier labs, what happened to me? What is it, you said. Within 80/90 different variables changing, such as simulation I was doing. Maybe I stopped doing Project A, follow Project B or anything else. In all this, for me required...

Speaker A: Answers. Because it is not available, I am disappointed. That's very not a crime, I said, but I said, "You very and get away, be soft and doing." Something, anyway, I am blocked. Done. Yes. Its total me block done. That is, the model said, to you two hands when lost with you, then they also called the dealer cards handle doing method need to adjust. "Please card." Don't burn it. Burn it down. Just make trouble. For me Scooby slide should be done. Do this. A