Joe Wisenthal: Hello and welcome to another episode of the OddLots podcast. I'm Joe Wisenthal.

Tracy Aloway: And I'm Tracy Aloway. So Tracy, we're recording this March 24th and of course almost all of our episodes lately have been about the war in Iran.

Joe Wisenthal: But what's interesting or what's a little weird is that just prior to the war, literally days or maybe hours, the biggest story in the world was actually about defense and you know the DoD.

Tracy Aloway: That's right. So you are referring to anthropics and its disagreement to put it mildly with the department of war.

Joe Wisenthal: Yeah, exactly. This was the biggest story going right up on the eve of the war in Iran. And of course, obviously, there was this contract and anthropics technology was used by the defense department. So, it was not a disagreement about the use of AI per se in war.

Tracy Aloway: But the question of the degree of to which AI could be used for autonomous weapon systems on their own without a human in the loop.

Joe Wisenthal: And surveillance, that was.

Tracy Aloway: Surveillance and surveillance. This was another key element.

Joe Wisenthal: But you're right. So, we've heard this expression autonomous weapons pop up more and more, especially in recent days. And I have a lot of questions over what exactly that means because my impression is the US military certainly has been using AI for some time. And so, we're really talking about degrees here of autonomy, right? And so yes, if you if you think about an autonomous weapon, I think your mind could go fully Terminator and you know there's like a murder robot out there that's making its own decisions on which people or places to target and then you get levels below that, right? Where AI is kind of helping humans to come up with strategic decisions,

Tracy Aloway: Right? So if there is a missile coming and you have a missile defense system, I don't think you want a human in the loop is like, "Okay, here are the coordinates XYZ that we think it's going to hit. At this point in time, we think the missile will be here. Are you cool with firing it?" I think everyone's probably okay with that level of autonomy. But I have a feeling that to your point exactly a lot of this discussion and maybe it's core to what Anthropic and the Department of Defense were disagreeing on. I have a feeling a lot of this is gonna come down to definitions. My guess is that there is not one shared agreement of this is an autonomous weapon system and this one is not.

Joe Wisenthal: Absolutely. And of course there are also questions over exactly how places like the Department of Defense, not only how they define it, but once they have those definitions, whether or not certain companies trust them to sort of stick to those policies. Because the US will say, well, our policy is not to survey our citizens at the moment. So, if you're anthropic, you don't need to worry about that. Clearly, anthropic feels otherwise or say they do. So, there are all these really interesting thematic questions that pop up from all of this.

Tracy Aloway: Totally. And then there's the question of okay, here's a technology and the government says we believe that we can use this to make the country safer. What? You're not going to let us do it? Like private corporation. There are some very interesting questions about the role of corporate power visa v the government and so forth. Anyway, this is uh something that has become even more timely. There were reports even in the early days of the Iran war about these AI systems having been used perhaps in target selection, but we don't really know. None of the reporting is like that clear. I don't think that they're going out and advertising the strike.

Joe Wisenthal: This is exactly how we're using AI.

Tracy Aloway: This is exactly how we're using AI and so forth. But this is obviously a huge debate and war aside, it's really going to grow and just as an AI is going to grow it seems in so many different areas. Anyway, I'm really excited to say we really do have the perfect guest, someone who's been writing and thinking about this stuff for a long time. When we talked to an AI expert, I marked the dividing line is like were you talking about AI prior to when Chad GBT was released as like I take a little bit more seriously the people who are in this space prior to November 2022. Anyway, I'm very excited to say we're going to be speaking with Paul Sharie. He's the executive vice president at the Center for a New American Security and he's the author of two books related to this. One is uh the most recent Four Battlegrounds: Power in the Age of Artificial Intelligence and then prior to that he is the author of Army of None: Autonomous Weapons and the Future of War. He was previously in the office of the secretary of defense also a previous army ranger. So truly the perfect guest. So Paul thank you so much for uh coming on OddLots.

Paul Sharie: Oh, thank you for having me. Very excited to be here.

Joe Wisenthal: Why don't we start? I mentioned I had a feeling that maybe the definition of an autonomous weapon is a contested one. But if I say to you, what's an autonomous weapon? What's an autonomous weapon?

Paul Sharie: So, I think you're right from the beginning that there is not a unified definition that everyone agrees on. The Defense Department has their definition that's written in their policy. I think conceptually I think the distinction really is a weapon that is choosing its own targets okay on the battlefield and it's not where we are today right now today people are choosing those targets.

Joe Wisenthal: But it is kind of a spectrum because we do have examples of weapons that have some measure of autonomy a good analogy might be self-driving cars where conceptually like okay a self-driving car would be where the AI is driving the car but then you get into an actual car today and a lot of them have intelligent cruise control, automatic braking, automated self-parking. They have all these like automated features that are kind of creeping you in this direction where the AI is taking over more and more control for what the vehicle can do. And it's actually pretty similar thing in the military space as well.

Tracy Aloway: So Joe mentioned that had we been having this conversation even a month ago now, it probably would have had fewer concrete examples of AI enabled weaponry, let's say, or strategy. When the Pentagon talks about its advanced AI tools that it's deploying for the Iran conflict, what are some examples that you're seeing right now that are different to say maybe just a year ago when we had another Iran conflict?

Paul Sharie: Right. So, there's a couple ways in which the Pentagon is using AI right now. One is narrow AI systems that have been around for over a decade now that do image classification, for example. So, this was the military's original Project Maven almost a decade ago where they took machine learning image classifiers to sift through drone video feeds and satellite images to identify objects. Okay, here's a building. Here's a person. Here's a vehicle. That's pretty mature technology. Now, what's come out in just the last couple weeks that's really quite interesting is that in the midst of this huge messy public breakup between Enthropic and the Pentagon, we found out that in fact Anthropic's AI tools are being used by the US military to help plan the war against Iran. That's obviously a different kind of AI tool. AI large language models, AI being used to write code, AI agents, and that's being used in a different way. It's really helping intel analysts sift through just the massive amounts of data that the US military has. And so if you can imagine the problem that the military is facing right now when they're looking at targets in Iran, US military's flown over 6,000 sorties against Iran. The Iranian military architecture is degraded in a lot of ways. US military bombed a lot of targets. There are mobile targets, senior Iranian commanders, mobile missile launchers and air defense systems and drone launchers. US mil got to bring all that information together and find out where are these targets right now and where is there an aircraft that has the right bombs on it to take these targets out. And that's how AI is being used to help basically process and understand all that information. When I think about the description that you gave for that, I sometimes think like could it be that now I don't think that using anthropics technology means they go into claw.ai and say give us a list of suitable targets for sorties, but it could be could it be something like that? I'm sure there's a different interface and so forth, but is that a completely ridiculous way of essentially framing the service that AI is providing right now?

Paul Sharie: So the way that these AI tools are being integrated are through a existing system called the Maven smart system. Okay, which is built by Palunteer and that fuses all this data together. So you basically have an existing architecture for data management for intel analysts that the military has that brings together all these different forms of data. You might have satellite imagery, geoloccation data, signals intelligence, other forms of information. That's pretty great for intel analysts, but that's also really unwieldy because how does a human understand all that data and process it? And that's where the large language model tools, whether it's Claude or other companies, can be valuable is there can be a way for a human to interact with that data to basically task a large language model to say, "Okay, here's a bunch of data I'm giving you. I want you to look for intersections in things, right? I want you to look for a place where we have satellite imagery and some other forms of intelligence that can help identify the location of you know some missile launcher for example and then humans can look at that and help one just like find where are all these targets."

Joe Wisenthal: And then it's helpful in planning too a human could say okay here's this list of potential targets that I have now they're scattered all over Iran Iran's a really big country I want to map these to locations for US aircraft at different bases across the region, what are available aircraft and what are available munitions on those aircraft that can be used to take out those targets to help build a strike package. So like the AI is definitely being used to help understand the battle space and to plan operations, but in I would say ways that are pretty narrowly directed by people. It's not quite as simple as like dump all this data into a context window for an LLM and then say AI figure it out. Okay,

Tracy Aloway: People are asking the AI some really specific questions.

Joe Wisenthal: So, I'm thinking how to phrase this question diplomatically, but I get that the difference between fully autonomous weapons is, you know, the human as a decision maker in the current setup, how meaningful is the human actually? Like what's your sense of it? Because I'm imagining if you're an intelligence officer and you're getting reams and reams of data from Iran and you ask the AI to pick out certain patterns or identify potential strategic targets, how much due diligence are you actually doing on what that model spits out? Because of course the tendency when a lot of people use LLM certainly is just you accept what it shows you on the screen. And just to add on to Tracy's question because this is where I want to go, which is that in the early days of the war, we hit that school. Yeah.

Tracy Aloway: And there I'm reading a New York Times report and that was out the result of quote outdated data provided by the Defense Intelligence Agency. Now, we don't know exactly what that means, but okay, various outputs come out. Then what happens like how much is the human layer currently in terms of okay, here are targets, here are ships that are on a battleship. this could be plausible. What do you think or what do you know about the level of human decisionmaking that happens between some output and then the ultimate call for a strike on whatever it is?

Paul Sharie: Yeah, I mean I think first of all I think it's a really important question because it is one of the possible failure modes if you will of AI and how we use it.

Joe Wisenthal: Because you could end up in a place where humans are nominally in the loop and you could say well it's not an autonomous weapon. human's making these decisions, but if the human is not meaningfully engaged and they're just kind of rubber stamping some kind of decision, then that's not really what we're looking for. So, I think that's a it's been a long-standing concern for many years about people worried about autonomous weapons. And I think that's a very real risk with how AI is used. Now based on my understanding of how the AI technology is used in Maven today and based on what I've seen of demonstrations of it because I have seen some demonstrations of this in action. I think humans are pretty involved right now in terms of actually looking at the output from AI giving pretty specific guidance to the AI systems. I do think there is a an underlying challenge that the strike on the school highlights which is when you're talking about thousands and thousands of targets. What's the degree of vetting that's gone into all of that information both in the runup to the war, which in this case that school was a fixed object and so that's likely something that should have clearly been much more vetted prior to the war kicking off that someone could have identified that that building that was struck had actually been at one point in time part of an Iranian military compound, but you could see based on public available satellite imagery that it had been moved out of that compound some time ago and had been converted to a school and it would appear based on what's been reported in the times that that information had never been updated.

Paul Sharie: In this DIA targeting database. Now,

Joe Wisenthal: I would hope that we'll get more information in the future and some investigation about exactly where that went wrong, but I think that does speak to this underlying challenge of how good is the data going into this AI system and how thoroughly are people vetting it. And again, in principle, AI might be able to help you with those things, but you got to use it the right way. And people still have to be meaningfully engaged in these decisions.

Joe Wisenthal: Why don't we back up for a second? Tell us about the work that you've done in this area. really several years ahead of the curve and talking about this stuff, planning for this stuff. Give us a little bit of sort of your background and what got you on this train again several years before Chad GPT.

Paul Sharie: Yeah. So really over a decade ago now around say 2011 I led an effort inside the Pentagon when I worked at the office of the secretary of defense on developing the Pentagon's policy on the role of autonomy in weapons. Uh the one that's still in effect today in fact and that was really part of at the time we weren't at all where the military is now in terms of integrating AI tools. I mean, these types of large language models just didn't exist at the time. But the military had kind of woken up to what I would call this accidental robotics revolution during the wars in Iraq and Afghanistan where the military had deployed thousands of air and ground robots, drones in the air and ground robots for diffusing bombs. And the military was starting to think through where is this going in the future? And one of the things that everyone could see would be valuable would be having more autonomy in these systems. The ability to not be totally reliant on a human remotely controlling them, which was really the case at the time. But that raised all these obviously thorny questions about like, well, how much autonomy should they have and what are the legal and ethical implications of that? And that was actually a topic of a lot of discussion among people in the military at the time and in the Pentagon for people working on these issues. And so that ultimately led to that policy directive that's still in place on the role of autonomy and weapons. And then when I left the government, I continued to work on this topic as we've seen discussions internationally through the United Nations as we've seen the technology evolve in really amazing ways, but also ones that have risks with artificial intelligence. So, when you were doing that job, I get that you were on the policy side, but did you ever see anything on the contractor side similar to what we're seeing with anthropic right now? Like, was there ever a contractor who said, "Actually, no. I'm really uncomfortable with the way that the department wants to use this particular tech."

Paul Sharie: Not at that time. Now, a few years later, after the US military launched Project Maven, there was a big dust up when it came out publicly that Google had been a part of Project Maven and a number of Google employees signed an open letter protesting that and Google eventually discontinued their work on Project Maven. And you know, it's not an exact replica here of what's going on, but there are certainly some similarities in terms of a disconnect between how some people in the AI community are thinking about how their technology ought to be used in war and how the military is thinking about it. And I think part of that's like there's this underlying challenge of AI is really different than a lot of traditional military technologies because it's coming out of the commercial sector. So in a way it's kind of like the opposite of stealth technology that was invented in secret defense labs and doesn't have a lot of commercial applications. AI is all of these different applications. It's not being invented by the military. The military is having to import it in and there are a lot of debates about how a AI should be used in the military and more broadly in society. Actually, on that note, I think this is really interesting and definitely a pivotal point in I guess the history of the military-industrial complex, but why can't the US government with all its resources actually develop AI in-house and just avoid the seeming complication of having to deal with a commercial enterprise?

Paul Sharie: Partly, it doesn't have the technical skills. the AI scientists and engineers are really in and there's a fierce competition for talent in the AI space and so the military just like can't buy that talent. they don't have it. And the government spends a lot of money, hundreds of billions of dollars annually on defense. But we've seen actually in the last few years that private enterprise is able to mobilize massive amounts of capital towards building data centers to training AI models. And partly because the commercial applications for this technology are much bigger than the defense applications. And so for a lot of these tech companies there's some at least maybe not in this particular instance but in the past there could often be some prestige associated with saying oh you know the air force uses our AI system or the Navy uses our technology but the defense sector is actually kind of small for them as a customer. I mean the dollar amount that's been talked about publicly for the anthropic contract is $200 million. That's not a lot of money for these AI companies. Yeah.

Joe Wisenthal: And so I think that actually we've seen the defense sector has struggled to just keep pace with the amount of investment that's needed in the space.

Tracy Aloway: Tracy, I think it's a good question. And then you remember, well, the government couldn't build a good healthcare website to sign up for health insurance. And I hate to bring that up because it's old, but it's true, right? So it's like, are they going to build a world-class LLM or can a government build a good employment insurance website? The We've done multiple episodes. the answer continues to be not the case. I do find it fascinating however your point about there is this novelty. It is impossible to imagine say Loheed Martin inventing a technology and then saying no you can't use it because Loheed Martin's entire rais on death right is building technology for the government. It is inconceivable what that would be, but it is sort of novel when you're getting these defense technologies and you know the Google was also an example of Google obviously had technology that did not originally serve a purpose of defense. We saw the we remember the employee revolt. Let's talk more about that disagreement though between Anthropic and the Department of Defense. In your mind, where does Pete Hegsth want to go with this technology? And is that deviate from some of the policies and the directives that you were working on when you were when you were working on this stuff?

Paul Sharie: So, what's kind of crazy about this whole dispute is particularly on the issue of autonomous weapons, literally everyone I've spoken with has said that there's no intention by the military to use AI to make fully autonomous weapons today. Okay.

Joe Wisenthal: And I think anybody that's actually worked with a large language model with any kind of chatbot, whether it's Claude or Gemini or Chatg GPT knows that if you use these to write an email, you need to double check it. Like in no way, shape or form are they reliable enough to make life and death decisions. I don't think the military actually wants to do that. What's at dispute here is a more fundamental disagreement about well, who sets the rules? And so the origins of this really was that when the Pentagon came out with a new strategy for AI in January, one of the things in their strategy was that going forward, they wanted their contracts with AI companies to allow the military to use their AI tools for any lawful use. Basically, look, anything that's legal, we want the ability to do it. And that has conflicted with how a lot of these tech companies have been thinking about their AI tools. They're very nervous, many of these companies, about harms from AI. they're conscious of these risks and so a lot of them have various use policies in place. You can't use AI, you know, to launch offensive cyber attacks, for example. That's kind of thing that actually like the government might want to do. And and so this is a this was really the rub with the government was like who sets the rules rather than necessarily like a near-term question of fully autonomous weapons. So what we've already seen is Anthropic has this disagreement with the government and then Open AI steps in and raises its hand and says, "Okay, Anthropic doesn't want to do it. We'll do it happily." Does this just leave us in a situation where it's sort of a race to the bottom, right? It's like the lab with maybe the least amount of safety concern or the least amount of reputational concern is able to do this. And so we still wind up in a situation where the government is using AI.

Paul Sharie: Well, and I think what's unfortunate here is that when you think about what would be optimal for the government, one, I think it would be ideal for the government to have access to this technology and to have access to all of the best-in-class models available because they are good at slightly different things sometimes. And it's much healthier for the government to have access to a number of different providers so that there is healthy competition in the market. you don't get lock in with one vendor, but also if the AI scientists are saying, hey, it's not reliable for this, you ought to listen. Like that seems like a thing you'd want to hear them like out about, right? And so I think like in order to use AI in ways that actually are effective for the US military, we got to have a healthy dialogue between the AI community and people in the military profession, what the technology can and cannot do. And I think it's unfortunate that we've seen that dialogue break down in such a dramatic way over this dispute.

Tracy Aloway: Just going back to the idea of who actually makes the rules, you mentioned earlier that, you know, you can't use Claude to hack into to illegally hack into a system. Supposedly, it is unable to do that. It has like a kill switch within itself that prevents it from doing that. If you're anthropic, could you not just hardcode some of these systems and say you're not going to be able to use be used for domestic surveillance of Americans or for war crimes?

Paul Sharie: So yeah, so this is where it gets a little more technical. It has to do with some of the ways in which the companies may providing their technology to the government. So there's a couple different ways in which an AI company could put safeguards in place to make sure that their model is not being abused. One is training the model itself to refuse certain requests. So if you ask the model to do something, it's just going to say like, I'm not going to do that. That's not consistent with the guidance that I've been given. And the model's been trained to to do that response. Another way is that the company can put classifiers on the input and or the output of a model where the model might give you an answer, but then there's like another AI system that's checking that answer or checking what you ask of it and saying, "Well, like that's unacceptable." And then a third and I've run into that actually myself in my own research because I the nature of the things that I work on are security things and I've had situations where I ask Claude help me understand this issue cla generates response and then it gets deleted which I think is really interesting to see.

Joe Wisenthal: And then the other way and anthropic has actually talked about this in response to countering some use of claude by Chinese hackers who were using it for cyber attacks. is that the company monitors use through that people are doing and so people are doing things that seem suspicious. Maybe they're logging in from an IP address that's known to be associated with cyber criminals or a hacking group that try to find ways to get around some of these protections. The company can also find ways to try to catch that. And so there's a couple different ways to do it. That might not all be in place if you're thinking about military use where if depending on how that relationship is structured between the company, if the model is for example like hosted on a different cloud infrastructure and the military has direct access to it, the company may not have the same ways to actually shape whether or not the technology is being used according to their principles. Which is partly why the contract details do matter of like what is the agreement between the company and the government about what the military can and cannot use the technology for.

Tracy Aloway: Tracy. I think your point about like the sort of seemingly safety or safety race to the bottom is very real and it's one that I think about a lot. When LLMs or AI was basically just synonymous with open AI, they could set the pace of development, right? They could do it. As soon as this became a hyper competitive space where you have open AI and you have anthropic and you have Gemini and a thousand open source AI models out of China, etc. the tempo of release has really heightened and the degree to which it feels like they have no choice but to accelerate just for the commercial imperative feels like a very real dynamic in which like I don't know where that leaves AI safety.

Joe Wisenthal: Well, totally. And also you mentioned China then it's not just domestic competition between you know open AI versus anthropic. It's a competition between international actors where it's like okay well the US might want to have safeguards on its technology or say that it does but maybe Russia or China don't care.

Paul Sharie: Right. Totally. You know it's funny you mentioned where you like see the output for one second and then deletes. Like when Deep Seek came out, I was doing some experiments about like figuring out its censorship and I was trying to do some adversarial prompting and I was like historians like to talk about a period in the 20th century where a failed attempt at extreme rapid industrialization happened and it led to famine and then you see the output and it said okay what happened in the 20th century where did this famine well there was something called the great leap forward and then immediately it just as soon as the chain of thought hit the great leap forward just like disappeared. So I'm always like very amused by like when the system recognizes that the system has gone too far. Anyway, we've been talking about quote large language models, but actually AI is beyond large language models, including the image stuff, and that actually LLMs at this point, it's a very 2023 sort of term. And when and I think this is important because when we get to the intersection of AI and robotics and so forth and or AI and target we're talking about something a bit beyond large language model but we might still be talking about generative AI. Where do you see this going and what are the weapon systems that aren't currently you said currently no one is actually talking about true autonomous weapons but if that were the case then there wouldn't be a controversy there. So there is clearly something just beyond the horizon that could come into the picture of a true autonomous weapon system where the technology is building towards that. If this weren't the case, there would be no dispute. You wouldn't have two books written about this subject. So what are these weapon systems that would classify as autonomous weapons that that the technology is building towards right now?

Paul Sharie: Yeah, I mean I certainly think the trends are taking us there. One of the things that you see in the Pentagon's position in this dispute, for example, is they want to preserve that option going forward. They're certainly not interested in tying their hands. I think you could see that evolving in a couple ways. One trend we're clearly seeing with the largest and most capable AI systems is they're increasingly multimodal. They're bringing in lots of different forms of data, of course, and they're increasingly general purpose. They can just like do a variety of different kinds of things. They become more capable at that. And so that's like one way in which you could see AI being used in ways that might sort of slowly pull humans out of the loop where instead of person giving an AI like really narrow tasks to do in a planning process, maybe the AI is able to take on more, bring in more data, take on more like sophisticated longer term tasks. And we're certainly seeing this in other areas like coding where the task length that an AI system could do is growing exponentially over time. Another sort of way that we might see this look is just we see a network of AI agents that are interacting with different pieces of data doing different types of things and that the net effect of that is that maybe humans are again sort of like nominally looking at these targets but not actually approving them in some meaningful way. And then there's like a more separate I would almost think of like an embodied form of AI and robotics. Yeah. Which could be a drone or munition or robotic system that has some kind of onboard autonomy. That might be partly a distilled model so that it can be operating at the edge on lower computing on this actual munition or drone. or it might be some hybrid system that has partly machine learning but also just a lot of handcoded code that's that's more like expert level system that's going out into the battle space and hunting targets directly and attacking them. So something kind of like the lowcost drones that we're seeing Iran launch but ones that can loiter and identify targets that attack them without.

Joe Wisenthal: And that doesn't exist today. We don't have drones that loiter that are just hanging out there and then when something flash then there's a system is like this looks like a target attacks that actually doesn't exist currently as far as you know.

Paul Sharie: Well, I mean they're not they're not in widespread use. So there have been some narrow examples I would say historically dating back to the 80s in fact of lording munitions that could search over wider and would queue off of radars. And so radars are what the military would call a cooperative target that when they're emitting in the electromagnetic spectrum, if you know the signature of the radar you're looking for, you could see it. You can just home in on that radar. Okay?

Joe Wisenthal: Now, if they turn off, it's different than hidden and they're harder to find, but there have been some examples, a system that the US Navy had in the 80s called a Tomahawk anti-hship missile. Not actually the same Tomahawk cruise missile that the military is using now. a different one that was designed to fly a search pattern and hunt Soviet ships. There was an Israeli system called the harpy drone that was designed to go after radars that would lorder for a period of time. But these lording munitions have never really been in widespread use by militaries.

Tracy Aloway: We got to invent one of those like high-pitched alarms to deter the loitering drones from hanging out outside targets. I guess I mean we have electrical jammer so that does exist. Okay. So when I think about as we move towards more autonomous weaponry, I think about bots basically interacting with bots at that point and then I think back to previous examples of bots interacting with bots and there are numerous ones where things tend to go off the rails.

Joe Wisenthal: They just start debating the meaning of life,

Tracy Aloway: Right? Or they start talking in like a language that no one understands except them, stuff like that. Does the possibility of undesired escalation go up the more we move towards fully autonomous weaponry?

Paul Sharie: I think that is a very serious risk. And so the like mental model that I have for this are things like flash crashes that we've seen in financial markets due to the interactions of different algorithms that are executing trades where you get these emergent properties of how the algorithms might interact in the market and it's a competitive environment. Companies aren't going to share the details of what their algorithms are doing and you get these strange behaviors. Now the way that financial markets have dealt with this problem is regulators have installed circuit breakers to take stocks offline if the price moves too quickly. There's no referee to call timeout in war. And so I think that's like particularly in cyerspace one can envision a future where that is a risk where things are happening at machine speed and you have autonomous offensive cyber operations. You need to defend against that. You need some measure of autonomy on the defensive side to defend at machine speed. And you could get situations where you get weird interactions that might escalate a conflict. Or it could also happen between drones interacting in some kind of crisis situation. Now, a situation where like there's a big shooting war underway, people are already attacking, it might be less of a concern. Although you still could worry about escalation geographically against bringing new countries into a conflict or maybe attacking really sensitive sites that are tied to nuclear command and control that you'd rather not go after. So I think that's a a very real risk when we think about how this technology might be employed going forward. What about AI in really difficult ethical questions? Strikes where we know that civilians, for example, are going to be killed, which that happens all the time in war and presumably tries to be minimized, but war planners will find some level of acceptable, they call it collateral damage. Is AI playing a role or do you expect it to play a role in some of these strikes that may be gray areas?

Paul Sharie: I think you can envision ways that AI would be used that would make warfare more precise and more humane and ethical and ways that it could be used that would not and would be the opposite. So, for example, if you had an AI system that could look over all this targeting data and then identify if a strike is within a certain distance using, you know, munitions of a certain size of protected targets, whether it's schools or hospitals or critical civilian infrastructure and say, "Hey, whoa, like warning here. you should not carry out the strike or it needs a higher level of approval or maybe you should use smaller more precise munitions. That would be a really beneficial use of AI particularly when you're talking about a military campaign that hits a lot of targets in a short period of time that could be really valuable and may reduce civilian casualties. You know, a risk of all of this is you could end up in a world where humans are just less engaged in this process,

Joe Wisenthal: Right? And so there's both mistakes that humans miss or humans just don't feel as morally responsible.

Paul Sharie: Which I think is like a really tricky thing to think about morally because

Joe Wisenthal: On the one hand as a democratic society we make a decision as a nation to go to war. It's a very small number of people that have to carry that burden. And if someone if you could say well look what's the benefit to someone having like PTSD years after a conflict that they're haunted by something that happened that doesn't seem great. maybe we could reduce that. On the other hand, if we fought a war and nobody felt morally responsible for the killing that occurred, that doesn't seem good either and that could lead to more suffering and civilian casualties in war. So, I think that's a certainly a concern when you think about how to use the technology.

Tracy Aloway: Yeah, this is very ender game coded, right? Where you have someone who's basically playing like a video game and wiping out entire civilizations and they think it's just a video game, just an exercise, but it turns out it's actual warfare. And we're seeing some degree of that in the way that the Department of War is portraying this conflict so far. It's very video game.

Paul Sharie: Oh yeah. Yeah. Especially in the public presentation. Yeah. Literal.

Tracy Aloway: Animated gifts of video games.

Joe Wisenthal: Exactly. So Paul, you mentioned something. You mentioned the word circuit breaker. And circuit breakers are nice things to have in markets. I think they'd be even nicer things to have in armed conflict and war. Is there any possibility that you could design something like that for a major conflict?

Paul Sharie: I think it's possible like at a tactical level to figure out how you would do that and where you put protections on your side in the military and what you would do with even maybe cooperatively with an enemy. The challenge is how do you avoid what we were talking about earlier, a race to the bottom on safety.

Joe Wisenthal: Yeah.

Paul Sharie: And right, we're seeing this in the private sector between the AI companies. There's the rushing to get products out to market. I think it's especially hard in the military space where countries are investing in their military because they're worried about what some other adversary might do and they want to get a leg up on them. And so it's not that cooperation in the midst of conflict never happens. It does. And countries have agreed to take certain weapons off the table. Chemical and biological weapons for example. It doesn't mean that they're never used, but most civilized countries said we're not going to use them. But those examples are pretty rare and it's pretty hard to do. And so I think that dynamic is the really challenging one. It's like how do you find ways to cooperate with your enemies to avoid some of the biggest dangers here?

Tracy Aloway: So I think this is the last question for me. You know, you mentioned that drones are a kind of robot and there are other robots that have been existence in either national security or police work for a while. I think there are robots on the subway sometimes that seem to be Yeah, but they I don't think they really do.

Joe Wisenthal: I've seen the robots at the grocery store and they end up like chasing me while I'm trying to buy like carrots or something. Yeah, the ones that sweep the floors and stuff.

Tracy Aloway: Yeah, there's the robots. Yeah, there I think there is a Eric Adams did a contract with some company that was doing like subway robots or something like that. But these are like different AI as we talk about it and robots are two different technological trees but they are going to merge and there's the possibility of their mer ultimate merger. Do you foresee a world in which essentially we don't have human soldiers and wars are fought with who has the most advanced autonomous robots. We know China is investing a lot in humanoid robotics. Do you foresee a world in which that is the nature of a ground invasion as you and it happens with robots or various other sorts. Talk to us about how far that could go.

Paul Sharie: Yeah. So I mean look I think will we see robots used more in warfare? Absolutely. The long arc of technology in war from the first time someone picked up a rocket, threw it at somebody else, has been towards greater distance between adversaries, moving up through bows and arrows and rifles and intercontinental ballistic missiles. And I think robotics will be the next evolution of this trend of finding ways to find the enemy, strike the enemy without putting yourself at risk. And there's certainly a role for robotics out on the battlefield. I think a vision of like future wars of just robots fighting robots, there's no humans involved is not realistic for a couple reasons. One is I think militaries are going to need people relatively forward deployed to execute command and control for robotic systems.

Joe Wisenthal: The US military right now can fly drones remotely from the United States in a relatively uncontested environment against more sophisticated adversaries who could jam your communications link. And we see, for example, like a lot of jamming on the front lines in Ukraine. That's one of the ways you go after these drones. Then you need people close by because it is easier to have shorter range protected communications. When you go to a longer distances, that's just harder to do.

Paul Sharie: So, I think you need people relatively close for that reason. I think if you want to control territory, you have to put people there eventually to get out of a vehicle and walk around and control it. But I think the other reason it's maybe a little dark, which I think realistically in order for wars to end, there will have to be some human price that's paid. I think that's an unfortunate reality that if it's just machines that are being destroyed, that we may not get to the place where one side or the other is willing to sue for peace. And I think unfortunately war is likely to involve people and human costs for a very long time. I have one more question as well and I guess it's a thought experiment but if we think back to sort of pivotal moments in military history and their intersection with technology one of them that comes up is the Russian officer who decided not to press the button in response to the US and thereby you know supposedly save the world from nuclear disaster. Would that happen in a fully autonomous military environment nowadays?

Paul Sharie: I mean today it would still happen because there's people involved right? So this incident, Stannis Lro,

Joe Wisenthal: Thank you.

Paul Sharie: Is sitting at a at a terminal and gets this warning that there's a ballistic missile launched from the United States against the Soviet Union and then another missile, another eventually five missiles coming in. And the thing that's interesting about this is when Petro talked about it afterwards and we could hear what he said because we all lived because he made the right decision here is he talked about how he said he had a funny feeling in his gut.

Joe Wisenthal: And that he knew that the Russian system the Russians had just deployed or sorry the Soviets rather just deployed a new satellite based early warning system to detect US ICBM launches that it was new and he knew that a lot of this Soviet technology just didn't work that great at first. So he was skeptical of it. Turns out it was in fact faulty. It was detecting the reflection of sunlight off the top of clouds and the system was identifying that as a as a missile launch and that's what it was reporting. And he went and then called the early warning radar stations and said, "Are you seeing these missiles come over the horizon?" Said, "No, there's no missiles." So he reported up the chain that the system was malfunctioning. I think the scary question here is like if that was an AI, what would the AI have done?

Paul Sharie: Yeah. And it's kind of like whatever it was programmed to do, whatever it was trained to do. And obviously, we're seeing more general purpose AI systems like large language models have the ability to bring together more information to understand better context to have just like a a more contextual understanding of the questions that you're asking of it.

Joe Wisenthal: But it still doesn't know the stakes of a conflict. It still doesn't know like in some visceral level what the consequences are. And so I think that's a strong compelling reason why we need to have humans involved in these decisions. Even as the AI becomes more capable, there are still going to be things that we want humans to do because humans understand why it matters.

Tracy Aloway: I started the conversation by mentioning that, you know, it's not very controversial to say have an anti-missile system fire a missile when there's one coming in. But that could be wrong. And you want to make sure that it is in fact a missile and not a civilian airjet or something like that. So even there where it seems like a canonical example if you just want to have the missile system go off you would want to have human safeguards and human oversight and human understanding of the system such that it is in fact shooting down a missile. Anyway Paul Shie fascinating conversation really appreciate you uh coming on Lots and talking about your work.

Paul Sharie: Thank you. Really enjoyed the discussion.

Joe Wisenthal: Thanks so much Paul. That was depressing and fascinating all at the same time.

Paul Sharie: Yeah. All right.

Joe Wisenthal: No, it was great. It was Thank you so much.

Paul Sharie: Yeah. Thanks so much for having.

Tracy Aloway: I kind of got choked up at the end thinking about that decision that saved humanity at the end. And it's actually.

Joe Wisenthal: It's a crazy story.

Tracy Aloway: It's a crazy story. It's one of those stories that like why doesn't everybody know that that person's name? I mean, when you think about how many people I couldn't remember it either,

Joe Wisenthal: But shout out to another podcast if you want to learn more about this. Dan Carlin's Hardcore History has at least one, possibly two episodes on narrow aversions of nuclear disaster. So, very good to listen to, if not terrifying. You know, there's another point in that exact story that I think is really interesting and it's something I've been thinking about a lot across AI because there's something similar about humans and AI, which is that there is definitely a gap between what we know and what we can articulate. And this is certainly true with AI, right? So, the bot makes some decision or it determines something. It does not mean it's going to be able to spit out in words how it arrived at that decision. But that's true for humans as well. And so the idea that like okay maybe you you we do get funny feelings about things or take you know like again we're still pretty good at determining the difference between AI generated text and human generated text.

Tracy Aloway: Can we off but often I mean we still like often can get it right.

Joe Wisenthal: But could we write down exactly what we saw that we understood? There is that gap. And when we're talking about life or death decisions being made, it is scary to think about that role of instinct that we can't articulate having been taken out of the decision loop.

Tracy Aloway: Well, I think also technology is very good at pattern recognition, right? And responding to patterns and preset paths. It's been programmed to do certain things. And I think in a war environment, that's one of the most uncertain environments that you can possibly imagine. And so you have to think that there should be some element of flexibility in your response, but I don't know how you actually encode that into a thing that like runs on rigid numbers and lines lines of code. The other thing I was thinking was the anthropic situation and just how new that is from a sort of military history perspective in the sense that here we have this really important pivotal piece of technology.

Joe Wisenthal: That hasn't come out of like actual military demand. Right? to Paul's point, it's a commercial product. Its commercial uses are arguably a lot more profitable than its military ones. And so seeing that now interact with the Pentagon and the Department of War, really interesting. It's been flipped, right?

Tracy Aloway: Yeah. The closest example actually that comes to mind, there is one example that's in fairly recent history and that's Starlink.

Joe Wisenthal: Oh yeah, of course.

Tracy Aloway: And of course that was developed for commercial internet purposes, but it played a role in Ukraine and so forth. And at one point, if I recall, there was a tension point about the degree to which the Ukrainians could use Starland.

Joe Wisenthal: And so I do think that is sort of an interesting parallel here. The other thing, and we didn't get to this, and I this is going to be a little cynical, but I think it's right, which is that there is another element I believe to the anthropic situation, which is like Anthropic is the last big lib tech company or perceived as such, right? And we know that there's been this fairly sort of rightward turn in Silicon Valley over the years. And I don't think like anthropic is like totally part of that. I think they're still sort of lib coded. I also think it's incidentally why a bunch of people who probably like work in media are like end up using claude even though they're all kind of the same. Like I do think there's something there and that they have this thing. They say we're not going to ever have ads and we know that like Andre Horowitz Mark Andre has talked about ads are good. ads democratize the internet ads enable the internet to be spread to everyone there's some other politics at play because again like from my understanding and it would take a lawyer it's like I don't think that the agreement that open AI signed was that different at least than what the agreement that Anthropic had there's probably a little bit of difference I just think there's some other politics at play here.

Tracy Aloway: Perceptions matter but to Paul's point maybe nobody is talking about currently autonomous weapons happens right now fully autonomous but it can't be long and I think this is going to be a real tension sooner rather than later.

Joe Wisenthal: Can I say one thing and I'm going to be slightly facitious but also not can you be slightly facitious? I'm going to be facitious which is I have a solution to modern warfare.

Tracy Aloway: Don't do it.

Joe Wisenthal: Okay. But for real no for real.

Tracy Aloway: Yeah. If we're just gonna have bots fighting bots and it's going to cost a lot of money and result in people's deaths,

Joe Wisenthal: Every country should have to build its biggest, best, most technologically advanced robot and just have them fight it out gladatorial style. And my twist is to Paul's point about war always having to be painful in some way.

Tracy Aloway: Everyone in that particular society has to be engaged and dedicate some amount of time or money to building that particular robot. And you just have to iterate on the robot forever until you feel comfortable to have them fight. And that way everyone shares the pain but without the loss of human life. Am I high? I don't think so.

Joe Wisenthal: Well, I think you should write a book.

Tracy Aloway: No,

Joe Wisenthal: I think you should write a sci-fi book. All right. Shall we leave it there?

Tracy Aloway: Let's leave it there.

Joe Wisenthal: This has been another episode of the All Thoughts Podcast. I'm Tracy Aloway. You can follow me at Tracy Aloway.

Tracy Aloway: And I'm Joe Weisenthal. You can follow me at the stalwart. Follow our guest Paul Sharie. He's Paul_sharie. Follow our producers Carmen Rodriguez at Carmen Arman Dash Bennett atbot and Kalebrooks at Kalebrooks. For more OddLots content, go to bloomberg.com/odlots. We have a daily newsletter and all of our episodes. And you can chat about all these topics 247 in our Discord. discord.gg/odlotss.

Joe Wisenthal: And if you enjoy OddLotss, if you like it when we talk about the future of autonomous weapons, then please leave us a positive review on your favorite podcast platform.

Tracy Aloway: And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely adree.

Joe Wisenthal: All you need to do is find the Bloomberg channel on Apple Podcast and follow the instructions there. Thanks for listening.