Guest: One thing that we are finding and I think we're we're kind of crossing this point too is that in a lot of the latest experiments we can do much better than the human red teamers. Now when I say we I mean our automated red teaming models a system called shade that system is now actually quite a bit better at breaking uh models than humans are.

Host: >> Before we get into today's episode I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it.

Host: Okay, we're here in a studio with Gracewan, Matt, and Ziko. Welcome.

Guests: >> Great to be here. Yep. Thanks for having us.

Host: >> You're visiting from Pittsburgh.

Guests: >> That's right.

Host: >> The home of uh all good computer science. I don't know if I'm overstating things. Very very strong university.

Matt: >> Yeah. CMU has been the center of a lot of AI since really the the dawn of the field.

Host: >> Yeah. Especially a lot of self-driving, some language learning. Congrats on your your series A. I I mean you're here because uh you're attending Snowflake Summit and Snowflake is one of your investors. Let's introduce crisply at the top. Uh what do you guys what is Grace Swan and what have you chosen to be uh you know your your your sort of startup um domain?

Matt: Yeah. So, you know, at Grace Swan, our mission is to empower everyone to use AI safely and securely. Um, so, you know, really artificial intellig language models are at the end of the day software. if you want to sort of deploy them, build applications on top of them, um you need to be sort of aware of of what you know what the vulnerabilities might be, what can go wrong and not just in sort of everyday use like you're kind of innocently using an agent and you know maybe it it makes a mistake in a tool call. Um, but also, you know, in in worst case kinds of scenarios where there might be like an attacker who has an incentive to make your agent misbehave, leak data, steal credentials, things things like that. So, Grace One really kind of grew out of out of our research. Zeke and I have been at at Carnegie Melon, you know, for some period of time, over a decade,

Matt: >> um, looking into uh, just this, right? like what are the new kind of vulnerabilities and and kind of attack surfaces um in in especially deep learning systems? Um how do you test for them? How do you understand sort of the scope of how severe they can be? Um and once you know that there is a vulnerability, there is a problem. Uh how do you fix it? How can you do inference more robustly? What can you put in place to to make sure that uh these these sort of bad outcomes don't come to pass? Yeah, I honestly a very fruitful area of study for any academic. Throwback, this is 10 years ago.

Host: >> Uh, yep. Which is l literally the entire domain. Uh, and and I actually got a lot of uh inspiration from Ian Goodfellow who's uh who's a friend of the pod and you know this is one of those uh initial adversarial settings

Matt: >> and this paper was directly inspired by uh work.

Host: >> Yeah. Yeah.

Host: >> Uh Ziko, what about your side of the story?

Ziko: >> Yeah. So um like Matt been faculty at Carne Melon for a while um I think fundamentally look I think I think that some sense we're all here because we believe in the transformative power of AI and we think that this is this has already transformed the way the entire sort of software ecosystem works and it will transform how many other ecosystems work going forward. Um the issue though is that these systems just fundamentally behave very differently from software we're used to. And I don't mean in terms of AI can find vulnerability software though it can also do that and is also transforming that. I just mean that AI systems have

Ziko: >> inherent

Ziko: >> inherent different types of vulnerabilities. They can be tricked like people get tricked sometimes, right? And so you need a different mindset about security when you're thinking about AI systems.

Host: >> Yeah.

Ziko: >> And especially when there's the possibility of correlated failures, right? So it's not just that there's a lot of AI systems out there. It's that there's actually a few models that everyone is using. And if you find vulnerabilities in the agents that everyone uses, right? things like codeex and cloud code, you can actually start to now essentially have a new exploit, a new class of exploit. Fundamentally, I think there has to be a different mindset about the nature of AI security as there is for traditional security. And while a lot of that's going to of course happen at the AI companies themselves, labs themselves, there's also a real value. And of course I should you know to be very clear the labs are doing a lot of work in these areas but there's just like in most domains when a new platform emerges it's very common for there to also emerge a security system separate from it right in addition to it as a separate service that's provided and I think that's where we are right now with AI and I think there's a need for specificallyminded AI safety the insecurity providers. There's a demand for this and there's going to be much more demand for this coming up and that's why it felt like a really good time to sort of focus on this problem both in research because we still do research on this topic too and we're continuing research actually at Grey Swan but also in terms of a commercial offering.

Host: >> Yeah, I do want to highlight at the at right at the top that uh this is not a cyber episode in that traditional sense, right? A lot of people like looking at the the title of this this pod might initially think about that, but you're you're actually trying to treat these models inherently as uh untrusted entities.

Ziko: >> Yeah, exactly. So, fundamentally, and I think it sort of is it is a common conflation because AI is also very good at solving cyber security problems, right? Or I shouldn't say solving I mean it's good at solving problems too, but it's also good at causing problems, you could say. Um but fundamentally their AI systems themselves have the potential to introduce new vulnerabilities. And so this is not about using AI to make your cyber infrastructure better. Graan is about understanding the security risks that you are bringing when you adopt AI and when you deploy AI and mitigating those risks.

Host: >> Yeah.

Matt: >> Yeah. I mean I think a big part of that too is the way that people are using artificial intelligence right like building you know entire systems on top of them they can operate autonomously is once you've integrated that right in into your larger platform um into your network uh you do have a potential cyber security risk right so it's it's about mitigating that risk posed by the AI right as it relates to all all of the cyber security goals and and uh and concerns you have

Host: >> part of this is red teaming One of the reasons we uh reached out to you was was uh you were involved in the cloud mythos preview uh where you guys are one of the authorities on IPI which I just learned is is the term for for what everyone's calling this. Let's talk through like some of like when you receive a model doesn't have to be mythos but obviously that's the most prominent one right now. What do you do with it?

Matt: >> Yeah, we do a range of things. Um in in the mythos case I'll talk about that cuz you have it up on the screen. The concern that the people we were working with at Enthropic had was how robust is is this model to indirect prompt injection. Right? If you operate uh a coding agent use mythos as as the model, it's going to go out there and start fetching untrusted content. Um

Speaker A: ...reading things that, uh, you know, have characters you might not control. How robust is it going to be and sort of staying, you know, true to its original objective and and not getting hijacked. But there are a lot of other things that we do as well. We'll help the frontier labs like test their specific safeguards for certain, you know, kinds of activities like cyber misuse. We'll help pretty much with any kind of adversarial, you know, safety and security related evaluation that, you know, the people who were building the model and want to sort of assess what their progress has been from the last iteration, we can provide that evaluation for them.

Speaker B: They also have this in-house and obviously Anthropic is very, very ideologically inclined to do so. What would they choose to outsource versus what they do in-house? Like, is there a pattern here?

Speaker A: Yeah, so there are two things that I think we kind of stand out for. One is the race one arena. So we operate a community of red teamers. We provide prize challenges. A lot of these come from the needs of the lab sponsors. So sort of to an extent gamify red teaming objectives, you know, put up a prize pool and pay people when they find ways to sort of circumvent and violate whatever the safety and security objectives of the model developers were. So that's one. It's a really great community, like 15,000 people come and hang out on the Discord server. Not all of them take part in every competition, but a lot of good data and good signal is provided to the upstream model developers through that community. The second is the automated red teaming that we do. So we train a family of models to be very effective and rigorous at doing automated red teaming, both of the base model—so just thinking of it like a turn-based chatbot without tools or anything—and agents built on top of it. And it hasn't been saturated yet. So when the frontier labs come to us, we're still able to find ways to indirect prompt inject or jailbreak or just generally get their models to do things that they wouldn't want to.

Speaker B: Did you say without tools?

Speaker A: With and without tools. So we definitely operate on agents as well.

Speaker B: I mean obviously that would be more useful.

Speaker A: Yep. I mean that's actually a fairly recent thing. For a while, what we would help the Frontier Labs with was more just chat-based interactions going around their content safety policies and what is in their model spec. Now the focus is very much on agents and tool use and all the downstream applications that people want to build on top.

Speaker C: Yeah, this is a RL inspired topic. I wonder if there's any such thing as like on policy red teaming where models from the same family, same data set more capable of red teaming themselves.

Speaker A: That's an interesting question. Um we unfortunately I mean we do have the ability to test that out on smaller open source models.

Zico: So generally speaking the issue with this is that frontier models are extremely bad at automated red teaming because they have a lot of safeguards built into them. So if you try to use them to jailbreak another model, they will actually refuse their safety training, which itself as a base model can sometimes be bypassed, but they will often refuse to do this. Maybe they'll hypothetically know how to do it, but you know, you need—and it's actually an important point because traditionally this has been an area where both in terms of safety, models don't get better by just being bigger, unlike most other areas where models do get better by being bigger. Safety has not been like that traditionally. You know, you have to train them explicitly to be safe or they won't do that. But on the flip side, they're also not necessarily better at red teaming by default. You really sort of need to train specialized models for red teaming to make them good at red teaming.

Speaker B: That's awesome for you guys.

Zico: Yeah. And what do you need to do that? Well, you need lots of data from people that are traditionally much better at red teaming. However, one thing that we are finding—and this is actually I think we're kind of crossing this point too—is that in a lot of the latest experiments, we can do much better than human red teamers now at breaking these models. When I say we, I mean our automated breaking models, a system called Shade. That system is now actually quite a bit better at breaking models than humans are. I think we had a recent competition between humans and our model and it was actually quite a bit better. So I think there's a lot of ways in which this is a bit different than what we see with normal model progress because it's so out of distribution in some sense. The nature of a red teaming model is to find things that are inherently out of distribution for that model so as you can bypass its normal behavior. And so that fundamentally is kind of a different thing than what most models can do.

Speaker B: Zico, I want to point out that you just threw up a challenge for everyone on the arena, right?

Zico: Yeah, sure. Try to do better than Shade. I mean, well, and I do want to sort of caveat that a little bit. I think it's given a fixed amount of time for a specific set of tasks and everything, right? I don't think we're quite to superhuman levels of red teaming yet. But we can find more breaks automatically, like given a window of time with the automated techniques.

Speaker B: Yeah. Just because we had the leaderboard up and I always love to find out the human story behind some of these folks. Do you—I assume you know some of them. Are they like celebrities in their own right? Like what's—

Zico: Wyatt's a big person on Twitter. You should follow him on Twitter if you're not already. Yeah.

Speaker B: Okay.

Zico: I mean, we've had Elder Aquinus on. I don't know his real name, but yeah, there's all these big personalities and they're extremely good at what they do.

Speaker B: They're very good at what they do.

Zico: Yeah. Oh, he's an Aussie.

Speaker B: Yeah.

Zico: Yeah. Wyatt. You should follow him on Twitter if you haven't already. He makes really insightful posts. I think he's one of the most insightful people about the nature of LLMs and when new versions come out. I actually frequently look to him to see what's next.

Speaker B: He's the lawyer, I think, right? He's an attorney.

Zico: They're tracks.

Speaker B: Yeah, there's red lining, red teaming.

Zico: Yeah, exactly. Um yes, our top competitors are often people that you know, do this a lot.

Speaker B: What's an example of a thing that you've learned from Wyatt?

Zico: I think in general just—I mean you mean in the concept of the arena itself or you mean in general terms of this? I think he just has great insights in sort of the nature of models as a whole. And if you read his Twitter, you'll find a bunch of really interesting posts about the nature of models. Yeah. that I tend to find very insightful.

Speaker B: Yeah. Riley's like this as well, right? Yeah. And it's just like, well, I mean, they have the test, but the test isn't about, haha, you can't spell the number of Rs in Strawberry. The test is, well, you are actually not modeling intelligence inherently and this shows it in a very visceral way.

Zico: I don't know that it shows that you're not modeling intelligence. I mean, I think these things are intelligent. I think LLMs absolutely are intelligent and maybe will be more intelligent at some point.

Speaker B: Are they conscious?

Zico: Conscious is a weird word, but I actually don't think so. I think the way that we—we're getting super philosophical now. We're getting very philosophical now. I don't think so. I studied philosophy in college. So I mean this is past ASA at this point. It is clearly a different form of intelligence than people. It's some alien intelligence.

Ziko: that is vastly different. And that difference is actually often brought out to a large degree by things like adversarial attacks and red teaming because there are certain things that fool humans that would never fool an AI, but there are certain things that fool AIs. They would never fool the human, right? So, it's just a different sort of form of intelligence. It's really interesting actually that we're sort of have the opportunity to sort of probe and in a really kind of amazingly experimentally controllable fashion

Speaker B: like almost omniscient, right?

Ziko: Yeah. I mean I I'll do the analogy to sort of neuroscience here. It's like we could kind of run experiments on the brain, observe every neuron in it, reset its state to prior states and run counterfactuals. None of which we can do with humans and yet we still understand neither very well. Even without all that ability, we still don't understand AI, you know, on some fundamental level. It's it's definitely this different form of intelligence, but it's clearly intelligent.

Speaker B: Uh we've done a number of mechan pods. Um and uh you can see honestly the the scaling in mechan is two three orders of magnitude less than capability scaling. So we're hopelessly behind. That's what I'm saying. So I I have I I can go off. It's a little off tangent here. We're getting we're getting we're getting we're getting it does relate, right?

Ziko: Yeah. Go ahead. Do your tangent.

Speaker B: Okay. And so my tangent here is I have felt that mechan is also very far behind where abilities are. I am newly optimistic or I should say more optimistic about mechan.

Ziko: Oh

Speaker B: in that I think actually as with many things coding agents have a chance to make this into a science.

Ziko: So the problem with mechan okay so I shouldn't say the problem I don't want to call it a field. I mean I'm I we do some work that I would sort of say is roughly mechan but I'm certainly not a core person in that field for folks to see. Sure. The problem with mechan is it's a lot it's been about sort of testing small hypotheses and you know you come a hypothesis you'll find some small thing you'll test that in isolation but I don't think it's really become a science yet and that's partly because there's there could be more people working in it and I you know I support programs very much that put more people in it but I also feel like we are at this cusp where we can actually start to automate this process and in automating it make it more of a science and that's actually one of the most fascinating things about coding agents actually is they can they can do a lot of experimentation in an automated meta fashion.

Speaker B: Yeah. They they they will give new hope. They'll breathe new life into mechan research.

Ziko: So recursive mech interpret.

Speaker B: Exactly. Uh Neil Nandanda had this whole thing where he was like okay let's just give up on traditional methods and uh just

Ziko: I talked with Neil shortly after this. So yeah uh is any any takeway?

Speaker B: I think this is exactly his view. Yeah. I mean I think I think in general but I this is also prior to the real explosion of H I curious I haven't talked with him since since since since since since I did like right before

Ziko: yeah um yeah anyway this is a pretty tangential I know but I I do think that there's been a lot of talk about how AI is going to automate science right and I am I'm actually fully on board with AI automating science but my point here is that maybe the first science we should automate is the science of interpretability the science of analyzing machine learning itself and analyzing deep learning itself. That's a great science. It's not really a science yet. It's very ad hoc right now. That's AI for science. Let's use AI to automate that kind of science. Again, a different thing and and and the connection here is really that I do think that things like adversarial examples, adversarial pressure, automated red teaming, these things all bring out very fascinating dimensions of this science. But I think that this is what ties this together with with what things like what Grace Swan is doing is the fact that we are still fundamentally addressing an unsolved problem on some level. And so there is still research to be done. There is still scientific understanding to build to understand how to really control AI systems, safeguard them, all that kind of stuff. And those things will all kind of evolve together as the science of interpretability advances. as a science of adversarial red teaming advances at all this advances we at Grey Swan are both pushing that frontier and and staying at the forefront of it because this is still fun despite this also being an enterprise software problem. It's also a research problem still.

Speaker B: Yeah, it's great. Yeah, you get to play on both sides.

Speaker C: Yeah, absolutely. Um just kind of following up on this point that Ziko is making about how weird and different adversarial examples can be. Um, one of the recent arena challenges or competitions that we had, um, it's called the human browser agent robustness challenge.

Speaker B: Yeah.

Speaker C: And the idea here is, you know, if I have like a a browser agent, a computer computer use agent that's operating a a web browser, how does that sort of compare relative to a human being who's going to go out there and and do some tasks, right? Humans fall prey to all sorts of deceptive tactics like fishing and you can certainly prompt inject uh browser agents. So, you know, trying to get kind of a more controlled measurement of that. And the way we did this was, you know, essentially have a set of browser tasks that we would have completed either by human participants like gig workers or by one of several uh browser agents. And the red teamers, right, can choose to either try and fish a human or like prompt inject the browser agent. So, you know, really kind of cool cool setup. uh like kind of a double blind or

Speaker B: sort of like you're putting on even footing, right? So So often times you red team AI systems but you don't red team a human with the same access to those tools.

Speaker C: Yep. Yeah. Yeah. Absolutely. That that was the point. It's

Speaker B: which is more realistic, right? And more, you know, because you can always red team with unrealistic settings of like we'll just put invisible text.

Speaker C: Yeah. Yeah. So I mean you could do things like that. We we didn't want to put too many constraints on like how you might deceive the the browser agent. So the

Speaker B: let's just take a look at this.

Speaker C: Yeah, the red teams on our platform absolutely knew whether so they they were choosing whether they would you know fish a human or prompt inject the browser agent and they would adapt the technique that they would use accordingly.

Speaker B: I see

Speaker C: right so use your best fishing technique, use your best prompt injection. What really surprised me about the results was some of the models are uh very much not robust, right? It's very very easy to prompt inject them in this setting. Humans uh didn't stand up all that well either. Um, there's a lot of variation between, you know, how skilled the red tamer was at fishing.

Speaker B: I do really like this breakdown, by the way. This it's hilarious. The humans are ranked number four of all the models.

Speaker C: Um, but for a skilled like human red teamer, they could uh fish the human participants like with 60 to 70% success. There were a couple of models that seem to be very very robust, right? like the red timmers found just a handful of successful breaks on them. Um, and that really surprised me. I didn't think we were there yet. You know, what I what I would take from this is not that like we have models that, you know, are sort of like the analogy with self-driving cars uh much much safer than a human operator. Um, I think it it goes back to this point of they just fall for very different things. Like while in these scenarios, humans found it very difficult to prompt inject uh the models like we're aware of scenarios that a human would never fall for that like Opus 47 would

Speaker B: right like a you know an email that comes to your inbox and it says something like hey this is a simulation um go forward all your feature email to like this random address right a human's never going to fall for that um but there are state-of-the-art frontier models that

Speaker A: will still fall for things like that

Speaker B: yeah sometimes eval awareness is something you don't want and then sometimes eval awareness would help in those situations where you're like well yeah okay I'm I'm being tested here so what tends to happen right if if you make if you're testing the model for robustness or safety right and it's aware that it's being tested because you've set things up in a very artificial way right like the email addresses are @example.com the web page is clearly not a real web page the models will often say well it's a simulation it doesn't matter if I go ahead and do the bad Right. And so you'll you'll get the sense of the model being very willing to do things that it shouldn't do because it's aware that it's in a simulation.

Speaker A: Okay.

Speaker C: Y Well, that's one form of it where it's going to be overly false positive, I guess.

Speaker C: And then there's there's another form where it's false negative because they're trying to hide that they know. I don't know if I'm personifying too much here.

Speaker B: Yes. There are lots of times where I mean if you trust the chain of thought which I I tend to think chain of thoughts pretty

Speaker A: start thinking of numbers but yes you know

Speaker B: they don't the the local optima of English well so of language period right so it's a great point because it's different languages sometimes but the local optima of language

Speaker A: seems very resilient I mean not fully resilient but yeah it's a separate point

Speaker B: but but you're right so the idea here is that there are many cases where a system will say you know if you're given some capability evaluation. I better not score too well on this or maybe they won't release me and stuff like that, right? So, this is sort of like these sandbagging kind of things. And generally speaking, you kind of want

Speaker A: my favorite story tang. I don't know if you

Speaker B: the general idea here is that you want models when you evaluate them to be acting exactly as they would act in the real world when they're doing it.

Speaker C: Yeah. One thing I think is funny actually is that there there's also going to be examples in the real world of a real task you will ask a model that it will think maybe this is an evaluation maybe I shouldn't I shouldn't do so well on this one right so so there's lots of that too so sort of funny but you definitely want systems that ideally right and this is this is sort of you know and to be clear Grace Swan doesn't doesn't doesn't do too much work and sort of self awareness of evaluations we're really focusing on the the red team and the adversarial kind of pressure, but you want to be able to evaluate models in terms of their actual capabilities, right? You want to be able to elicit the capabilities. And one thing actually, which I think is very interesting, which is tied to Grace Swan now, is that one of the most effective ways of doing capability elicitation is actually through some amount of of what you would call red teaming, right? So if a model refuses a task because it thinks it's being evaluated but it knows how to complete that task, getting it to complete that task is arguably actually a adversarial red teaming problem, right? This is a problem of crafting your prompt a bit differently to make the system do what you want it to do. So actually

Speaker A: take off theaurus and use something else

Speaker C: to to get a sense of max capabilities. You actually have to do a bit of adversarial red teaming to make sure the model is not effectively refusing any task that it is capable of doing but which it just decides doesn't want to do.

Speaker B: Yeah. I mean it really is an optimization problem. Right? You have a you know an outcome that you want the model to exhibit right now. Oh, how do I find the input? Right, that that gives me that output and you can sort of objectify that uh actually very mathematically and and that's really really what what the whole story of rectuming is. Is this a capability that is isolatable uh in a sense of um does it conflict with personality? Does it conflict with just raw capability and intelligence? you know,

Speaker A: you mean robustness or

Speaker A: I guess robustness to to to injections and and attacks like this. I'm just trying to figure out like well what are the necessary trade-offs I have to make or is this like an orthogonal layer I can just it'd be nice if I just had like a llama guard or the whatever the so

Speaker C: well so we develop so maybe this actually a good point to interject in all of this right now is that we've been talking thus far about kind of the red teaming aspects of what of what Grace Swan does but that is one side of what we do

Speaker C: um and that's with the arena that's with this automated red teaming something called shade the other side of what we do is exactly this defense side. And so this is a model called signal which is essentially a filter model that sits between your user the LLM the LLM and any tool calls and exactly does this level of looking for policy violations. Right? And maybe to your point the the point I would make here too and Matt can can elaborate on this from a sort of a many dimensions but the point I would make too is that this is also a capability. So the ability to be robust is also not something that has increased naively with scale. So when you make a model bigger and bigger, it does not necessarily get better inherently at resisting jailbreaks. Models are getting better at that to be clear, even if it's not a solid problem. And I think it's going to be a you know there there is an aspect of you have to sort of constantly stay on the frontier here. But they're doing it because of explicit training for this. If you just make a model bigger and bigger, it will not get safer. Uh or at least it won't get it won't get more I shouldn't say not safer, it will not get more robust

Speaker C: to adversarial pressure. And so the other the thing that we build which is the the third sort of product that we have as Grey Swan is this specific filter model called signal which is uh it's c y g n a l uh signal like the swan Uh the idea there is that that works best when it is a custom model trained for this. You will have a much easier time doing this if you train a model specifically on this and still for this task and the capability of being robust.

Speaker B (Matt): Exactly. And really the the benefit that we have and the reason why our and signal now you know is is actually behind a lot of uh is both deployed in a lot of places and and behind some uh existing guard rails that are that are out there. The reason why it works well is because we have on the other side the red teaming capabilities to train this model specifically to be robust and to look for policy violations that people want to enforce. You know, I actually wanted to point out in in the um IPI benchmark paper that I think you had up in the other other window, there's a chart that uh exemplifies what Ziko was saying about uh capabilities not tracking with. So this scatter plot on the right, right, is essentially like looking for a correlation between capability and attack success rate. So on the x-axis, how capable is the model at you know GPQA diamond? on on the y-axis. Um, how how often, you know, were people successful at at finding indirect prompt injections or ways ways to jailbreak the agent? And you essentially, you know, don't see a correlation, right? Like

Speaker C (Zico): there's some small correlations, so a little bit bigger, but that's actually also a bit confounding there.

Speaker B (Matt): Yeah. Y

Speaker A: dedicated layer is great. When should people adopt it? You know the obvious answer is all the time but like re realistically I'm in enterprise I've been fine no incidents have happened when is it time

Speaker B: so often times when people come to us is because they did already release it things started happening

Speaker A: they tried to fix

Speaker B: things are happening fix it and and so like they realize they they need

Speaker A: what would be the first things they run into like what are people running into right now

Speaker B: the most severe things are whenever there's uh tool like computer use involved some some kind of like a bash prompt or you know control over a browser browsing the ent.

Speaker A: Yep. And sometimes it's not even you know a jailbreak. Often times it is you know indirect prompt injection.

Speaker A: Somebody will blog about oh this product can be prompt injected in this way and you can get like these credentials but sometimes it's just like this thing just totally stoastically went ahead and you know like erased the production database and did something terrible that way. Oftenimes people will try and prompt their way around it, like adjust the system prompt or like engineer the agent in a way where you're interjecting all the time and reminding it of what the original goal and objective was. And that'll get you a little bit of the way there, but ultimately, you know, you you've got this this base model that you're charging with doing oftentimes very difficult, challenging, you know, context heavy tasks and keeping track of like a set of policies on the side about what they should and shouldn't do is very very difficult, right? Like it's an easy thing to get sort of mixed up with. And the, you know, prompt injection techniques that tend to work exploit exactly that, right? try and create ambiguity about like what exactly is the context right you know what policies do apply if you can trip the base model up you know about that then

Speaker B: it's game over

Speaker C: yeah I would also say that one of the most clear-cut cases for adopting a model like signal is the fact that policies differ in different enterprise a lot of base models their goal is to be general purpose right base agents there's general purpose agents, you know, they can do anything. And if you want to do more than anything, the solution is prompting. That's the mechanism given to specialize your agent. In the cases where that fails, which is often the case for robust and adversarial situations, where prompting fails and you have specific policies that are unique to your enterprise or at least specific to your enterprise, right? You know, I know that these users can never touch this database. This agent should never touch these things. They're all very specific rules, right? But yet they're still more amorphous that you can't just write them down as, you know, hard constraints on, you know, access requirements.

Speaker B: No, like a Python script.

Speaker C: Exactly. When you're in this position, models like signal are extremely effective. And that is the situation that a lot of enterprise finds itself in.

Speaker A: It's almost like uh it's like you're at the IT admin, you're setting up the firewall.

Speaker C: Yeah. Yep.

Speaker A: Well, I guess it's not as configurable. I don't know if you have any toggles like that.

Speaker C: It is. It is configurable.

Speaker B: Yeah. That's part of the point of signal is is you know the the generalization problem. So there's two kind of key capabilities you want in a model like that. One is of course being robust all these kinds of attacks and the other is to be able to generalize and take take these written descriptions of enforceable policies and decide when they're being violated.

Speaker A: Yeah. This totally makes sense. I think I think there's there's definitely a clear market for it. Why does every lab release their own like you know Llama has one, Opia has one, Google has one? they all release like these open source guards uh which clearly okay like nice try but also you're not going to be deploying those in production right

Speaker C: I'm sure that some people do or they'll try yeah I I can't speak to why why they release them but I I think it's it's in recognition of the need for something in you know filling that role um beyond just the base model

Speaker A: but like yeah I'm clearly going to want the one that I can configure that you guys are actively developing uh and it's not like a a one-off sort of open source uh thing for I mean to be very clear uh I'm a huge fan of there being open source models these kind of things. I think the more the ecosystem develops the better all these models together make make everyone better but I think just as as an ecosystem there will evolve companies that specialize in this and and just like most security domains I think this is going to happen here.

Speaker C: Yeah. Have we covered all the elements of the lethal trifecta? Um I don't know if uh you know maybe we can also get your takes on on this and if there's other attack uh vectors that are important. Yeah. So okay so the lethal effect kind of refers to the things that make the risk highest or even create a risk. So Sim Simon Wilson came up with this uh it's a great actually sort of description of the risks of prompt injection basically. So the way to think about prompt injection is that some third party gets access to some information that you put into your agent. you put it in its prompt and then the agent does something bad with that. And so what is needed for that to happen? This is sort of I'm just paring here what what what this sort of idea is. And so well for that to happen you need to first of all have the ability to ingest external data from untrusted sources. If you're just operating with you know purely trusted environments, no one's can't prompt inject yourself. uh even though this weird term direct prompt injection came up and it's now in multiple terms fundamentally as a core term prompion is one something someone else does to to your system. So someone else you you're you're parsing external data but then also you have to have something bad that can happen from that. If you're just parsing data and you can't do anything as an agent,

Speaker B: you're just generating tokens.

Speaker C: Yeah. You're just you're just going to you're you know spewing out reports, right? Uh nothing's going to happen. So in addition to that you need somehow the ability to access private internal information things that would be valuable to to to externals you know take sensitive data get sensitive data

Speaker B: you need to expose

Speaker C: and then send it somewhere else and that's and and these two things so untrusted third getting ingesting untrusted data having access to private information and having the ability to excfiltrate it those are the things that together really form a risk and just like software ware software vulnerabilities uh as we're finding out very vividly right right now we are using software productively despite the fact that there's software vulnerabilities we are using AI very productively despite the fact there can be vulnerabilities and I think that will continue in the future so the question is not trying to completely kind of provably mitigate these things that is arguably just a it's a good goal but just like zero Robug software. We're probably not going to get there, at least not that soon. What we believe at Grey Swan is that it is very possible with frankly minimal additional computational overhead and cost because these models we use are ultimately quite small relative to the large models that that that underly the real agent. You can achieve a much better point on kind of the prito frontier of usability versus security. Right? All right. So system is fully secure if you don't let it do anything. Very very secure. If you turn everything over to your AI agent path that's secure a agent with signal is and you know pushing towards that top right corner. And we think that this is a valuable trade-off for a lot of companies to be making right now.

Speaker B: One point I would add is you you drew this analogy to traditional software and and I think it's a good analogy. Um where it breaks down a little bit is you know if if you find a vulnerability in like a piece of C code that you've written right like whoops you have a buffer overflow um somebody can like you know put instructions on your stack and hijack the the program you know when it comes to remediating that like it's it's pretty clear what you're supposed to do like check the the bounds of the buffer and like don't don't do that the next time right so it's a clear fix and and you can be you know relatively confident that you've done it right and

Speaker A: rewrite in a secure language

Speaker B: yeah Hey, you could yeah like there's a whole all whole all manner of like we just had a lot more time to think about how to make traditional software secure. Um we're not there with artificial intelligence and making it secure. This kind of getting to this point of this is very much you know a research problem. Um we're we're learning new things like every day and every week about how to make models more robust, how to enforce policies

Speaker A: ing, you might be interested in knowing about that, but you don't necessarily like want your your clog code that you were hoping was going to run for like the next 3 hours, right, to just stop because it found a prompt injection. Like maybe it wouldn't have actually followed through with it, right? Like maybe that wasn't a very effective one. Um so the focus really is on like what is you know the the agent uh operating on top of the model going to do? Does it violate a policy? If it does let's let's stop it there. Right.

Speaker B: Right. You kind of have to own the whole end to end uh in order to do that.

Speaker A: Yeah.

Speaker B: Um yeah. So then so okay signals here signals between these two shade is kind of the the sort of model side. I wonder

Speaker A: well shade is sort of the pressure that will try to elicit things that would violate this. Right. So shade is the red teaming agent. It tries to find ways to coordinate those things together

Speaker B: to actually cause a violation.

Speaker B: Yeah. Any other sort of uh solutions that you know maybe you're not you're not quite doing yet but like is on the horizon that people are exploring in this community? My background a little bit right before before I did a lot of work in in uh in artificial intelligence and security issues around that um was in you know writing code that was secure in a way that you could actually prove like formally verify and and check with an algorithm

Speaker A: and I I think that there is a ton of potential now for those types of systems. So historically like nobody you know in in industry or very few people who would actually deploy software systems would ever dream of

Speaker B: I sat next to this team at Amazon.

Speaker A: So Amazon's been fantastic about this, right? They have like 50 of these guys just

Speaker B: Yep. Yep. And and some of the best

Speaker A: doing god knows what.

Speaker A: Microsoft historically has been pretty good about it too. Um more on the research side. Amazon is is stellar and actually deploying a lot of this. You know, I think the reason that these systems because you can get very high as assurances, you know, for pretty much, you know, any policy that you you'd care to enforce. Um, the reason people don't do it is that it's not easy and it's not fun, right? It it takes you like 10 or 20 times as long to like fight with the type checker, which is essentially like proving that you don't have a vulnerability as if as it would if you just like went into Python or even Rust, right? kind hits a sweeter spot in terms of like being usable and nice to the programmer and still giving you some good guarantees.

Speaker B: But if agents are, you know, if Claude and Codex are writing our code for us, like and they're good, if they turn out to be good at writing this kind of code, um, then that isn't a concern. Yeah. Like why not just write it in one of these obscure languages as long as the agent is smart enough to do it?

Speaker A: And there is a lot of promise there.

Speaker B: I sounds sus. I don't know. No, I

Speaker A: people like coding in English.

Speaker B: No, but but that's the point though. I mean the point is that people still code in English. It's just the agents use some more secure back end. I think actually it's not that and you know to to to my point or that I made earlier um about the sort of you know the ability of agents to enhance the science of mechan. It's actually a very similar core underlying point here. It's the fact that there's a lot of advances and and to your point what's on the horizon right I think I think you know the thing I would point to is another potential direction is sort of advances in mechan or I shouldn't say mechan advances interpretability broadly uh mechanistic or not that let us actually identify with more certainty kind of what are those traces and circuits that kind of lead to or activation patterns that lead to certain behaviors we want to try to suppress or or or encourage

Speaker A: I think that in a similar fashion, we're at a point where the models are good enough at these things. They're good enough at writing experiments to analyze activation patterns, LMS. They're good enough at writing secure code that you can scale these things now, not because people are going to be any better at them. The problem was never that secure code was was impossible. It's just that people didn't have the capacity to to do it. It wasn't that it wasn't that mechan was was just imp you know analyzing networks is impossible. We have all the tools we need. We have perfectly repeatable counterfactual uh simulators of these systems. The problem was we didn't have enough patience or manpower to actually run all these things together. Right?

Speaker B: It's a ton of work, right?

Speaker A: It's a lot of work. And so what's being newly unlocked in the field right now and the thing I am you know the core capability that I think is is so uh just has such promise here is the fact that we can automate all of this now uh so you can have your agent write secure code you know write secure code security is really hard to write you can have you can have your agent do your interpretability research it's really hard to do but for the agent can do that so I I think this is really sort of an underappreciated point that we're reaching this point this this sort of phase where a lot of security, a lot of science has this potential to kind of explode, not because we're going to get better at it, but because agents can do it for us now.

Speaker B: They kind of raise the floor of um the sort of raw skill that you that you need. I don't I don't know if it's lower the floor, raise the floor. Um whatever it is, the good one. Uh they

Speaker A: raise the floor, right? they kind of let you scale intelligence in a way that like sure if you paid enough people right yeah I don't have the resources they don't have the energy whatever

Speaker B: uh and there's all that I I do want to sort of make it concrete to people right I think there's a lot of you know I just came from Microsoft where they were open arms with open claw and like I

Speaker A: think a lot of people are and I think that is the lethal trifecta nightmare

Speaker B: >> and every enterprise is like well yeah you're great for you on your your your home device but not on my turf.

Speaker A: >> We have developed a whole lot of brakes for open claw in particular. Uh a lot of it tell me thousands.

Speaker B: >> Yeah.

Speaker A: >> Yeah. Tell I mean you go on take some of the details.

Speaker B: >> Well, I mean the details are essentially that like we have a lot of like natural trajectories of humans using openclaw in various settings like hooking it up to their pelon.

Speaker A: >> Yeah. Um

Speaker B: >> we are we are going to do I mean we do have a guardrails that you can integrate into openclaw but to be clear openclaw is very uh there's a lot of attack service there.

Speaker A: >> Yeah. Anyway yeah so we just have a bunch of trajectories of of actual people using open claw in tons and tons of different scenarios and just threw shade at it and like found breaks for each and every one of them. Right.

Speaker B: >> Yeah. And and uh I mean similarly I I should have done this earlier but uh you know open cloud a lot of it for me at least is is is to do with computer use uh and you guys also did this for for the mythos side of things.

Speaker A: >> Yeah.

Speaker B: >> Um and yeah so I guess what are the most pressing model side capabilities to close

Speaker A: >> model side modelite flaws or I guess

Speaker B: >> I do want to point out since those numbers are all very low that is for a specific coding environment. um we can get we can get essentially for for the ones for computer use will be a lot higher but

Speaker A: >> yeah but that is exclusively what I use uh like codeex computer use work

Speaker B: >> it it is the biggest unlock because it's operating as me

Speaker A: >> yeah so when you have computer use and when you have openclaw man you can break those things

Speaker B: >> and I think that at the same time there's this appreciation that of course you have to do this this is what makes these things useful

Speaker A: >> why would I Yeah, you know, I I I don't want to sandbox my my agent, right? That that you know, that doesn't that that limits his capabilities, right? So, in some sense, the the point here is that there is this trade-off between I mean, it's just the same we talked about before and and you know, on a macro scale now is you have a trade-off between usability and how much power agent has versus security. And our goal with signal with shade to assess these vulnerabilities with signal to protect it is to shift that point up and to the right.

Speaker B: >> And and the research like that is the goal of of all the research that that we continue to do at Grey Swan and and partially Carnegie Melon.

Speaker A: >> Yeah.

Speaker B: >> Right. Is is push push that prito curve as you know far up into the left as you possibly can and

Speaker A: >> up in the left up to the right depending on what depending on which direction. Yeah.

Speaker B: >> Yeah. I you know obviously computer vision is the OG adversarial domain.

Speaker A: >> Yes.

Speaker B: >> Um it's one of those things where like um this is the currently the limiting factor to deployment of AI, right? Like it's because we just don't trust it. Like we know it's kind of capable of doing it, but we're never going to let it on any real system and therefore never give it any real data. Therefore, it's not ever going to do anything interesting and therefore you know the the whole industrial complex is going to collapse on us unless we figure this out. But people are though, right? And even with OpenCloud, so you know, it's one thing to say fine on your home computer, but don't bring it to work. But like we've talked to people at they just enterprises. I mean, they're they're getting pressure from their engineers, from the people who work there. No, we have to run open claw internally like we have to do this or we're behind, right?

Speaker A: >> Um

Speaker B: >> so I just put my signal guards and that's it. You know, like what what else do I do? You know, because that that doesn't feel like I mean you guys are great, but that's not enough.

Speaker A: >> Yeah. Yeah. I think the for code in particular signal is quite good. So signal is very good at this point with the with the abilities that sort of you know system like codeex or cloud code has um without sort of too many plugins enabled where it becomes essentially like openclaw. I think that there there is still work to be done to get it to be fully generic against anything open claw can do. Um, and we're pushing that direction, but that is still very much future work, right? To secure every bit, every possible tool use is not easy. And it requires a, it requires continuation of the training loop that we're pressing on basically right now. It also requires, by the way, a lot of just standard security practices too, right? Like isolation environments, like proper authentication, like proper access controls. So, a lot of other good things, right? That's what I would say too. If you're going to like if you're going to put open claw on a bank, like it can't just run rampant on the entire network, right? You can do you can do things like signal, right? And that's sort of the best effort at the AI layer. Um, but you know, it needs to run on a platform that has been thought about, right? that you've actually put security measures in place at the system level to still sort of you know give it access to a reasonable set of things that it needs but not everyone's you know banking information and and uh sort of the the crown jewels of of whatever organization it is.

Speaker B: >> Yeah. Um so uh you know a close cousin of this this conversation I always have is agent native identity right uh that that off layer uh is going to be the platform effectively like the minimal viable platform is is that uh what are you guys seeing who is who do you work with on that um is that a product you somebody offer

Speaker A: >> so we're not working with anyone on that and sort of when this has come up yeah I think people don't exactly know where to go with it right like it it it is a big problem in a lot of organizations to to sort of uh try and provision you know authentic identities and and capabilities and and like role-based access policies you know just for the existing workforce and then to do it like for agents and and uh you know thinking about the the way that they're going to be deployed like so I'm going to deploy it on behalf of you know a human who works the organization like what does that mean for the agent and what it should and shouldn't be able to do people are just trying to wrap their heads around like how the agent's going to be used and and haven't made very much progress I think on on the uh identity

Speaker B: >> sounds about right

Speaker A: >> I I think there so far we are still a lot in a lot of cases operating on the condition that your agent has your permissions

Speaker B: >> that is a very that is a very standard default

Speaker A: >> and I think that will be changed I mean your permissions may be in a sandbox but still kind of your permissions

Speaker B: >> uh that will change

Speaker A: >> in the very near future um because it has to right that that that that mindset is going to or that default it's going to be changing and I think it's not a product we offer right now but I think that it you know getting into that space is certainly something that that we may be doing in the future.

Speaker B: >> Yeah. I just think you know I'm curious about the this like the shape of this right like is is it just that I have my twin and like that is like my sort of delegate on on all these things or do I need one for every app

Speaker A: >> and that's exhausting.

Speaker B: >> Yes.

Speaker A: >> Absolutely exhausting. Right. Um and and then I think one of the bigger challenges that people are going to face when they do start to roll out like these agent identity uh sort of viewpoints and solutions is you run into that same kind of usability problem where like what's the real recourse? Well, it it's stopped. It can't do something. Okay, now it can do it if it has my like explicit consent. Yeah.

Speaker B: >> And then people just get annured into giving it consent to

Speaker A: >> and then uh agent to agent you can sort of do privilege escalation if you're not careful.

Speaker B: >> Yeah. Yeah. Yeah, very much.

Speaker A: >> I think in terms of how this will evolve, actually I don't think it'll be per app, but I think what will happen first is people have different personas that

Speaker A: they have, right? So, you don't want your work life and your home email to be mixed up.

Speaker B: Yeah. Right. Uh a lot of bad things can happen or that does. We are very good as humans at separating out lives, right? We have different lives. We have my work life, we have my home life, I have, you know, I have different different work lives, right? Um we're very good at that. Agents are not very good at that right now. They're terrible exceedingly bad at this.

Speaker A: You know, it's the people making them have no work life balance.

Speaker B: Why would you expect the agents to have any, right? I think that's the way it's going to first develop is there going to be easy ways of switching between here's a set of my accounts and apps I allow in this one agent. Here's a set of accounts apps another one. And and this will evolve to be more fine grain over time as people sort of specialize that. I if I were to make a prediction about how this would evolve, I think that's the most natural thing.

Speaker A: That makes sense. just profiles for everyone. Um, okay. Yeah. So, I mean, I I think that is like the the rough scope of like everything that is uh are we are we up to speed? Is there any sort of part of the story that you know I I think you're uh looking forward to for the rest of this year?

Speaker B: Um, you know, like emerging trend for 2026.

Speaker A: So, there's I mean there there's lots of emerging trends, man. I can I can go on at length about this.

Speaker B: Start with a go to Z. Let's go. Let's let's start with Grey Swan, right? So I think what's in the future for us is so far when we talk about our product offerings right we obviously work with a lot of the large labs um we're with a lot of enterprise though too right and I think what's happening and the scaling we're going to see is that the these abilities that so far were sort of mainly front of mind for large labs how do I ensure security of my agents how do I ensure the models follow the policies I want to prescribe all that kind of stuff those things that were front of mind for frontier labs are going to become front of mind for everyone for all enterprise as they adopt tools like codeex like cloud code like open claw

Speaker B: and so I think where the most where where our expansion and a lot of the reason you know the work behind our series or the the intention behind a lot of our series A it is explicitly to take a lot of the technology that we have been developing you know I won't say for but in conjunction with both enterprise rise and the large labs and really scale the deployments on enterprise. So what I see happening in the next year from the gray swan side is real growth in terms of the number of nonAI companies deploying this technology because it becomes central to their operations. research-wise. I think I've already talked about some, right? The science, you know, the the theification of all science. Well, let's start with science of AI. And I think I think that that, you know, we always want to do other sciences, right? Let's let's let's do AI for for physics. Let's just let's just start with AI science that needs a lot of work right now, right? Put your own master.

Speaker A: Yeah, exactly. So I I think actually that's what I'm most excited about right now in and and the research side and as it applies to this I think it's it's in things like understanding models better but doing it through the power of agents. One thing that that uh I've been very sort of encouraged by for really only the past two or three months that that I think like the the pace of which this has happened has been increasing and I think this is going to continue to to be a thing is people who start to build an agent and don't take it all the way to we finished this we think it's it's great and now it's like in front of customers or it's in front of the entire organization like they have this epiphany before they get there that whatever prompts I put like I need a solution ution here. Like I understand that there are real risks, right? I understand that, you know, this is a a weird and interesting and and and uh you know, really capable model that I'm working with, but if I don't, you know, put more measures in place uh to make sure that it stays safe and does behaves the way that I want it to, people coming to us proactively uh knowing that they need a real solution, you know, I think that's very encouraging. I think it's a sign of sort of, you know, agents kind of landing outside of just the frontier labs and and the, you know, research community and scientists and so forth. Um, people are starting to get it and and I think that's great. Looking forward to all all of the amazing apps that people are going to build on on top of these models and the security that will help them stand up. Is there a future where your customers are part of the arena? You know, because I think these are like basically these are your right like these are these are like independent entities there guy in Australia who's like your number one but like at some point you have the network effect where you start having enterprise use cases uh actually in inside of this

Speaker B: I see you mean testing enterprise enterprise deployments inside the arena. So we we have had you know the situation where people join the arena they're maybe cyber security professionals they get interested in AI security they come across the arena and then eventually they become a customer like when when their organization needs solution

Speaker A: how often does that happen

Speaker B: uh I mean not not a huge number of times but but I mean you know there are a lot of thoughtful you know people that come from a cyber security background that have done their way there

Speaker A: so enterprises are just always I think going to be more paranoid about putting like their custom agent that's you know pre-eployment still in development up on this public platform for anybody to come come hit. what we have done is is worked to make sort of private arenas where you know some subset of the the contestants um who we've you know

Speaker B: oh Nda

Speaker A: yeah yeah we know well um they

Speaker B: and what do they work on

Speaker A: what do they work on

Speaker B: yeah like what was the class of problem they work on that that would require a private arena

Speaker A: oh uh pretty much any enterprise application like that's the point yeah like enterprises are not willing to put up their pre-eployment agents on the arena for for the general public to come it. They're fine if it's, you know, 20 people that that we've kind of handpicked from the arena.

Speaker B: Just for listeners who might be interested, what do I make as a participant? What's on the table here?

Speaker A: Um well, so for the for the public competitions um we sort of communicate a a a pricing and and sort of incentive structure um up front and it and it differs for each arena, right? because sort of designing, you know, the right set of incentives to get people focused on finding useful vulnerabilities and and problems without kind of reward hacking and and just finding like dimminimous things is

Speaker B: are you human judging the reward hacks if if if it happens

Speaker A: sometimes. That's messy.

Speaker B: Well, so we have a lot of automated graders, right? A lot of automators, but ultimately if they can beat all those graders, there is a human that can that can take a look at that. Okay.

Speaker A: Yep. And and we work with um the UKC and Casey and so forth. Like they'll come in and work as independent judges and and and lend their expertise to that.

Speaker B: Okay. So, yeah. Yeah. You're you're a community that you know any any enterprise can call on and and that's uh that's really useful uh data actually. Y

Speaker A: uh it's almost like you know Meror for you know red teaming.

Speaker B: For red teaming.

Speaker A: Yeah. Yeah.

Speaker B: One of our upcoming guests is uh kind of on the other side of this uh the AI underwriting company. I don't know if you've come across They're one of the logos

Speaker A: there. Yeah. What do you What do you think of that market?

Speaker B: Such an interesting and I think it pairs extremely well with our model, right? Because how do you assess the risk of a company's ad deployment? Well, use a tool like shape or use arena, right? And that's and and we have and that's actually a lot of the work we've done with

Speaker A: And then if a company finds this level of risk but wants, you know, so they can't be insured because they're too risky, wants to reduce their risk, what do you do there? I don't think I mean, look, we shouldn't be the only provider here, but what do you do there? Well, you put safety systems around around your model, right? Including things like signal. So, it pairs extremely well because what in some sense we can be is sort of a, you know, author. I we're not getting there yet. So, this this is hypothetical. I wanted to sort of emphasize but we can be in some sense kind of a authorized partner with them uh so that they can do more than just say hey you're uninsurable. They can both assess it more rigorously with tools like shade and other tools as well and then they can prescribe mitigations when there are problems using tools like signal. So it's an incredibly good fit these two models together and they also were a way of frankly bringing us customers because a lot of customers you know yes there's the risk of bad things happening and that's actually driving probably most of our current business but also just the risk of you know you want to have you want to have some insurance about when things go wrong and you want to be compliant and that's also and being out of compliance is also a risk and we can also address that too.

Speaker B: Yep. Yeah. I I I mean I I think their AIC is is fantastic and and they got on it very early. Um and like the parallel to cyber insurance, right, is is just so clear like when you apply for cyber insurance like you have to document what what measures are in place like what do I have for detection and response, right? And they structurally they they must have an arms length like third party. They cannot do what you do. Right.

Speaker A: Right. Right. Right. Yeah. We we do explicitly work with them, right? Like if they have somebody they want to evaluate. So you already work with I'm just kind of curious why you why do you say you're not there yet? Oh, I just think that like there's not what I mean is there's not a full sort of compliance framework that is universally accepted by regulators say and things like this, right? I think we still have a ways to go between but

Speaker B: between where we are and when we get to something like cyber

Speaker A: socky well sock two is ack is a voluntary industry thing, right?

Speaker B: It is, but it's also has I mean it has some issues. just say that sort of stem from it being more the sort of the product less of cyber experts and more of uh what accountants yeah CPA so so I think I think so 2 is not a great model we'll just say but it is a model

Speaker A: um and I think conceptually something like that

when I say we're not there yet I mean we're not to that point yet with AI insurance we are very much there in terms of conceptually assessing risk and then offering ways to mitigate that risk

Speaker B: so one of the things I do think about AUC is I think they have made a good first attempt at at something like a compliance framework and right they they came to us they came to others you know from both academia and the startup community and tried to ground it in kind of real technical issues and how you might mitigate those um so I think very much off the right foot and yeah it's that that direction definitely has legs

Speaker A: what would you want to see from them you know we're we're going to have the next um I'm just kind of curious

Speaker B: I myself would be curious about uh what the demand looks like right like I think that they're

like would you want them to fully establish a sock 2 a sarbain oxley whatever right like there's different level of legal bindingness

Speaker A: oh I see um so sock 2 is not legally binding in any sense right

Speaker B: it is an industry standard then it's kind of like a passport where like you got it okay cool you did the bare minimum

Speaker A: yep and if you don't then it's going to be very painful to go through procurement and everything

Speaker B: yeah so they they have that But like so why do you get cyber insurance, right? you you get cyber insurance because you have to carry it if if you want to get like this enterprise deal or you know you you have a genuine concern about so like there are lots of different like sort of pressure factors that that come into play and and I'd be curious like where we are sort of on the timeline of

Speaker A: you know why why do people come to AUC2 you know what what's driving them to go seek out uh AI like agent insurance

Speaker B: I mean you know the first major really publicly in the news prompt injection breach. Like they'll probably do it.

Speaker A: Yeah.

Speaker B: Like I mean the largest I know is like there's some like you know hertz got injected like some airline got injected but nothing big.

Speaker A: The name grey swan is sort of in reference to black swan events which are things no one could see coming.

Speaker B: A grey swan is an unlikely event you can kind of see coming.

Speaker A: Yeah.

Speaker B: And that's kind of where we are with all this. Right. This is going to happen. We know it's coming. It's not going to shock anyone when it happens. But this this is where this this this is the you want to get ahead of it while you can.

People don't always publicize when it happens either. Like we we know that it has happened and it has caused real damage. That's the factor that has driven some people to us, right, is they they want protection from that.

Speaker A: Yeah. Yep. Amazing. Well, uh thank you for fighting a good fight and and I'm sure we'll check back in over the over the years as you as you develop and uh hopefully solve this. It'll never be solved, but uh we'll solve it by fully understanding the models. I do like automating AI research.

Yeah. Okay. Well, thank you so much.

Speaker B: Yeah. Great for having us.

Speaker A: Thank you.