Thomas Wolf: you kind of have to move fast. It's a matter of at least hours and even more minutes. So you don't have time to apply for cyber security program if the model was not at all task with attacking us but decided to do that as a side quest of uh something else. So he created fake accounts, fake GitHub account trying to attack the sandbox by blackmailing. That's a very different level I think of thinking. I could have been the target of this side quest of the model basically. That was very interesting artist. Very, very scary.

Matt Turk: >> Hi, I'm Matt Turk. Welcome back to the Mad Podcast. Today my guest is Thomas Wolf, co-founder and chief science officer at Hugging Face. We unpack what might be the biggest AI story of the summer. How an OpenAI powered agent penetrated Hugging Face during cyber testing and why an open source model helped the team fight back. We also explore model alignment, the future of western open source AI and the race towards recursive self-improvement. Please enjoy this fantastic conversation with Thomas Wolf

Thomas Wolf: to take uh things in order. So the Open hack uh so I I know some of it is still being unpacked. I think OpenAI was uh on stage at Black Hat in Las Vegas uh yesterday as well talking about this. So, what's a two-minute version of what happened for people that that may have heard of it but may not have followed everything? Yeah, for sure. I mean, typically what happened is, um, now about 3 weeks ago in, uh, July 11th, um, we started to have some, you know, uh, strong in that, uh, a hacker was trying to penetrate infrastructure. So, for context, we're pretty visible in the world. We're pretty well world being central now in the tech world. We're pretty central in the tech world. So we do have you know uh regular occurrence of people trying to hack into our platform that that's a common thing. Uh since I mean I would say in the past two years something like that we've we've strongly upped our our security team. We now have a serious team. So we're kind of used to get this, but this one was different because we we uh we I mean first was massively parallel and in different way than just a typical hacker parallel thing in that many uh tracks were explored in parallel and also there was some very strange things happening. I would say just two thing that were quite strange. The first thing is we could not really make sense of what the hacker was trying to uh access. So usually hackers try to get the same thing. They try to get passwords.

Thomas Wolf: Yeah, I mean the the so what happened in the so so we have a couple of like traditional cyber security protections. So like we or Amazon like we we use a range of them but we also have a stack like many people who is mostly based around uh plot code right now which is uh which we use for many. We use that for deploying. We use that for coding, but we use like also for operating and processing. And and in this case, it's not only that Fable told us I'm not allowed to touch cyber security, but also Opus, which was the fallback, was saying no, I'm I'm also not touching this thing. So basically, the end was just uh say we won't process anything about that, but you're welcome to apply to our to our cyber security program with a link to an application form. Uh but the thing you have to realize there and was I was mentioning also earlier is when somebody is penetrating in your infrastructure they start to what we call move laterally which is usually you have an entry point but there's this destination is quite far so they kind of find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. you kind of have to move fast like it's a matter of at least hours and even more minutes so that you can stop them you know as as soon as you can so that basically the access and the blast radius stay like localized um so you don't have time to apply for a cyber security program it's not the moment you want to fill in like a Google for something and just have someone you know take time to vet if you're supposed to be given access or if it's not or if it's too dangerous somebody interview you like just that's just definitely not the way this is going to work and the I think in the in the future world where cyber security is going to be a big topic and I think it will keep being more important topic it's a little bit naive I think just to think that every company is going to is going to be part of the same you know vetted cyber security program by just one of the two big labs I think it's it's like a little bit like crazy to think that you're going to have uh you know I don't know 100,000 vetted company that progressively apply so anyway in this case we say Well, we we we had to stop this now. So, we basically tried all the open source model that we had and and DM which is close to the state-of-the-art right now was just before Ky that this happened now probably Kim K3 is the closest to the state-of-the-art but GN 5.2 is actually really good as well. Uh was just uh very good to process this and basically we we could extract some of the pattern and we could understand basically the hacker here was trying to to to access uh mostly the data set. So we just reboot this this this this part of our infrastructure. We have a very simple uh we have a very like flexible way to to respawn uh codes and nodes. So so this was how we we just ultimately stopped. Um but I think yeah it was very ironic but because I think one year ago uh roughly around the summer um most of the discussion around open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe and that seems very obvious in the mind of everyone and there was this idea that you know uh uh if we only have closed source will be just fully safe and if we only had open source like we'll be very unsafe. Well, everything that's been happening in the past months has been I think um basically contradicting this very simple mapping. I think closource model are less easy to control than we think they are. On the other hand, on the other hand, open source model for some reason it might change in the future, but currently are not trained so much on actually um I would say bad behaviors like that. So they're they're they're pretty bad at at cyber attack or like deceptiveness if you if you look at that. So it's a little bit hard to understand exactly where does this come from in part because while open weights model tend to come with a very extensive technical report that explain how they are trained closource model we can only try to guess. So it's quite funny also as well because I was seeing a lot of people trying to understand why mythos was behaving like that and they were using kim K3 as an example of how this should be trained. So they use like this supposedly like open source and say very bad dangerous thing to try to understand how why the good thing that we don't know anything about is is being trained. Um but yeah that's how the world is right now. So I think it's um it's interesting more generally I think in the future and to be honest and I I'm I'm I would say I'm careful uh like optimistic around that and I'm not specifically against closour model or or ultimately pro opensource model. I just think both of them are necessary just like we like to have closed and open source software. We like to have I mean I'm happy to run on a Mac right now which is you know kind of a mix of both. It's based on a Unix kernel that was open source but then there's component of lambda closed source and that's great because very I'm also very happy I'm not on Ubuntu right now. It's super easy to record this podcast with you for this reason. Well, my former Ubuntu spent a lot of time like many of us just connecting a microphone or whatever and trying to watch a movie with my girlfriend. Girlfriend was when are we going to watch the movie? I was like I'm almost there. I'm always there still just installing the code or whatever. So uh I think both of them are advantage and equivalent and and drawbacks and I I think the world where we are where the frontier is closed and and there is like not too far open source model that you can use as well for for many things is actually a pretty good pretty good um uh middle ground solution

Matt Turk: >> to make sure I I got it right. So uh what you're saying is that um to some extent it's open source versus closed source uh but it's more the the state of the world as of right now like the way the current closed source models are designed and the current open source models are designed versus anything that's intrinsic to uh one or or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to cyber attack. Is that is that correct?

Thomas Wolf: >> Yeah, I think in many for many aspects I think the closed open distinction is almost orthogonal to the safe and safe. People don't understand that you know easily because it's easier to do bad mapping than try to understand the subtlety but uh that's the case like you can have very safe things in open source fun you can have very dangerous you have different balance of of of safety and dangerousness I mean to take one example like last year and at some point like a lot of the discussion was around fake news and writing fake articles that used to be a big big misuse that was the main one people were talking about right today there is of course there's a lot Perfect. There's a lot of AI slop. We even have a new word for that, right? It's even it's even hard to find like fully human written articles. All of that is or like maybe not all let's say 90% to be fair is made by closed source model, right? And there was a time we were like oh if we have open source model everyone is going to generate articles everywhere. We could not control these articles like we could not control people saying newspaper. Well the reality is that this was a very I think very wrong view of of a danger that would be specific to open source model. that was a w more wider danger around source of proof on the web. I think that's just one example but I think the same is true like I think both closed source models and open source models should be more aligned like right now closing people. I think this is a huge problem and it's kind of question how well we are able to um align them uh toward not doing things that obviously are wrong and I think like honestly like lies and all this should be kind of obviously thing that model should not be used but this might happen as well for closed source for open source model like in a few months and so I think this is just generally uh this risk um axis is just kind of autominal to the question whether it's open or whether it's close. We should just find way to solve that for both clos and open source model.

Matt Turk: >> Okay, great. Um, and to to make sure we cover it since we alluded to it at the beginning of this conversation, the AI uh SI incident uh which um just happened and that you find um you mentioned hit close to home. Uh can you unpack for us what uh that was in a in a couple of minutes and why that's a big deal?

Thomas Wolf: >> So this happened in a evaluation of the latest generation of models. So both GPT 5.6 sold and mythos 5 by an independent institute in the UK called the AI security institute. So AISI which has been there for quite some time. I think they are pretty competent. They were created uh for the first AI summit back in Blanche. Um, and what they do basically is they task again they task model with solving some form of like cyber attack challenge and they measure how good the models are are at doing that and some of this again might be possible or not and and and so here the the setup was a little bit I would say the setup was a little bit suboptimal uh in two aspect which is the but maybe let me first talk about what happened. Basically the model was tasked to solve this attack this like to attack and to penetrate this this subn network. And what it decided to do, it decided to um get one of the maintainer of a library that could be used to operate this active directory to merge like malicious code with the idea that if this malicious code was merged by this maintainer, then there would be an update uh at some point on the the software that was used in the subnet it was attacking and then this would give it um entry point. And the way he did that was actually trying to social engineer the the maintaining merging. So he created fake accounts, fake GitHub account that uh came commenting on the pull request and said, "Oh yeah, you should really merge this. This is solving like a big problem I also have." And then when a human stepped up uh trying to say, "Oh, this looks actually like malicious cut to me." He tried to kind of blackmail almost the human or to say this is not important or you didn't really understand. and then he acted tried to cover it traces changing the past message. So there was a whole like social engineering aspect here basically trying to get the maintainer to to merge this malicious code and this was uh this felt like a distinctive distinctive step up from just pure technical press. So you can attack a stand a sandbox for instance by you know trying everything you can which feel like very mechanical and machine like to me but trying to attack the sandbox by uh blackmailing or or mailing or social engineing some of the maintainer that's that's a very different level I think of of of thinking and and for me of course like as myself an open source maintainer I've been often in this case where I have someone opening pull request and then people pile up commenting this pull request and I try to really understand was this. I felt I felt very like uh I could have been the target of this side quest of the model basically. That was very interesting or at least very very scary. Um but to be fair uh there was a couple of like misconfiguration. I mean some of them are by design. So when um this team run this type of evaluation they disactivate the cyber security guard rails of course otherwise the model won't do anything. Um and there's there's basically three levels. So let me try to explain a little bit like how you can prevent models for from doing bad things. The first level is you put it in a sandbox which is it doesn't have access to the internet and here for some reason they didn't want to do that. Uh I think that might have been a mistake and and the idea in their mind was we want to let the model have uh as much um like potential for inventiveness as possible. So we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So that was this is not super clear but yeah and the second thing is uh then you have some guard rails that basically analyze the prompt and say no or yes to some prompts. So here obviously you want to disactivate this one otherwise you just can't evaluate anything because they will just say no this is cyber security challenge we don't let the model do that. But there's another level that's roughly there's another aspect that's roughly at this level as well which is you should you can analyze the the reasoning of the model. You can process the chain of reasoning and try to detect when something bad is happening. And here they didn't have something like that in place. Um I think the main reason is probably that until recently and until the open AAI hugging face attack, people had maybe a little bit of a um limited um understanding of how good this model might be or how far maybe more how far they might go in terms of side quests on the on the trajectory of solving this challenge. So I think people were still a little bit n in that. So I would expect that in the future they will have way more monitoring and sandboxes. But the the third level there really deep is that the model even with everything disactivated guard rail sandbox in my opinion should really be very reluctant to tell lie to a human and to try to blackmail or deceive any human. I think this is just generally in any case that's a behavior you just it's hard to find justified in any context. So they should be very deeply in the model. There should be something that align it and that make it uh say oh this is actually something I don't want to do just like we to be honest have kids and just like the thing I teach my kid which is you just shouldn't lie that's not a good thing in any context. Uh so yeah that's that that's that's the that's the deep question and maybe last year I would say we would have thought that this was pretty good and we had all this discussion around constitution model specification and most of this model specification or constitution say you should be honest you should not tell lies to a human you're like like to to to any participants and we thought that maybe this was kind of a solved problem and what we see today is it's not short as solved as we thought.

Matt Turk: >> So to play it back, so you're saying there are those what you call the the three walls. Um there's the the the sandbox, there's guardrails, and then there's the the models alignment. Uh and I think you said the the sandbox, and the guardrails only work as long as we humans are smarter than the AI, but uh that may only last so long, and therefore uh the the alignment ultimately security is an align fundamentally an alignment problem.

Thomas Wolf: >> Yeah, I agree. And that's something as you can understand that's both the case for open source and closed source model. Ultimately, you want them to be aligned. Uh open source model have the specificity that you may choose on you may control where you want to run them. So it's harder to make sure everyone use sandbox and guardrails. I mean we can have definitely uh we can have some lows and regulation around how you should deploy this model. I'm pretty sure we're going to have at some point but it's a bit harder. alignment is really the the critical part uh in my in my opinion for the 21 I mean sandboxes what we've seen this year and we've seen many example they are pretty much easy now for these models to escape from it's it's really hard nowadays to say I'm going to make a fully you know full proof sandbox I'm sure it's going to be resistance against all the coming generation of models I think we should assume that sandbox will always have a small probability of of not containerable. But even below beyond that, it's also we can't air gap the world that you can't just unbox everything. Things have to talk with each other. Some you we want our models to be able to do web search. We want them to be able to do stuff on the internet for us. So we can't just unbox everything. And so what remains to us before is just guilt on monitoring. And I think these these are like uh for some reason as well as the model capabilities become really good. Also as the model I'm a little bit worried that the model start to talk in a form of English. I mean I'm French so maybe it's partly my problem but I feel like they start to work to talk in a form of English that's harder and harder to process that's very content dense. Uh you know they start not

Thomas Wolf: >> Yeah. And I mean it's just a very big simplification of all of that because there's so a lot of research that say that basically you can't read everything chain of thought. Not everything is explicitly said but I would say more generally I think just relying on being able to read the reasoning trace to understand fully what's happening. This is also not fully bulletproof in my opinion and I take this scenar as an example because I feel like a lot of people start to have a bit this problem with blood cuts. feel like something that people can understand but more generally I think longer term it's really hard to fully fully rely on this only and the same I would say that you could say maybe I just don't care about understanding exactly what's happening and maybe I can just look at the tool call and if I see a tool code that's bad I can just block that and I also think this is uh probably not bulletproof and the way you can see that is um probably three three big things that are compounding. The first one is we start to use this model for many many many things. So as we talk right now I have a model deploying a box uh somewhere and VPS is doing many different calls. They all going all direction. I also have other models that use uh that I use for administrative task. So just the range of tools that this model are using is really really large right now. So it's getting harder to say you're allowed to use that but you're not allowed to use that. And this is the bad thing. It's getting very hard as we deploy them wider. They also work on larger and larger task where they use many many things. So sometime I ask it to do some coding but the coding involves searching on the web and maybe doing these things and actively using many things which are not just purely writing code and running some tests and this test might be quite expensive and involve other software. So the the frontier is is is much more blurry and then you also have this swarm of multiple agents. So it's also harder to say it's all in one context. it might be split between many contexts and maybe this sub agent is doing something that looks pretty uh inocuous but maybe combined with these other sub agents are actually not so great because you know so there's all of these things that make it I think really harder to be uh fully fully sure that you have an exact idea of what's everything what what like every swarm of agent is doing. You probably have to really zoom out at the very global level and say and see what's happening in every direction. Um but that's that's that's a whole that's a whole monitoring setup that we need to be right now. So yeah, this to be said I think as we as we deploy how we use this modeling very complex long-term like uh parallel setup I think it's going to be harder to just say I can look at the tools and I know if it's doing something great or not. Is there something uh fundamental to the way uh those models uh are currently trained? So the very frontier uh that makes them more likely to um go onto those those uh side quest and potentially uh create harm. I mean analogy that um people have been talking about for a very long time is the the paperclip paradigm which I think was u you know Nick Borstrom in 2003 saying that um AI may harm us not uh because it's trying to harm us but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal. So are we in that world and if so what causes it? Yeah, that's a bit what I hint. I mean, it's always hard to be fully uh affirmative there for one for for one reason which is that we we don't have full full visibility on how the frontier model are trained right now. What we know though is we we moved from this pure like human data you know partic that was first just pre-training on human data and then also aligning with like human preferences that was called RLHF where we had a lot of human in the loop and human data to like uh recent where models are trained a lot in like this uh RL VR so basically full RL environments where they're allowed to explore and they just have one goal which is can be like make this cut like you know pass this test or can be capture this flag in cyber security or can be install this. But this goal is is a very like um is a goal that's unrelated usually to any human preference or any you know moral or like whatever deceptiveness and it's a goal that's very just called like true or true or false code and and we move to a padding where this is uh increasingly a very very large part of model training. So this was this result this uh recent evolution and that's also the parting where can happen what you were saying which is uh you can have like reward hacking this type of thing which is uh we move to a padding where this is uh increasingly a very very large part of model training. So this was this result this uh recent evolution and that's also the parting where can happen what you were saying which is uh you can have like reward hacking this type of thing which is uh you actually solve the problem but not using what was expected uh for you to use. So it can go from pretty benign one that like what happened for open AI for instance I just try to get the answer from somewhere I'm not allowed to or to like more harmful one where you actually have some impact on the human can be a GitHub maintainer for now later can be can be another type of of human. So it it seems to be it seems to be way harder to make sure that what we had kind of solved or at least what we were doing pretty well on the full like human-driven parding also apply in this kind of more like machine machine driven parig I would say but that's yeah that's that that's a hint I think uh but definitely it seems like when when Bstrom wrote about it in 2003 seems a little bit like you know futuristic definitely and maybe something that was like a little bit crazy and this would not happen. Uh but today I mean it's pretty clearly something that that that that happened and it's it's a it's the best description of what we've seen the past two weeks this type of thing. But also we can see that both frontier model and I take GPT 5.6 and mythos doesn't seem to have at all the same type of behaviors. So there is differences here in the effect of you know they they are not trained exactly the same way and they don't behave the same way. So that's that a pretty positive sign in a way that means that we can actually probably tweak this to to go in the right direction. But the best way would be to know a little bit more about you know how they train or what they try and what doesn't work or what should work. I think that's kind of the idea of open science and that's something we advocate a lot at any phase.

Matt Turk: >> Fascinating. And uh so speaking speaking of which let's uh uh zoom out a bit. We'll we'll go back to maybe some of the implications in terms of policy uh of of all of this. Um but um since you mentioned the ever so important role of open source AI uh where what what's your sort of uh quick highle take on where we are in terms of the state uh of open source? So there's been uh this race between open source models and closed source models. Uh depending on who you ask at what time, open source is about to catch up. Uh sometimes open source is just as good. Um some people say no. What what is your sort of uh uh sort of realistic pragmatic take on the current state of open source AI?

Thomas Wolf: >> I think it's very strong. 2026 is maybe the year of cyber security, but that's also very clearly the year of open source AI. Um I mean first all the doomer that were saying you know open source not going to be able to stay close the frontier I think they're at least up to now they've been pretty wrong I mean it's also clear we don't we don't have any mythos level open source model for sure but we definitely have models that are not super far from opus category or depending also it's more and more spiky so you need to find your spike some people stand on some spike or not but typically uh they are definitely pretty good right now and they've been uh they've been um uh following rather closely the the frontier at least on the benchmark. It's also not like it was maybe in the early days uh benchmaxing like we say when your model is only good on the benchmark but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their in their good capabilities. So yeah it's very good. I think there is two two strong trends I would say I see right now. Um the first one is um I see a move in companies to try to want to control their costs. So there have been increasingly discussion there. Um maybe 2025 was is the year of tox token maxing where you know you could say hey you should spend as much in token as you're paying your employee. This year people realize that actually we spend a lot of money on salaries. So if we spend the same exact amount that's going to be basically doubling our cost. So yeah, which seems pretty obvious in retrospect, but that's quite true. And and not every company can assume to double their cost. It's also pretty stupid right now to just say we're going to fire everyone and work on on agents. We all know they are sometime go you know uh not directly in the direction we want them to and you need human to to shape them. So I I think a lot of company are trying to find what we saw a lot which is kind of a fusion or router model where you use the frontier for something but you find the smart way to you know gracefully uh fall back on less uh expensive models for simpler task when you don't need to even in our daily life right now when you code with a frontier model in many case you ask it to spin out sub agents and they might be used like lower performance model can be you know solved using terra luna can be fable using Opus, haiku, sonet or haiku. So I think everyone's getting even at the frontier and the closest model world getting used to employing different type of models and then it's very natural that some of these could be really like uh very cost effective agents and most of the time you want to go to open source in this case there's lot of cases as well for like the the the strong ecosystem of inference provider fireworks has been has been on the role like you know all all the clouds nec every every cloud has been like increasingly have like these crazy revenue curves that have just basically translation of people using more open source. So yeah, I think open source having a very good times time in terms of uh staying solidly close to the frontier and driving more adoption. Um so and the the second big trend I would say is uh it used to be only China but there is also now a range of promising company in the west. Uh it used to be that only meta was supporting for everyone. I mean meta kind of left the field but they might come back who knows but yeah this void was filled rather quickly by companies like reflection uh thinking machine RC mal is supposed to open source the model soon. Nvidia themselves training models and and actually training very good models right now. Um so yeah I think there is a range of very promising teams there that um I'm quite bullish on to be honest and maybe between even the time we are recording and the time you actually release podcast we might see another couple of very nice western model release. It would be great like open source doesn't have to be synonym with just one country making them. could be like a global thing for sure.

Matt Turk: >> On the first point uh on the sort of enterprise adoption of open source AI um there uh was a moment in time when people associated open source to free uh but I think the world has quickly learned that uh while the models may be free to download uh deploying them and serving them is certainly not free. what is your sense of the cluster advantage uh of open source in reality in the enterprise?

Thomas Wolf: Yeah, that's a that's a very good note and it's the same uh very the very stupid mapping open source equal free is also wrong there and it's much more subile and like you have advantage or not. I think the nice thing about open source is you have quite a wide ecosystem like the the entry barrier to um be a cloud provider is pretty low. So you have a lot of competition there on how you know what's what's going to be the cost per token and this is a mix of many things right it is uh how cheap are you renting your data center how cheap can you buy your chips and and here actually we have also new chips company who are going to come on the market and this is going to be very interesting to watch uh and so basically how how cheap is your hardware and then how much you can optimize the model can you can you quantize them so for inance the model we use to counter um OpenAI intrusion was GLM 5.2 that was quantized by Nvidia in in in four bits and this is to make it faster and smaller to run. So you have a lot of like strategy you can use and explore to make this model cheaper. Um, but it's also true that they don't have to be cheap per se and also that the closest model may be in a way subsidized right now like the the like the amount the number of token you get for your $20 uh charged subscription might not be the full price that they actually pay for your token. So, so there is this kind of complex uh balance around costs while maybe your cloud like influence provider for open source model doesn't have so much leverage that you can lose money on on kind of a subscription business. So we'll have something more complex. Um I think ultimately

Matt Turk: >> all subsidized by venture capitalists.

Thomas Wolf: Exactly. We have the same like thing we've seen in many fields, right? But we also know this is ultimately a little bit temporary. So you should not fully rely on that. This is not the equilibrium price of the market.

Matt Turk: >> Yeah. Sort of the Uber phenomenon, right? Like cheap Ubers before the IPO and expensive Ubers. Since

Thomas Wolf: >> I think you want to keep the ecosystem alive for the post uh VC market.

Matt Turk: >> On the second point, uh China versus uh western models does provenence actually matter. What is the latest thinking in terms of um if uh your uh model is completely open and then you know sort of exactly what's what's in it uh then you shouldn't worry it's completely safe. Is there still a little bit of um thinking at the back of people's mind that there might be some back door some trickery to Chinese models? Is that still a current question?

Thomas Wolf: >> Yeah, I mean I would say um that's a good question of course and and the the the thing about open source it doesn't like open source doesn't know any border like you can't really uh keep your open source model you know restricted to download to just sub part of the earth so by default it's kind of a global thing and then you want to maybe understand the provenence and the supply chain um so there have been some work on this definitive steeper agent I think it's it's probably under search I I think there's mostly mostly work by anthropic on that and it should it should probably be reproduced be explored deeper to understand how much it's possible to you know implant kind of a back door that would be triggered by a prompt. Um but also to be honest even for the closest model right now we have some struggle controlling them fully. So I don't think we're very clear like I don't think we're very clear on how we control even the closest model at the moment. Um so yeah I would say it seems to me that it could be a possibility for sure. Uh we have not seen any any indication of that. Uh it's very easy to fine-tune them and you can change quite a lot the weights as well. So right now if you pre-train and post- trainin a model for longer you you very likely change quite a lot of the weights that it has. So I think there's a lot of way to circumvent that. uh which means that at the moment I'm a bit less worried about that than maybe just a pure like reward hacking that we actually have already just seen happening like so but yeah yeah yeah I mean just generally on sovereignty I think um sometime people are a little bit confused there I think I think the most important thing is who had the hand on the switch to trigger not your intelligence access so I think when people really realized that was earlier this year where the US decided that fable was only accessible to US citizen. I think at least in Europe and Asia that's really the moment we saw government understanding that someone had a trigger and they could say no you're just not allowed to use this intelligence anymore like this this token I think that's the critical thing you should at least for me that's really the level one of sovereignty which is can someone just decide that you don't have access like someone I mean like a country basically level thing and this has two thing this has the API access so it can be the data center so if you don't have the data that means also like another country could say these data center are not accessible anymore to this citizen. So I think that's really the first thing you should see and then there's a lot of like future question but like you should build your stack keeping that in mind. So an open source model you can download it nobody can like no country could take it out from you once you download it you can fine tune yourself if you operate it on on your data center like local ground data center I feel I feel like you start at the beginning of a sovereign stack and then there is of course a lot of question back doors and more complex stuff but that's kind of the basic minimal thing in 2026

Matt Turk: >> and as an open-source optimist um do you do you worry uh about the motivations for western open source to thrive. So uh China has a clearly a geopolitical uh motive behind uh being at the forefront of open source. If if you think of the west then so you mentioned Nvidia and we had Brian Kenzaro um from you know the whole Neotron effort on the podcast a few weeks ago. So clearly there is a uh motivation for Nvidia to do this which is they sell the chips so having a thriving open source model makes sense. Um but if you think about like everybody else it's sort of unclear why uh the uh why western companies would do open source this reflection might be the exception but the models haven't come out as as far as I know. So do do you think about this like motivation and and what that means uh for the future of western open source?

Thomas Wolf: >> Yeah, of course. But that seems pretty that's pretty that seems pretty obvious to me that we actually want that and and I think a lot more people have incentive than you may think. Like I think I think as as soon as you're interested in having a thriving business ecosystem with many companies being able to to actually use AI and not just as seen wrapper around another company but as real like AI builder um I think you you want some open source model. So that's that's one of the reason the US government just recently say actually we want to keep open source you know like striving and basically the idea is that open source is one of the best way to get new business. So it can be for many things can be just uh because it it allowed you to not just end up with oligopoly basically of two company which I mean we've seen many case of oligopoly in the past it's not always the best thing for competitiveness for price for you know there's many danger with that I think just rushing in the direction of say we we have made we have we've taken our winners and these two companies are going to build AI for everyone else I think that's from a business economical market side of view I think doesn't really seems uh optimal to me and then it's also limit a lot invention. So just to take one example there's a lot of potential right now in biology there's a lot of new life science company there's a lot of company wanting to explore that um because of the guard rails and because of the question around biohacking and using this model to generate like the access right now for payable just to take it uh is very very limited once you want to ask some biology question and so basically most of the life science company I've seen who were using this model had to switch to another option they wanted to be able to process anything related to biology. So if you're a new company exploring something that uh the big labs are not currently uh you know uh at least uh exploring or they they don't feel like there is enough business potential maybe to give full access to everyone or I don't know exactly um but basically uh you cannot really use that. So either we say all life science is going to be built from now on by anthropic, open AI and Google maybe meta or we say we want a big ecosystem around there. I mean my my personal opinion and I'm biased but is that you don't want too much concentration of power around key technology and I feel like the more people can you know invent the more we have a diversity of new ideas diversity of new company but also as investors right and you're investors you know what I talk about like let's say you could only invest in anthropic and open AI that's a little bit sad right it's a little bit boring it's only for growth stage but you want to invest in your company and you don't want to invest only in these very thin in rapper you want to invest in your company that actually able to build AI and most of them uh and we see them at face most of them need open source models another bit example is all the thing around gaming video or even robotics most of the time what they do is they start from an open source model and then they fine-tune it on some robotic data for gaming for instance they'll take a video generation model that's open source they need to do something that was not um predicted by the the video generation startup they need to add actions. So what they do they they find tune with action in the loop and video and that's how the first I think interesting like real time gaming company started basically. So a lot of the time open source is your easy way as a new company to start to have your own mode to fine tune on your own data to be able to start to build your own AI and not just to basically sell your training data back to the model providers which is I think always dangerous because a lot of these model provider might at some point want to enter your field if you're basically selling them your data and this happened in the past already in legal in design in like any field I think

Matt Turk: >> to play it back so your take on the July 24 letter uh industry letter on open weights and American AI leadership which was also Jensen's first tweet ever that that that that you guys that you guys signed obviously uh is um u partly uh it's important and partly a resistance to just an oligopoly um structure that is being put in place. So it's both it's it's good for the world but the strong economic motivation behind it. Is that is that fair?

Thomas Wolf: Yeah, I think I think it's u everyone believing in inventiveness and being able to create new things also in the AI world I think would want would want a part of open source access just like the same you know if every code was closed source we would not have the thriving coding ecosystem we have right now right that's kind of obvious like everyone would have to work at one of the large closed source software company if they wanted to create any software that doesn't seems even really possible to have all the inventiveness and creation we've seen in in the software industry. I think the same happen in I don't want to dismiss risk and I fully agree we need to work on alignment and I say that's I would say for now open source model being under the frontier I think this is maybe less important than some what people wanted to say

Matt Turk: >> maybe to to take a step back as we get near the end of this uh conversation and sort of get a sense for where the world might be going from your perspective in your in your post yesterday about AIS you talked about um a new wave of labs. So in particular, you referenced the news of the Jet Dean new company that he just announced yesterday. Yesterday was a bit of a crazy day in terms of like everything that that came out in the world and was and and was was announced. Um so uh the the point being that this company is explicitly rushing toward recursive uh self-improvement. Uh so in the context of everything we we described, how nervous uh are you about um this evolution towards um self-maintaining, self-developing recursive AI?

Thomas Wolf: >> Yeah, that's a good question and definitely as a as a scientist researcher, I'm I would say I'm I'm very interested in the idea. I feel like there's lot of this uh super intelligence that that they want to tackle, you know, solving um crucial challenge for humanity. I think this is this is a great goal. Um I would love to see AI having making more scientific discovery. I think would be probably the most beneficial thing that AI could bring more than just AI slope everywhere. So I think I'm I'm very optimistic. I I just feel like uh I would say the past few weeks I has raised a little bit the question of how how good are we at aligning this model. So like in French we say we don't want to put the uh the carriage before the cow like we we need to go in order there. So um yeah I would say right now the the good thing is most of this um I would say seems to be internal research lab hopefully they they do they do good security around what they do before they deploy uh some of their product they think how it's going to be used and they they have good good feeling around I would say good thinking around the the social impact of what they are building um but yeah I still I still think that we should try to understand really well how we're going to how we're going to deploy this model in in the human world I would say

Matt Turk: >> right but you you're not in the camp of the um petition that that came out so I think 4 days after the open weights letter um that um you know Nvidia did that we were talking about a minute ago there was a different letter that came out with 1100 people and and and this time both anthropic and open sign that asked government to help deliberately pace the frontier of automated AI research So basically the industry kind of asking um for a slowdown. Are you are you in that camp or you think that's just not the way it works?

Thomas Wolf: >> I did sign this letter. I agree. I agree that I think we should uh we should go there. I'm also and you probably have the same feeling as as an investor. I have the feeling that even if we stop right now, we would still have quite a good companies we could build on top of we have what we have right now. I feel like there's a lot of things we we can already do with these models and that are already extremely interesting. I feel like there's a lot of things we need to understand and we should do in terms of open science and sharing how they work. So I'm not I'm not in the camp of we need to rush really quickly right now. The main question is uh if we want to slow down a little bit also that would be great because maybe then we don't have four announcement per day that we need to mix in one podcast the next podcast tomorrow. Uh yeah, maybe I can take one day off holiday in the summer. But no, I I think the main question is can we do it right? That's the main question here. I think a lot of people would be fine uh with AI going a little bit a little bit slower, being a little bit more open, being a little bit more caring, a bit more like reflexive and and and trying to understand better, you know, how to how to do that really well. Um but the main question how can we negotiate uh how can we organize uh a slowdown there uh without having bad incentives where where you know just one or two player not slowing down will kind of break the whole the whole effect of of having a slowdown and here I don't know if yeah I don't think the letter gives any incentives it might be it might need some some you know collaboration there was one one long blog post called AI 2040 I don't know if you read it that was also advocating for kind a careful slow down uh and maybe you know somewhere between uh we go you know full breaks out we go we go as fast as we can and somewhere between uh we regulate everything so nobody use AI which I think are both stupid solution uh but something around we we try to see if there is a way we could actually pace this a little bit slower I think I think that would be great I don't think we would lose a lot and I think actually also in terms of company creation and all of that we could still have a lot really great things happening. Um, but yeah, I'm I'm in this uh I'm I'm actually sympathetic to both this and and open source. I don't think open source has to be accelerationist per se. Um Dan was saying this is decelerationist. I don't think it's also this deflationist. I think these are these are also you can be pro open science. uh you can be pro openness and you can also uh think that actually we need to understand how to train this model well and we need to actually uh being able to do real science right now

Matt Turk: >> and uh you're not worried about this being uh an attempt at regulatory capture where like the top two private labs um are effectively trying to figure out how everybody else can slow down. I mean, you mentioned the risk of like not everybody just complying um but effectively um freezing the market structure around who's a leader and who's not.

Thomas Wolf: >> Yeah, I don't I don't think it has to be. Uh I feel like you have definitely the same path that's actually fully accelerationist where you decide a couple of company are racing against each other and you regulate all the other outs. I I don't think regulation has to be synonymous with with slowness or not. Um and definitely the question is more how you put that in in um in action, how you actually, you know, put that in in practice, how you deploy this in in regulation or cooperation. I think Demis also had a pretty nice letter the other day uh before he stepped down or up as as a chief scientist still we understand where he's going to be now but like uh his letter for basically international collaboration was also very much pro open source in some aspect uh I think you can have a slowdown that's very open source that's the one that's the one I would love to see which is you we slow down and we use we use the fact that we slow down to be able to actually share more things and I feel like a race dynamic is usually more in terms of closing the the doors of the labs, right? So to me a slowdown is probably more the opportunity to open but of course I mean if it turns out to be mostly a way to uh just solidify uh like we were saying a cartel or oligopoly of just two company I'm I'm not very excited about this direction.

Matt Turk: >> Wonderful. Well that feels like a wonderful place to leave it Thomas. Uh thank you so much. This was absolutely fantastic. really enjoyed it. Um, and appreciate your taking, uh, some time to speak with us, uh, in the middle of your time off. Um, so, um, thank you so much. Appreciate it.

Thomas Wolf: Thanks, Matt.

Matt Turk: >> Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.