西蒙·莫 (Simon Mo): The fun thought experiment is if GPUs dropped in price by 99%. Then do we get back to a real open-source world? If moderation is never solved in the future, people will go to open way by default because that is where you know for sure you can control your guardrail for trusted use cases.

肖恩 (Sean): Can you talk about where VLM sits in that stack?

西蒙·莫 (Simon Mo): VLM is a inference engine. It is kind of like databases and operating system other critical software to power AGI. Nvidia, AMD, Google, their newest chip will make sure VM can run on them. And in a lot of cases, they use VM as a benchmark. We're bridging almost a 10x gap. For proprietary model, there is a regular mode and fast mode. But for open weight, every provider can offer potentially even 10 different levels of speed.

肖恩 (Sean): 5 years from now, open source AI models, have they closed the gap with frontier models?

西蒙·莫 (Simon Mo): Capability wise, I don't really see a big gap. Not even today because

肖恩 (Sean): today we're here with Simon Mo, co-founder of Infact and a lead maintainer of VLLM, the open-source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Bournestein, an A16Z general partner. Simon, Matt, thank you so much for joining us. I think first we should start uh with open-source AI and kind of the more recent history of open source AI. So VLM actually has its origins kind of back in 2022 pre-CAD GPT and your team set out to make a slow open-source demo faster and instead just found this pile of unsolved problems. So can you talk about what made serving an LLM so fundamentally different from the ML workloads everyone already knew how to run?

西蒙·莫 (Simon Mo): Yep, good to be here. So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs and it it is a computationally intensive process that will require a lot of engineering and ensuring that for each request user can see the LM's response quickly and efficiently. So this typically means we need to handle uh differences in input distribution how long each request is output distribution which is non-deterministic and batching and scheduling a lot more in the at the core of the inference engine.

肖恩 (Sean): Yeah. Um and I so so the team the project has been around for about four years now. Um but the company is a little bit more recent. So I want to throw this out to either of you Matt. I know you've known the team and observed the team for a very long time. So at what point did you see this sort of transitioning from being, you know, a a muchbeloved open-source project to critical infrastructure and then a company?

马特·伯恩斯坦 (Matt Bornstein): Yeah, I mean I think you have to go back a little bit, right? O open source um was was the norm for AI models early on, right? I mean, we literally have this company called Open AI, which,

肖恩 (Sean): you know, it's come a little bit of a joke. It's not as open as it once was, or not nearly as open as it once was, but uh but but early on, all all the frontier AI work was being open source or or at least released into open weights, which is a little bit different than true open source. Um and and people could mostly run these models sort of on hardware or computers that they already had. Um I'm I'm curious son like maybe you remember like what was like the first model that like you actually needed to go out and get special software and like a you know a special set of computers to be able to run

西蒙·莫 (Simon Mo): probably BERT and before that it was like ResNet for computation like uh like images computer vision classification. So, ResNet already need to run on uh Nvidia K80 which is kind of one of the first

马特·伯恩斯坦 (Matt Bornstein): on AWS and other places and and but way over but even at this point ResNet you can still kind of run on a uh commodity even CPU devices it's just very slow but for Birch where running at it is like wow you have to run it on GPU to make it anything faster and efficient for anything translation any task so that was like before 2020 even. Yeah,

肖恩 (Sean): it's so funny thinking about this. You're I'm like all the memories are flooding my head. Like hugging face had like a thousand BERT variants on it. You have to go find like the right BERT variant for your particular task and Yeah. And you're right. You had to

西蒙·莫 (Simon Mo): some people I guess had their own GPUs and could run it. But but yeah, a lot of people had to go, you know, provision cloud and stuff like that. That's really funny. And so yeah, so so look I mean um

马特·伯恩斯坦 (Matt Bornstein): you know BERT was an early language model um that uh you know newer models are much bigger, much more sophisticated, take up a lot more memory, a lot more compute and and and and um you know so VLLM really from the early days right was about running these you know more powerful models that um you know that that that you couldn't just sort of do it you know figure it out on your own.

肖恩 (Sean): Yeah. Um, and I I think like to get us to to this present day, you know, um, I I I think it would be good to to talk about kind of at what point it really became critical infrastructure for these, you know, even larger open-source models and when did we even start to see these larger open-source models kind of come into the field? What we really see the criticality of

马特·伯恩斯坦 (Matt Bornstein): Yeah. It's sort of it's sort of interesting from a startup standpoint. um uh you know like like you mentioned sort of GPT3 or like early chat GPT um those closed source solutions were starting to become critical to like a small group of people around that time

马特·伯恩斯坦 (Matt Bornstein): and open source existed but it was a little bit of a curiosity or sort of an enthusiast thing um a as the frontier has expanded you know particularly with closed source models

马特·伯恩斯坦 (Matt Bornstein): more and more open source has been dragged in as kind of like critical behind it like if that makes sense like like at any given point in time including Now I think models from open AI and anthropic are are kind of more widely used and more critical kind of in general than than open source models. But I do think we passed a threshold in like

马特·伯恩斯坦 (Matt Bornstein): I want to say about a year ago

马特·伯恩斯坦 (Matt Bornstein): where a bunch of smaller companies or like new application companies as they were trying to figure out how do I really build an AI without just being a wrapper on top of open AI. The answer to that question turned out to be open source. I mean this is what cursor did. This is what sort of Decagon and Harvey are are in the process of doing now and and a bunch of other like really really strong application level startups sort of made the determination we can't build just on closed source. We need to do our own mid-training our own post- training our own sort of inference and deployment tricks and all of that means it must be built on top of open source like you know the the closed source vendors won't won't give you the access to do this. So my read is like kind of a yearish ago. Open source became really central in a way that's not always visible because it's it's deeply embedded in some of these products but you know some of the most innovative products and applications now um you know really depend on this very deeply.

肖恩 (Sean): Yeah. Yeah. And can you talk about where VLM sits in that stack where where we do have these larger enterprise companies that are choosing to use open source models like where where does VLM sit in the stack for them?

马特·伯恩斯坦 (Matt Bornstein): Yeah. I mean you should like just about everybody uses VLM. You should describe it.

西蒙·莫 (Simon Mo): Just about everybody uses VLM. VLM is a inference engine. That means its job is to turn available GPUs into a running endpoint for intelligence. So that means it is kind of like databases and operating system other critical software to power uh this uh economy or power of the AGI that everybody really uses today to ensure they can have uh cost effectiveness, efficiency, reliability and also always staying on the frontier because for VRM we support more than a thousand model architecture up to today and a lot of those are proprietary But also a lot of those are openweight right and a lot of those model architecture when they're becoming transitioning from a research prototype to world accessible open way model architecture they are live on VM on immediately so that's what a process we call day zero model release and additionally VM also work closely with all the hardware vendors so that means across like Nvidia AMD Google and Amazon Intel and lot more their newest chip will make sure VM can run on them and then a lot of cases they use VM as a benchmark to make sure it runs well on them. So this kind of fusion of where models run and where hardware where it gets gets to meet the hardware is where the magic happen and this is where VM is

肖恩 (Sean): you've told me some of the behind the scenes stories like it's actually not easy these days model releases it's like a lot of human drama in addition to like technical work I guess are there any stories there that you think are okay to share?

西蒙·莫 (Simon Mo): Oh, it's actually a very fun co-design process because from model labs point of view, right, these are brilliant researchers who have built this model. Now their biggest question becomes how do we get this out of the world and make sure everybody's able to use it and run it well. And we have worked with model labs that are um very just because they just use VM already in production or in their research process. they will just done everything for you because this is a moment when we go to them it's like hi we're the VM team and we would like to support your open source model we would like to offer in a way this kind of open source but why gloss service to get your model running well on architecture and then you return and we get the model lab is like oh we got it working already because we're running it for the RL process here you go just review our code and merge our pull request and then on the other end we really have model lab that just don't know how this can work so

肖恩 (Sean): because systems is not like their core.

西蒙·莫 (Simon Mo): Yeah, cuz sistain is not their core and they have been training or maybe they have their internal inference engine that just don't know how it will adapt to the open way ecosystem. And by the way, this is also a very much a multi-party kind of involvement process. Every model is typically involves um the the model lab, involves the primary or secondary hardware vendors, involves us, involves hugging face who are the model format and like model hub vendors and then depending on the appetite of the model lab involves 10 or 20 different kinds of release partners. These could be inference clouds, these could be public hyperscalers, whoever is going to run this model and you want to them to ensure success the model are running successfully, right? So, uh even up to today if you look at the K3 model release is a whole partnership and a drive to make sure that the model is once the model is released because it's just a few terabytes of files sitting on the internet that people are actually going to be use it really really well. E even from the beginning of 2023 2024 if you remember when Mist draw dropped their first model they just dropped a torrent link for PTP and then everybody's like struggling and trying to get it up and running and then we're working behind the scene with MRO team trying to get the inference engine support working uh in VR and this one of the most uh probably early on exciting weekend that we are able to spend on this and then and after the weekend when everybody's trying to really analyze what's going on and uh Monday Tuesday we'll m and us just announced here you can run it on VLM successfully here and everybody will able to easily reuse a lot of the work and start building on top of it

肖恩 (Sean): that was sort of a fun time where like enthusiasts like me could just like scramble to like download the model and like get it running somewhere

马特·伯恩斯坦 (Matt Bornstein): I'm glad the professionals have taken over because it never worked like very well but it was it was like a fun moment in time

肖恩 (Sean): so so to bring things you know forward to the present. Um I think open source models and also you know distillation have have been in the news recently. Um Infact signed the NVIDIA open weights and American AI leadership letter uh that that was signed by also A16Z, Meta, Amazon, dozens of other companies. Can you just talk about um you know your decision to sign that and and sort of what what you were really kind of responding to uh in the market and and kind of in the news.

西蒙·莫 (Simon Mo): Yeah. So for us what really want to stand behind is open way absolutely matters in the ecosystem. The world cannot just be controlled by proprietary APIs and where open way open development and research of these models are blocked or banned right uh the pledge that infra sign up for is we want to help and foster this ecosystem where we are typically in little bit downstream of this ecosystem right inference engine are not part of the pre-training process nor the RL process but we're where the model actually meets the world And from what we're seeing, people are just really using their imagination and ability to materialize this imagination of open way model. They're able to leverage these open way models so much effectively. There's almost two pieces to this, right? There's like the the cost thing where it's like the closed models are too expensive and then there's sort of the control thing where I I I want to sort of be in control of my infrastructure and and and in control the model, right? if I need to extend it or or put on my own guardrails or anything. I I'm just curious, have you heard from

肖恩 (Sean): from customers? Like are both those things important to them or or like are they kind of willing to pay as long as they have the control or or maybe they're different use cases?

西蒙·莫 (Simon Mo): I think like it fluctuate over time. So control matters a lot over the last few years and then costs just start to matter over the last few months. So cost really matters starting from people trying to migrate off uh right uh very expensive coding plan and like uh every skyrocketing token maxing spend but control has always been on in the backbone of this is they want to even in a way to control the cost right but also it's about controlling the system performance against what they're paying for. So uh there for example for a voice agent company they want to control their own model so that they can make sure the model actually respond by their required time. So the customer when they're on the phone they can ensure the agent is responding according to a SLA and this sometimes is only you can do with your controlled intelligence um because you know the whole hardware you're running and whole system you're monitoring versus signing up for relying on your critical infrastructure with a proprietary API where they might go down any time or have violation of the contract any time.

肖恩 (Sean): Yeah. Simon, you also um to to go back to the cost point, you actually make the point um in an essay you recently wrote about the release of Kimmy K3 that actually the economics is besides the point. And it's actually, you know, in in the case of these, you know, just really really great openweight models that are on the frontier that were designed by really brilliant researchers. like like these models are in some cases just as expensive as you know the closed source models. So in those cases kind of what is the point of running them and kind of what do we learn architecturally in the in the course of running them.

西蒙·莫 (Simon Mo): Yeah. So first on cost it's not necessarily they are as expensive as a proprietary model but rather first the cost cost discourse has been discussed over and over again with even GLM 5.2 too a few months back. So, open wheel model are sometimes definitely a lot cheaper and but for this model there's a big sort of step change where we're bridging almost a 10x gap but strike somewhere in the middle where Kim K3 is not as expensive as plot or GPD soul but it is a lot more expensive than JM 5.2 to why is that and I do believe this is the point of where pricing intelligence with the market correctly and understanding where it is and but then the majority part of the discourse should be focusing on wow this model is bringing a opus 4.8 a level model to our own infrastructure that I can use, I can run, I can fine-tune, I can be able to understand exactly how many tokens do I need, understand the exact performance profile. The reason here for example is for proprietary model there is uh regular mode and fast mode and that's only the two switch here. But for openweight when you're running it, every provider can offer potentially even 10 different levels of speed going from like the slowest mode which can be a lot cheaper to um 400 tokens per second uh almost uh up to 500 in many cases that for some workloads and this is typically 2x or 3x faster than the fast mode out there today. So this kind of level of control even in terms of performance and then let alone control over how customer interacting with the model control over data retention keeping in mind fable doesn't have zero data retention policy and at least the a lot of the data need to be staying there and let it control security and compliance a lot more and yeah this is why I'm particularly excited about K3 not just from the cost perspective but a lot more on bringing this level intelligence to something people can own.

肖恩 (Sean): In terms of, you know, calibrating things like speed, calibrating other things just sort of on the back end, what needs to happen and and kind of what are you seeing your users do and like like who who is being really clever about this?

西蒙·莫 (Simon Mo): So, we do see users are able to get the maximum benefit out of this model when they enable fast mode. Like what I'm talking about here of course is V on its own fast mode getting up to 400 and 500 tokens per second because it is really a big step change from like the especially when developer interacting with uh the model they can see oh I can really just get my task done faster here and the model are not no longer stuck in thinking rather it is just executing executing interacting with the environment so for premium developer blocking focus task we're seeing is very benefiting but also K3 are just be able to uh have the ability for people to modify it and fine-tune on top of it allow them to make it better for their own workload and this is definitely happening today as well.

肖恩 (Sean): Can can you just explain what the um licensing term is for the most recent open source models compared to the past and and and like why you think they're doing that?

西蒙·莫 (Simon Mo): Oh yeah. So historically the um the open way model are just like Apache 2 like our software which is like take it modify it do it whatever you want with it here is a gift to the world and then recently the model lab are trying to understand a way to economically fund their own model development after all model training and research are and the data are very very expensive. So we have been starting to see uh terms even to the llama days for for when meta was releasing llama they do have a term of if you're daily active user or like annual recurring revenue exceed some threshold please enter into a commercial agreement with meta specifically right

肖恩 (Sean): I do remember that the numbers were like specifically chosen at that time that you could go find it was like two companies in the world that like fit the definition that they had excluded from their license. Yeah, exactly. But like people have taken a hint from that especially now the labs are trying to figure out a way to economically fund it. especially when they're open source the model everybody can just take it and run it themselves whereas nobody will use your API anymore in many cases while their API currently still taking up shape right and now we're seeing a very healthy ecosystem development starting from even miniax recently when they're releasing the M2.7 model they have a term specifically focusing on uh usage and ki initially also has like if you have derivative derivative works like this is kind of big news back then was um fireworks and cursor about how they built on top of Kimi model.

肖恩 (Sean): and so a lot of our developer are using like Kimmy K3 today even just uh making sure because it's similar right a developer can run it and customize it to their custom workflow. I do remember that we've had issues before where one company tried to release a derivative work and there was a bit of controversy. How do you see the license evolving over the next few years and is there going to be a clear standard for open weight licensing?

西蒙·莫 (Simon Mo): Yeah, I think we will see more structured terms. The openweight license is not OSI-approved open source, but it has created a new category of 'shared-source' or 'open-weights with commercial caps'. It aligns the incentives: small developers get it for free, while large commercial enterprises pay their share. This is the only way to fund the massive CAPEX required for frontier research.

肖恩 (Sean): And and it's if I I I could just expand on that a little like

马特·伯恩斯坦 (Matt Bornstein): I I I don't think it's greed at least what I've seen from open source model labs, right?

马特·伯恩斯坦 (Matt Bornstein): Open source models really what we're talking about are open weights. Yeah.

马特·伯恩斯坦 (Matt Bornstein): Right. And and it's just not software, right? Like an AI model is not software at the end of the day. Um, and so open- source software used to be supported by people donating their time or big companies kind of authorizing their employees to donate their time. So, it was sort of like a bulk inind, you know, donation of people's time.

马特·伯恩斯坦 (Matt Bornstein): That really doesn't work in AI,

马特·伯恩斯坦 (Matt Bornstein): right? Like like I can't just like go home at night and like train a frontier opensource model with friends for fun. Like we need millions or billions of dollars of computing resources in order to do it. So, so I think it does support your point that like like obviously there need to be economic incentives and there need to be funding mechanisms in place.

马特·伯恩斯坦 (Matt Bornstein): Frankly, I think even more so with with Chinese models than than with domestically produced models, right? If if if there's no source of economic if there's no source of funding

马特·伯恩斯坦 (Matt Bornstein): for for moonshot to continue to train models, like we know where the funding will come from instead and it's not like something we right, you know, it's it's government and and things that like are actually worse for us, I think. So like I I I think I think you raised sort of an interesting point that that this is an important economic structure and like I I I think this means we'll see more of this in the future. Would you agree with that? Yeah, it's really about sustainability in the end. is about how do you make sure that all this initial capex almost to train the model fail again and again and train the model again like how do you really pay it back and how do you make sure that there's enough confidence and funding uh from everybody involved to go to do the next one right and I recently heard someone uh I recently heard someone making analogy to this to the pharmaceutical industry is almost like how do you make sure that the R&D process of new drugs are properly funed there and there's proper sustainable uh method to making sure that people are willing to take big risk big bat to go to do research for new drugs and then later because they know there's a uh economic incentive in the end when the new drug released to the market a portion of those of course like besides the just distribution channels right a portion of those uh revenue will flow back to continue to fund the next R&D effort and this is where we're kind of seeing uh similar to the model development now Yeah,

肖恩 (Sean): that's a really interesting analogy because it's like

马特·伯恩斯坦 (Matt Bornstein): once a once a drug a molecule is released you have the strongest possible control which is nobody else can manufacture it at all right it's like the clo most closed possible source right like it's like a secret um uh but but yeah like in the case of models especially open source models you know once it's there anybody can take it use it extend it etc so so having yeah so having some economics attached to it probably does make sense

肖恩 (Sean): well Actually that that raises a a question for me too which uh Matt you were alluding to this earlier about how different open source models are from the dynamics of open source software maintenance when it comes to opensource AI what actually needs to be maintained is it the infrastructure around it do the models themselves need maintenance at all just kind of what what are those dynamics because I think even the the developer behavior around it is is pretty Yeah. I mean, Simon sort of said this already, but maybe maybe I'll just expand a bit, which is

马特·伯恩斯坦 (Matt Bornstein): um you know, you you see the results of of a big training run, you know, where training now means it's pre-training and then and then you know, RL kind of um uh you know, mid-training or sort of post- training on these things or you know, pre-training, SFT, RL, right? Like it's sort of a complicated pipeline. We see only the result of this at the end and the numbers are big. You're like, "Oh, you know, this was a $100 million training run." But what you often forget is like there may have been five failed training run, you know, large scale failed training runs before you even get to

肖恩 (Sean): the blood, sweat, and tears.

马特·伯恩斯坦 (Matt Bornstein): Yeah. Yeah. Exactly. One one of my favorite artifacts, maybe we could even track down the link is one of the early llama models.

马特·伯恩斯坦 (Matt Bornstein): They published the whole like conversation log between the or between the people who were babysitting the training clusters like while the models were training. And it's it's so funny. It's just like, "Oh no, everything's gone wrong." Like chaos. Like panic. And then like then the next comment is like, "Okay, we solved it. Everything's okay." You know, clusters up, losses going down. So there's there's a lot a lot a lot that goes in behind the scenes before these models get released, you know, once they're out there. I guess it's a little bit more in your in your zone, you know, to to kind of make sure it's like operationalized.

西蒙·莫 (Simon Mo): Oh yeah. But but this is also a very interesting point. Once it's out there is a whole community effort trying to opt in this model because the model is trained on a given type of hardware on given type of architecture and but when it's out in the wild everybody has different cluster topology and use cases and it's about how do you turn like a use case of one now to a use case of almost infinity. Now you have people trying to adapt it to the edge devices and people trying to run at largest scale ever adapted making sure it runs for voice agent but also for coding agent which are entirely different kind of use cases. So this is a whole community effort trying to further optimize specialize and making sure the running of it is reliable and continue to be able to optimize against it. So that's a whole village uh later uh throughout the open source still to make sure it's improved. Yeah.

马特·伯恩斯坦 (Matt Bornstein): And that's cool because that really is like open source software. I mean and this is what you do but you know like this is like anybody can contribute and and like make these better. Um my the the fun thought experiment is if GPUs you know dropped in price by 99%. Right? Like like if if if GPU based compute actually became you know kind of cheap and widely available like then do we get back to a kind of a real open source world where you know one person sitting in their basement or a hundred people working in their free time can like come up with something new try many of these sort of model training paths that you know that like are in the queue somewhere at one of the big companies and and you know kind of kind of see you know really expand and advance the field collectively.

肖恩 (Sean): Yeah. Yeah. Well, I mean, this relates to what you were talking about at the beginning. It's like, you know, at the beginning, the amount of compute you needed to be at the quote unquote frontier was negligible, and now it's just like it's it's enormous. And how do you how do you get that, you know, back to consumer parody again?

马特·伯恩斯坦 (Matt Bornstein): I've said this on the podcast before, but I'll keep saying it. AlexNet, first, you know, kind of like neural network to run on on GPUs that we care about, ran on two GPUs. And that's not that's not like there are no missing decimal points or commas in there. literally two now. That would get you literally nowhere.

肖恩 (Sean): Yeah. Yeah. Um so I guess this this relates to to another question that we've had which is which is inference has gotten a lot cheaper but also scale has changed how developers interact with models. Can you talk about how scale has changed the core design of vLLM and what the main challenges are when running vLLM at scale?

西蒙·莫 (Simon Mo): Yeah. So scale comes from a few points. It comes from whether or not you can run this gigantic model on a data center scale cluster with thousands of accelerators, where network bottlenecks (like All-Reduce overhead) dominate. vLLM has to coordinate tensor parallelism and pipeline parallelism efficiently. But scale also comes from request throughput. Under high concurrency, KV cache memory footprint grows exponentially. That is why PagedAttention was such a breakthrough: it treated GPU memory like OS virtual memory, eliminating fragmentation.

西蒙·莫 (Simon Mo): I think uh there's two parts to this. Our team always have open source kind of angle. We're from UC Berkeley a long tradition of building systems like Spark, Ray, and Mesos. So we think the open platform wins because it brings the best minds to solve the systems problems together.

西蒙·莫 (Simon Mo): Yeah, like this kind of goes back to our previous point about control. So for the huggy face incident, they break it down about how the open way developers are taking over. Because of the control, they can implement specialized serving layers, optimize routing, and enforce safety policies that match their own compliance needs without relying on a third party.

马特·伯恩斯坦 (Matt Bornstein): I think your social media analogy is a really apt one.

西蒙·莫 (Simon Mo): Yeah.

马特·伯恩斯坦 (Matt Bornstein): Cuz like in both in both cases, what's kind of happened is you've taken like distributed human activity and kind of centralized it into platforms.

马特·伯恩斯坦 (Matt Bornstein): And that allowed the moderation problem to be tractable, I think, right? It's like, okay, we're going to do our best to police this garden, right? But the moment it goes open-source, or open-weights, the garden wall is down.

西蒙·莫 (Simon Mo): Yeah.

马特·伯恩斯坦 (Matt Bornstein): Like we're just we we we just can't police everything. We can't be the world police of like all human communications.

马特·伯恩斯坦 (Matt Bornstein): I think something similar is sort of happening in AI right where a lot of work not just talking but work is kind of being delegated to these local agents and systems, and we have to accept that we can't control every downstream output.

西蒙·莫 (Simon Mo): Yeah.

马特·伯恩斯坦 (Matt Bornstein): And they don't have that carve out, right? Like they don't have that exemption of of like, hey, we're not responsible for what our users build, unlike the Section 230 protections that social media platforms had. AI labs are facing immense pressure.

马特·伯恩斯坦 (Matt Bornstein): And like some of them, especially, you know, anthropic like is is kind of going further than even what would be sort of legally required. And they're sort of taking ethical stances on these things, which may be right or maybe right. That's a decision they have to make.

马特·伯恩斯坦 (Matt Bornstein): legally required. And they're sort of taking ethical stances on these things, which may be right or maybe right. That's a decision they have to make.

肖恩 (Sean): Yeah. Um, Simon, we're we're nearing the close of the conversation. Um, and I just wanted to take a step back a bit and look at the bigger picture.

肖恩 (Sean): Just I guess to close this out five years from now uh do you think open weight open source AI models have they closed the gap with frontier models completely? Are frontier models always one step ahead? Kind of how how do you see that shaking out? Five years. That's like 500 years. All right. One year. One year.

西蒙·莫 (Simon Mo): Yeah. Five years. We're Who knows?

马特·伯恩斯坦 (Matt Bornstein): We're either, you know, we're all going to be like just floating around in our Wall-E pods on our spaceships.

肖恩 (Sean): Exactly.

马特·伯恩斯坦 (Matt Bornstein): Progress, right?

西蒙·莫 (Simon Mo): Yeah. Yeah.

西蒙·莫 (Simon Mo): Um for for me really at this point, there's kind of a point we haven't talked too much about is what really differentiate open way model from closed way model, right? In the end there's not much differentiation. is a more about the distribution strategy and go to market strategy and the capability wise I don't really see a big gap not even today because for how these model are coming to being they're really starting from the first principle right you have a computer cluster you have training data and you have brilliant researchers uh that group together and really to build this amazing artifact that is this mo pre-trend model and then later our old uh post trend model and that the world can use. But if you look at the ingredients right the one of the most important part just the data it's about who gets what data and then what are the environment you are building to let the model improve on itself and make better right one of the very useful uh benchmark that we have on arena for uh for for K3 has been front-end coding right that means for moonshot they have built some of the best environment for front-end coding Right. They have published amazing demo on the ability for this model to code and then see what the rendered is and then kind of continue looping and this iterative process. Now this is about their environment to improve the model. It's not about just source data. It's not about where they get the data from. other is who can build the best environment and who can make the most sort of uh optimization and algorithmic choices to leverage out of learning from this environment. So the next year is all going to be about that is about how open way model labs are differentiating and really getting the model to meet the real world and have this kind of what people are popular today like recursive self-improvement almost to really improve the model overall. And so really project out in your ear there's not going to be any difference.

肖恩 (Sean): Yeah.

马特·伯恩斯坦 (Matt Bornstein): Yeah.

肖恩 (Sean): And you've used this term brilliant researchers a few times. Um

肖恩 (Sean): there are brilliant researchers everywhere in the world clearly. Um what why do you think

肖恩 (Sean): you know in the US all the smart researchers are working on closed models and in and in China all the smart researchers are working on open models.

西蒙·莫 (Simon Mo): I mean from my point of view they are attracted to interesting problems not necessarily on the open or closed stance but rather but however open way model does give people a really really good boost on the impact of such models. So that is like a plus

西蒙·莫 (Simon Mo): and I I think all the brain researchers are attracted to how to improve the model overall right like actually one interesting point about this um uh maybe fairly technical uh for for this Kim K3 model is they removed a uh rotary positional embedding. So rope has always been there for a lot of the Transformers model and guess who removed it is the inventor of rope who are

西蒙·莫 (Simon Mo): like yeah like the the Jenning he he wrote the first paper introducing rope as a concept and then he now also wrote the explanation of why you don't need it in as part of the technical report in this case model. So like when we read it is like really come full circle is you have all these brilliant uh humble researchers that are able to really study how this work and really study the secret of training and pre-training and share it across the whole world and recognizing and iterating on their past. Right? So really a miracle I would say for this model to come alive. It's so

马特·伯恩斯坦 (Matt Bornstein): It's so interesting like you know AI is in this funny zone where empirically it works incredibly well but then you go ask the theorists and they have like no idea what's going on right and so like you have these kind of iterative things where when you go read you know

马特·伯恩斯坦 (Matt Bornstein): primer on transformers you read about positional embeddings and why or positional encodings and why it's so important because otherwise you can't sort of like understand meaning you know and then it turns out once you understand one level deeper because we've been doing this for a few years and you have all these smart people like oh actually you don't like it's actually, you know, simpler actually is better. Um,

肖恩 (Sean): we we didn't um

肖恩 (Sean): we didn't talk about distillation much so far in this conversation, but I think it's very relevant to this. Like

肖恩 (Sean): I I have just one question which is like I'm I'm not going to ask like is distillation happening? I think this is kind of speculation on the part of everybody, you know, in the world, but like

肖恩 (Sean): you work a lot with these Chinese labs. Do you think distillation like is a critical component of what they they do or or or like are they kind of just doing good work and and you know distillation if if it's done is sort of an incidental part of it.

西蒙·莫 (Simon Mo): I I would lean to the latter part specifically as I mentioned previously environment matters so much today. So these are RO environments right these cannot be distilled like you don't have other people's environment to really distill a copy from is about constructing it understanding also understanding the learning process you cannot distill how the model learns within environment a lot of these are just not doable today um there are things potentially you can do with rewriting the data sets right making better pre-training data but again you can do it with any models any models that are are going to follow instruction are going to be useful in terms of utility there. So I really don't think from currently what we're seeing uh this is a big cornerstone of what's powering the progress today. In the end what's powering the progress is still just um really smart people with very interesting algorithms, data environment and they will produce of course compute they will produce the models.

马特·伯恩斯坦 (Matt Bornstein): I think it has really interesting policy implications. I I tend to agree with you by the way that that you know we have smart people everywhere working on a bunch of smart things and it's not about you know distilling data from any one place. um has really interesting policy implications, right? Because it doesn't,

马特·伯恩斯坦 (Matt Bornstein): you know, it's almost tempting if you're if you're sort of, you know, in the White House to say, "Oh, sure. We'll just we'll just turn off distillation. All our problems will be solved." But like,

马特·伯恩斯坦 (Matt Bornstein): you know, I think it's more the case that they're just, you know, smart people doing interesting things. And so it's so it's like how do we how do we kind of like adapt adapt to that? I think it's

西蒙·莫 (Simon Mo): Yeah. And creative innovations, right? Like one part in my essay we kind of mentioned that open source and open way really helps innovation because it's set out this racetrack where everybody can learn from each other and see where each person like every each player is in this racetrack and then you'll be able to improve and stand on shoulder of each other kind of to improve yourself. So uh is that is where everybody can move forward faster.

马特·伯恩斯坦 (Matt Bornstein): Yeah. Yeah. And one thing we're looking for a lot from an investment standpoint is um is people doing more open source model training all all over the world. Not just in China, you know, not just in the United States, but all right. Because you know, you get that sort of magic of collaboration when everybody's doing it

西蒙·莫 (Simon Mo): and achieve global harmony in all.

肖恩 (Sean): Well, I think that's a good note to end on. Um Matt, Simon, thank you so much for joining us. Thanks for tuning in.

马特·伯恩斯坦 (Matt Bornstein): Thanks so much. Cool. Thanks, Sean.