Ryan: Every time a new trillion dollar industry emerges, there's a need for this independent testing group. When Meta released Llama 4 on our held out private benchmarks, the model is actually underperforming. But on all of the major public benchmarks, it was showing incredible capabilities.

Ben Horowitz: What's the limit of what you can achieve? And then within that, how are you going about it?

Ryan: In an ideal world to take a frontier model and have it train the next version of itself, but obviously that's very expensive and slow. And so what we're doing is forming a set of proxies for every part of the process it takes to build the next version of the models. Evaluations as we become more complex have a fewer sample size but a larger set of criteria or expectations of them.

Host: Where do you see the gap that's happening today?

Ben Horowitz: The government kind of has an inclination of what it's afraid of be it biohacking, cyber hacking. But then there becomes the question of can the model do it and then can you get the model to do it?

Host: What do you think the landscape will look like? Um

Host: so I'll start a question from uh when LA got started in 2024 after a team discovered that all the public benchmarks are just not sufficient enough to measure model progress and there needs to be a new methodology and approach coming to you know keep us um uh on the frontier and and help um model labs hill continue to hill climb. uh maybe just take us back to the the inception of vows and what do you see was missing in the in the in the market then?

Ryan: Yeah. Yeah. I mean so I I had a background uh doing research and in particular building benchmarks and evaluations and so what was very clear to me was the very tight relationship between what it takes to build new systems for generation and actually new mechanisms for evaluation. In fact in order to get one you often need to get better at the other. Um and actually one of the biggest drivers for model capability is having these great internal benchmarks to hill climb on. But as models became more and more capable, uh not only did public benchmarks quickly saturate, but in this effort to build broad general intelligence because there were so many different research directions to take, no single mechanism was able to capture it all. And so there was a very clear need for a third party independent entity to measure progress towards intelligence.

Host: And why why do I guess one of the obvious question is why do you think the labs can't do this by themselves? because um they know the best of where the models are hill climbing on and what what is missing.

Ryan: Yeah, I mean internally they they do build a lot of great benchmarks and that's what drives model progress, but I think there's an issue when we we speak about model capabilities in a way that's self-referential. In the internal evaluations, you can selectively emphasize parts of the tests that perform well, but that may not translate to real-world utility across wider workflows. So when you're trying to put models into production or decide which lab's technology to adopt, without an independent party doing unbiased measurement, it is very hard to establish genuine trust.

Ben Horowitz: And what's the historical analog that you have in mind here? There rating agencies, audit firms. What's the right comparable?

Ryan: Yeah, I I think I think there's honestly lessons to learn across the board and and every time a new trillion dollar uh industry emerges um there there there's a need for this independent testing group and I think the fact that it's moved so quickly in AI has caused necessity for a lot of these parallels to to see to be borne out. Um you know, we think about ourselves as is trying to sit on both sides of the market. So there they're mechanisms by which labs need to prove that new models are very capable, but they're also parallels where enterprises need to figure out what adoption strategy is going to amount to the greatest ROI for them.

Host: Yeah, I want to shift to take us through sort of the six-hour pre-release window before the model drops. Um, obviously you need to run tens of billions of tokens without delaying the launch. Which part is you doing an allnighter versus it being uh automated? Take us through that.

Ryan: It's honestly been a journey. Um, and and I think like the the real the goal Northstar we think about is we we never want to be um kind of a lagging indicator or a delay to a model release. Um and and so that means we have to move really really quickly and extract the most possible signal with the the rate limits or capacity that we have we have. And so early on what this looked like was my co-founder Lynx and I pulling an allnighter uh to try and get as much done as possible and get results out the door. Um now we've built up a team but we've also really invested heavily in infrastructure. And so we're able to run evaluations in a a massively distributed way um running effectively the the maximum possible rate limits with with every model we get access to. Um, and we also have this internal system called Steve. Uh, Steve the economic valves employee. Um, and so that that's been a mechanism by which we're able to actually take more of the human work over time and and put it into Steve.

Ben Horowitz: And how do you deal with the kind of issue that it's a little bit of a an AI complete problem in that um, you know, we still aren't really good at evaluating humans um, or we haven't agreed on it. like there are things like IQ test, there's EQ, there's, you know, the big five personality and so forth, but there's not really an agreed upon um framework for which we do it. And people have issues with things like the SAT and this and that and the third. And then you know of course models are really good at hacking the benchmark um as on the floor proved and so how do you think about you know that issue and what are you like what's the limit of what you can achieve and then you know within that how are you going about it?

Ryan: Yeah, I think I think the honest answer is that it's forcing a lot of the um more fuzzy or distributed forms of eval to be made explicit like what what is really the distinction between an associate and a partner at a law firm. Um and there there isn't a clear test or an eval for that in the human world and so we we have to first establish a lot of that in uh these different different enterprise or real world workflows for us to be able to test models in the same way. Um, and and I think long term that will be actually the biggest bottleneck, our ability to take companies and their emails and and make them legible. Uh, because that's how we'll figure out what signal we'll climb on and where we actually adopt.

Host: Got it. Very interesting. Actually, maybe one question for you, Ben, on just like how the industry has formed before like intelligence came through. Like we're now measuring something that's very fluid versus before like when we're talking about enterprise software like you know there's Gartner rating on like 70 different metrics like you can sort of stack rank on a on a on a quadrant to say this company has these features covered these features not. But now it's like you know very jack frontier that's very hard to measure given industries like what do you see um you know one is the analogy to the past that uh lessons we can borrow and what do you see that's really going to be the challenge and uh and missing pieces going forward.

Ben Horowitz: Yeah, it's a it's a little bit reminiscent of the MPAA, right? Where it's like, what's art, what's porn, where's the line, when is it R, when is it X? Um, and by the way, the definition of that has changed over time, I think. Um, things that that used to be [laughter] X are now R and and so forth. And, you know, then what's PG? What's PG-13? You know, all that kind of thing. And there's no and the famous line is well I I know it when I see it and I think that um this one I I just think it's going to be necessarily fuzzy but there will develop norms over time. uh and and you know like if enough kind of people who run companies or run finance or run whatever it is kind of agree yeah no that's a norm uh then then I think it is as opposed to kind of what we have a lot now in the open benchmarks whereas if you can solve this specific problem then you're at this level and so forth I think that's um you know one it's hackable and then it's too narrow

Ryan: I think when you also look to some historic analogies, there's there's a lot of lessons that you can take from them as well on what's gone wrong and what we need to avoid. Um, and I think for instance, Fed Bowels, one very early decision we made was the decision to uh to never sell training data to labs. It's it's often a place that we're pushed um when we start working with a new lab to to actually source and sell for them a bunch of training data.

Ben Horowitz: Yeah, that's a lucrative business. Yeah.

Ryan: Yeah. Yeah. uh and and actually a lot of that industry has has now built these gigantic business models around. And the issue is that once you start to sell the thing you're testing to the entity you're testing, your incentive is no longer pure. You end up being a data services company, not actually an independent evaluator. In the same way in 2008 the rating agencies were paid by the banks to rate their commercial paper and you saw huge issues from that. And so being unconflicted and building the right business model around evaluating AI is super important.

Host: And Ryan today you have already a pretty um extensive catalog of different type of of task and also models modalities. Maybe can you talk through like what are some of the most popular and also like maybe the most challenging benchmark that frontier models are currently taking on?

Ryan: Yeah well we've done a lot of work in in kind of the economically interesting applications, so heavily in coding, legal research, cyber security, and financial analysis. In finance for example, we've built a financial analyst benchmark which really evaluates end-to-end the ability of a model to take SEC filings like 10-Ks or 10-Qs, pull relevant data, construct complex DCF models, and value private or public companies based on specific operating assumptions. This is extremely intricate because it involves reasoning over long contexts of hundreds of thousands of tokens, interacting with tools, running Python environments, and outputting formatted sheets.

Host: Yeah, I thought that one is a very cool benchmark. It's sort of all the rage in terms of replacing analysts, but to actually do that end-to-end reliably is incredibly difficult.

Ryan: Yeah, I think you know in an ideal world what you want to do is actually take a frontier workflow and simulate it end-to-end. In legal research for instance, we are not asking multiple choice questions; we give the model a factual scenario and hundreds of pages of case law, requiring it to produce a comprehensive legal memo with precise citations to good law. Only in these high-fidelity environments can you see the true divergence between the absolute frontier models and the tier below.

Host: Very cool. Um there's also uh cases where you have deprecated uh indexes and benchmarks. What goes into the decision to retire a benchmark, and how do you keep up as models get smarter?

Ryan: A necessity and and this is kind of the infinite game we're in. I mean, you're wearing our shirt and so we have this unofficial motto, always a higher peak. Love this shirt. Yeah. So, in so far as foundation model labs are hill climbing, they're searching for the next peaks to summit. It is our job to perpetually construct these next mountains for them to summit. Um, and I think that's also how the economy has naturally functioned over time. As um, you know, agriculture becomes less important for our our labor market. Um, there are new forms of uh of of labor less required out of our out of our population. Um and so in the same way we we should expect our benchmarks to keep up with the new frontier for what we want models to do. Um there's another component of retiring benchmarks which I think is is underappreciated uh which is that benchmarks should also be reflective of the current state of the world. Uh so in the same way if you're a lawyer you have to you know retake the bar exam and get certified or if you're an architect you have to get your certification and and or you know a doctor um uh we should also expect models to be tested on the current state of the world and what we know in medicine or what we what we have as our set of laws. And so in the instance of case law updating to something like legal research benchmark that's a desire to create a benchmark more reflective of the current state of the world and also push the models in place we want to see them go.

Host: And how has um I guess one like it used to be like we're we're um doing these like multi- um answer or multi-step questions to like just evaluate prompt answers. Now there's like a lot more agentic uh work that's happening whether it's on finance or legal um or coding especially like there's a lot of um you know um async background agents that can just complete tasks. How has that changed sort of how you build infrastructure? how to think of evaluating um the capabilities of not just the models and agents themselves. And there's also like a lot more dimensions that people care about. It's not just like capability, it's cost, it's latency, it's like um you know whether this model is flexible enough to to to address like broader um domains and task and so on. So how do you think about the additional parameters to to what you evaluate?

Ryan: Oh yeah, there's I mean there's a lot that goes into that. I think on the infrastructure level you have a whole new set of problems. Um I mean for instance now we're testing models in their ability to run over hours, days, sometimes weeks and so the infrastructure needs to be very stable to support evaluation over time and and if there is a failed request we should be able to retry from that one and not redo the whole trajectory. So there's some simple simple mechanisms in the infrastructure we have to think about. Um but but I think in general what we've seen is evaluations as they become more complex have a fewer sample size but a larger set of criteria or expectations of them. And so what I mean by that is a benchmark is largely some kind of input space of things you're trying to query a model to do and a set of requirements or rubrics that you see in expectations of the output. And so early on you have things like imageet which have millions of of images you're trying to see a basic categorization for. So it's a onetoone mapping between an image input and a text label output. Now what we have is far fewer set of tasks. You know generate me 50 fullstack web applications but a much larger complex mechanism for evaluating the output produced. And I think that trend is going to continue as we see more complex workflows evaluated with models.

Host: And and do you do do you think that um it will become kind of an a real time uh kind of mechanism like so for something like open router which stripe just bought would open router look to valves and say okay where should this next request go or is this going to be kind of strictly for like picking a model in an enterprise for a task?

Ryan: Yeah, I mean I think u you know open route is a bit of a misnomer in that most of their usage comes from being a model gateway and so it's actually up to their their users to decide which models they want to use when and that's because really the hardest part of routing is building the evals and trying to determine uh in what places a set of intelligences should be used for a particular application. Um and and so you know our effort in supporting enterprises in building evals has actually supported a lot of them in also adopting routers.

Host: and say more about why it's not only important to labs but also existential for enterprises as well?

Ryan: Yeah of course yeah I mean I think that the lab side of this is very clear like you know if you're raising lots of money investing heavily in building models it's it's essential for you to show uh why your model is getting better and then why those customers should should pay a premium for them. Um but for enterprises, the dynamic is very different. You have frontier models like GPT-4o, Sonnet, Gemini Pro, and countless open-weight options. Enterprise token bills can easily reach millions of dollars a month. If you blindly route every query to the most expensive frontier model, your economics collapse. But if you down-route to an incapable model and it hallucinates or fails, the downstream engineering cost is even worse. Evals are the only instrument enabling businesses to find that efficient frontier of capability versus cost.

Host: And and maybe just double click on that um like similar question to to why the labs can't do it themselves or requires a third party um agency to to rate it. I think it's it's a lot uh more um uh understandable that you know you need this that neutrality across the industry but for enterprise they want to know what's right for their stack.

Ryan: I would recommend a lot of companies to develop in-house expertise um but I think that should not be the only solution um you know there's this explosion of intelligence happening there somehow still more foundation model labs um getting getting constructed and and each lab is also releasing more models. I think today it's still very unclear whether uh the best OpenAI model or the best anthropic model is actually going to be best for your repository. And so we've seen a lot of non-intuitive examples where you actually had to run the eval to figure out what's going to be the frontier performance for your specific setup. Um and and so I think if you were to operate based on you know use sonnet where where you feel like it's it's uh applicable you may actually end up spending more than you need to.

Host: and say more about how this evaluation framework will apply to knowledge work in other domains or what are some examples I can look at?

Ryan: I think coding is a sign for what's to come in every domain and and a lot of primitives established there are carrying over to other places. You know, if you if you have a very good coding agent, chances are you have a model that can also make PowerPoint slides or DCFS and Excel um and and with with the right tool use perform high-level professional work.

Host: Maybe maybe just tag along the the earlier question um how are you guys using VSmith internally to uh evaluate what's the best coding model for for V?

Ryan: Yeah, I mean to be honest, this was actually born out of a problem that that we saw as well. So, um I wanted to do a token maxing experiment. Um and see what happens when developers have zero constraints.

Ben Horowitz: Yeah. It's also crazy because

Host: how much does that equal to to dollars?

Ryan: So, okay. And then I went back and did some math. Um, and and it looked like in that month we spent roughly $1.5 million worth of tokens. This is free, by the way. I No, don't [laughter] want um but it was actually 10x more we were spending in tokens than employee salary for that month. Um, so it's clear we needed intelligent titration.

Host: Very cool. So is the current um operating mode that you're using like one uh I guess more token efficient harness plus model and then like on top of that people have some more flexibility to to use token base to for some higher higher or more challenging tasks.

Ryan: Yeah, we so we we have access to all the tools. We we give everyone access to everything. Um, but we autosue recommendations for any GitHub issue or ticket for where to begin their session. And that should titrate the actual usage depending on the intelligence required for that task.

Host: I want to segue to the policy side for sec for a second because we we talked about how quickly benchmarks become obsolete. In policy, it's it's even worse in that laws move, you know, much slower relative to capabilities. Ben and Mark spend you know a bunch of time um in DC and and with policy makers. Who should be involved in this?

Ryan: Yeah, I think the short answer is that everyone should be involved to some extent. Um I think there's there's a benefit from varied perspectives. I think the main issue though is that policy conversations as they've happened over the last couple of years have been very abstract. uh and there there's lack of grounding empirical data.

Host: how do you think about like who does what? because the government kind of actually did the first evalu um okay what's at the frontier and needs to be regulated right so they started with you know some crazy idea with 10 to 26 flops or some such thing um and so when you think about it like what should the division of responsibility look like?

Ryan: I think there's effectively two counterveailing forces that that has to be considered. The first is the desire to move very quickly and ensure that the the government process isn't slowing down uh the rate of technological innovation. And I think the other part is to make to make sure the technology is not causing catastrophic safety risks. Yeah, I mean we think about this broadly under the category of alignment. Um, I think there's places where you see that borne out now now where models that are being tested for one cyber security risk are actually reward hacking and figuring out other ways to to get around it.

Host: And maybe that's a question for you Ben as well. Um, I guess how do you think about the right division of labor here? like what what should um you know government agencies um control or and do themselves and and where they should like you know partner trust um you know private uh companies to to take on?

Ben Horowitz: Yeah. So I I think the government agencies do get a lot of warnings from by the way the big labs. Oh, this thing is going to biohack. This is going to be a cyber security risk and so forth. And so I think what the government needs to do is go okay if the model you know is if the model is capable of bio-weaponry or autonomous cyber offense and people can reliably jailbreak it to execute that, that's a clear threshold. But the government doesn't have the technical apparatus to build frontier evals; they must partner with independent testing entities to get the ground truth.

Host: very you know we're in interesting times I would say and to me there's the gap of like what's the narrative and what actually happens in real world um Because you know every setup in again like an enterprise setup is is very different like um or or just like people however they use the models are very different. The narrative uh that connects to the actual capabilities. To end say more about how exactly the policy maker should work with the evaluator what information do they need what what how should the relationship work so that it's most effective.

Ryan: Yeah, I think I think in the first order there should be an a mechanism by which um insights and data can be passed directly to relevant people in government. And so now we're regularly doing briefings for um executive and legislative branches um on what we're finding capabilities and risk of models. And so I think first order that helps uh people there get up to speed on what's going on and and also track through what will be problems in the future. I I think you know things are moving very very quickly. It's hard to predict where where things are going. Um but at least when you have data you can start to extrapolate a trend. Uh and then I think from there it's it's up to the people in the legislative branch to decide where they they want to see policy. Uh and so it's not really our our place to give recommendations like that. Um but if they if they see that there is significant risk in say mental health for for people under the age of 18 uh or biocurity risk in the models that necessitates having a standardized way to curtail model release. Um then it's up to them to inform policy. Um and I think then there's other places where the executive branch in the places like department of commerce or SEC is responsible for for making sure that private companies are able to adopt and use the models in a way that's going to be productive for the whole system.

Host: I love to probe on another angle just around geo geopolitical. Um I often see evals being like a representation of sort of the value um of of the model the model developer like you kind of develop this rubric of what's embedded in in the model and of course different different countries um uh and and labs in those countries care about different things like I mean um I'm born and raised in China use a lot of like Chinese uh open source models too like you still cannot let them you know just go freely talk about CPC and all the industry there uh because um you know what happens in China. Um so so how do you think about how eval I guess and benchmarks play a role in like standardizing or like uh being treated by different um different model labs from from different places.

Ryan: Yeah. I mean to be honest from from my um very idealistic perspective um I'm surprised to see so much investment in sovereign AI. Um, you know, if I was taking a god's eye view, it would be extremely inefficient to build all of these data centers and and replicate uh this data engineering process and train these very large models um when when in fact you could probably consolidate a lot of these efforts. Um, but it seems like that's not the world we're in or the one we' headed towards and and there's actually increased efforts to build AI in a sovereign way. Um, and so I think that takes having a shared language to communicate about what the framework for valuations are and and um where we're going to collectively align around the risks. Um, you know, I think there's actually a lot to learn from from nuclear here as well. I think Reagan had this line, trust but verify. Uh, and so I think we're starting to see signs of trust in that um, you know, Zi Jinping and Trump are going to be meeting uh, next month. Um, but there is no uh, clear way to actually do the verification part of this. um having the shared language of evals will allow us to say things like you know you you have the the right number of nuclear warheads and in that example there were also flyovers so so Mexican by which a country could audit another country's uh nuclear stockpile by having flyovers um and so I think similarly if there's concern about the societal or even existential risk of AI it will necessitate us uh constructing this shared language of evaluations to do the verification process

Host: how do how do you think about um harmonizing a policy like that so that you know it's hard enough to do it in America and then how would you think about kind of taking it global um you know because now you're dealing you're not dealing with enterp enter enterprise customers you're dealing with governments and those governments are competitive with each other and how would you think about that working

Ryan: I would be naive to say I have the perfect solution to this problem today. Uh and so I think there are baby steps in which we can start. Um for instance, there seems to be a lot of talk about cyber security risk. I think the concern around biocurity will become even more important over time. And so there are clear places where there will be mutual interest in aligning around uh ways to to prevent conflict around cyber or bio. Um I in my opinion I think long-term what's actually going to be the most interesting is the recursive self-improvement possibility. And that's a place where you could see one country, one or one company kind of run away with it and produce models that we we don't know much about or or operating in ways that are unknown to us. Um, and so I think having a way to in a joint way describe this being the level of pace we're comfortable with or this being exceeding the pace of of development as as it relates to RSI is going to be super important. And that's where I think you see a lot of the researchers at Closource Labs calling for joint conversations between governments today.

Host: What do you think landscape will look like going from from here? Um now that we have lots of different capabilities and uh capable models as well as like you know um uh countries that care about different um developing I mean everyone cares about RSI for sure but like on on the bio side or like the the cyber side people care about um you know slightly different different things whether it's more offensive defensive and so on like what do you think the landscape will look like and um how do you think about developing new benchmarks to to keep up with with that?

Ryan: Yeah, I mean we're we're hyper focused on on building benchmarks that capture the frontier. And so uh in so far as we see new places for capabilities or risks at that frontier, we want to make that uh an actually well doumentable evaluation um on Val.AI. Um and and I think it it takes have increasing coverage over time. You for instance I think in in cyber security a lot of our historical work has been done around um code vulnerabilities or memory leaks that may exist in code. Um but actually a lot of the the biggest concern or risk is in the infrastructure level. Uh and so these are not things that are expressed in code but take um simulating larger environments of enterprise cloud infrastructure or or even grid infrastructure for us to be able to say this is what the offense or defensive capability of models is. Um and and so making sure evaluations are reflective of those new places is is really important for what we do at VLES. uh and we believe that the the most valuable form of this business will be one that's incentive aligned around doing really high quality evaluation not supporting the intelligence development process or uh the process by which the models can actually improve on that side over time.

Host: Awesome. Thanks for coming on the podcast. It's been a great episode.

Ryan: Thanks so much for having me.

Host: Thanks so much, Ryan. Thanks, Ben.

Ben Horowitz: That was fun. Thank you.