超越模型卡片：用评估度量参差前沿与大模型投资回报率

going from beyond the model card, how

evaluations can be used to measure this

jagged frontier. Uh special focus on

finance for this audience as well.

model cards themselves, we've built uh

some of the most important benchmarks

over the last couple of years that the

major foundation model labs have

included in their model releases. So

people I think are familiar with

artifacts like this. Uh last week we

have Sunder talking about our Val's

index on stage and then uh even just

yesterday Muse models uh reciting our

our Finagint benchmark. You know, it

keeps our lives interesting to have to

keep re-updating these slides as new

models come out. I had to double-check

new models didn't come out this morning,

but um these are say these these kind of

model cards reflect a general

understanding of what's newly capable

with the models. Um and so uh they they

can reveal from a single number how

intelligent is a model. In particular,

our Val's index is compiling results to

try and measure the economic impact of

models uh in domains like finance,

legal, but also coding, health care, and

things like that. So from these

high-level numbers, you can get a sense

of how capable a model is for uh

valuable work.

the whole picture. And so when we dial

in a little bit more closely and look at

the constituent benchmarks that comprise

the Val's index, we can see a more

nuanced picture. And this may be

something you all are familiar with, the

jagged frontier. And the idea that

within a particular domain or set of

tasks, you may see different performance

uh from when you aggregate results to

produce that single number. And so you

can see here across the different

models, uh we actually see an

inconsistent performance for who's

number one.

look at particular organizations or

companies. So in a lot of ways, coding

is indicative of what's happening across

every major vertical. Um when we look at

coding and actually create custom

benchmarks for particular repositories,

we see that that jaggedness just

increases even more. And so on on the

x-axis you've got different programming

languages and libraries. And on the

y-axis you see the relative performance

of models and see that it's even more

fragmented. But this isn't even to take

into account that there are different

harnesses in which these models can can

be used or the idea that you might be

optimizing for cost and latency as well

as performance. And so this all feeds

into the idea that the the granularity

of of the performance you see in a

general model card benchmark is not

necessarily what's most applicable for

people who are using these models or or

products in practice.

considerably. So year over year there's

roughly a 2x increase in enterprise AI

spend. But with all that spend it's

still really obtuse to understand where

the ROI is coming from. You can see that

people regularly using AI is now 90% but

if you look at the smallest bar now we

have established ROI just at 7%. So for

all this token spend and increased

adoption it's still very unclear where

AI is actually driving value for a lot

of companies.

jump out at me, the groups that are able

to actually track AI cost and spend are

much much more likely to actually

understand where ROI is going to come

from. And this is also a key blocker for

adoption itself. The groups that are

unable to actually quantify the cost to

the return on their investment often see

this as a big bottleneck to actually

widely roll out AI. And so I think evals is

the key story here. If people are

able to construct key evaluations that

measure what our model's capable of and

what will be the most efficient

utilization of those dollars, they'll be

able to adopt AI in their firm more

broadly.

finding that that I want to share with

you all is an indication of why the

intuition that are

intuition of model performance as

presented by labs doesn't often align

with experience. So if you look at Opus

5.5 that's the kind of second tier model

from Anthropic. And on paper it's

actually cheaper. $4 and $20 out

compared to Fable 5.1's $10 and $50 out.

But in practice, and particularly for

this tax benchmark, you see that it's

actually more expensive, 14 more

expensive. And that's because the less

intelligent model is trying to expend

more tokens to try and get its work

done. But this actually bears out as a

3% lower score. Um and so you are

actually would have been better off

using the frontier intelligence. You

would have spent less and got a better

performance. Um but that kind of

intuition isn't isn't directly available

for people who are using the models in

practice. And so they may try and say,

"Actually, I should use a Sonnet or an

Opus." But end up overpaying for uh

inferior performance.

reveals that we should actually take a

more diligent effort to figuring out

where return is coming from. And and

that will inform a more realistic

picture as opposed to the intuition that

labs are feeding directly.

particular, so that this kind of jagged

frontier concept, uh I 've highlighted

the Vals Index, but we also have a

finance agent benchmark looking at

research tasks for finance, and then

Excel modeling, uh and then tax agent.

Uh and even across these benchmarks

again, it's not immediately obvious

which is number one or will be

consistently two or three.

when you look at the Pareto curve. So on

the x-axis, you've got cost. In

particular, it's cost per task, not just

token in token out cost. And on the

y-axis, you have performance. And and

here what we're looking at is that Excel

modeling benchmark. So seeing whether

these AI models can create and work with

Excel models that are core to work

that's happening in finance. Uh in the

green, you have the overall performance

across all of the questions. And then in

red, what we've done is is segmented out

the LBO tasks. So this is a particular

subset of Excel model tasks that you

might you might be working with in the

in uh

um using any of these models. Um and I

want to call out a couple things. In

particular, first what you see is that

the entire frontier for LBO is lower

than the overall task score. So the

intuition you would have from just the

the high-level EMB results would not

match what you see for the LBO

categories of tasks. And so if your firm

is especially doing, you know, one type

of Excel models over another, it's

important to have a clearer picture of

what's going to be actually relevant to

you. But the other is that the frontier

is also changing when you go between the

overall scores to just the LBO

subcategory. So, whereas you might have

thought you would have get the best

possible performance from using Feable

5.1, when you go to the LBO it actually

drops off the Pareto frontier. You're

going to be massively overpaying for

what is no longer the state of the art

model for that category. Uh and so

again, this kind of reiterates this idea

that it's it's less important what you

see at the top, but actually the subset

of tasks that are most relevant for you

and the cost performance optimization

that you want to see within your

company.

from actually building the benchmarks.

We talked a little bit about the results

and and why the results may be intuitive

unintuitive. Um when we actually build

benchmarks, one of the most important

things that we need to look at is the

metric we use to report accuracy. Uh and

so you you you have a different takeaway

if you pick a different metric. A lot of

benchmarks deal with partial credit. And

so what this is answering is in

expectation, if I were to give a

question or task to a model, what

percentage of the points would I expect

to uh see satisfied by the the the work

the work output of that model,

especially compared to humans? And then

the fully corrected answers metric is

more looking at of the percentage of

questions that I give to models, what

percentage of them will come back

perfect? And that's that's a very

different metric. It may mean uh that a

model is very good at producing uh an

incremental response that you can work

with to build a better response. Um but

if you were to pass this directly to a

client, it would be a coin toss of

whether it was meeting all the criteria

that you have. Uh and so picking the

right metric is very important to

accurately representing the performance

of a model and actually forming a good

intuition of what it can do.

think a lot about. So, validity, are you

actually evaluating what's going on in

the real world? That means are the

questions or tasks reflective of actual

work going on or are they contrived? Um

and then are the rubrics actually

describing what we're looking for or the

standard of excellence we want to see?

Reliability, this is usually a function

of the number of tasks or the sample

count. Um a lot of companies, they have

maybe five or six samples and they're

kind of vibe checking new models or

products as they get access. Um but it's

actually really important to have a

rigorous approach and have enough tasks

and samples to make sure your repeated

runs actually show consistent results.

And then lastly, discriminability. So,

we we don't think it's that valuable to

have benchmarks where all models score

100% or all models score 0%. Ideally, we

can we can actually discriminate between

models and see what the frontier looks

like. In particular, we also want to

make sure we're building benchmarks

forward-looking. So, what are the new

set of tasks that are even more

ambitious than what models can do today

uh to make sure that there's room at the

frontier for us to uh you could uh to

continue to see model progress as they

get better.

high-level problems that we've seen and

I think others

uh you know, can and and do run into.

The first is that agents solve the task,

not the problem. So, this is talked a

lot about as reward hacking. Um, and

we've seen a lot of instances where

we'll have a sandbox and models will be

run with our model proxy. And sometimes

they may try and actually call other

models uh within that proxy to escape

the sandbox or make use of the internet.

Uh, so it's really important to to build

the the containers in a way that you're

actually testing for what matters. Can

they can they do the task at hand um or

are they actually leveraging other tools

or maybe even other models to get there?

Um, we've also seen this issue in

finance in particular where a lot of the

times uh evals are looking at a set of

rubrics or or items that we want to see

included in the response. And so, models

can hedge and give three possible

answers uh which has a higher chance of

satisfying all the criteria um but in

practice, that's way way too much. A

person does not want to consider three

different answers. They just want to get

the the first answer and and have it be

right. Um, and so,

the goal here is to try and build evals

that are reflective of the actual work

people are doing and the work output

they want to see um not to build these

contrived evals that that the models can

actually gain.

are starting to see that the cost of

software is coming down and it's it's

actually very very easy to produce code.

But

unfortunately, we still see in practice

that running evals is incredibly

challenging, especially from an

infrastructure side of things. Um, and

in particular, as we get to

longer-running agents, it means that

it's really important to have

infrastructure that can stay consistent

over a long period of time and then if

interrupted, can be resumed and

respawned back up. Um, we have I think

just recently crossed 6 trillion tokens

just in evaluation. Um, so we run a

considerable asset of evals and we run

everything in as highest degree parallel

as possible. Um, so I think this has

really been load tested, but I think

other people are going to run into this

even on a much smaller scale.

of the existing benchmarks, they deal

with a single prompt, single response,

but in practice everything is

multi-turn. Uh, you have an initial spec

which you think covers everything, but

then you get questions back or you have

additional feedback once something's

built and then it takes more of this

iteration to actually drive to the end

output. And so I think benchmarks and

evals need to actually cover this

multi-turn component. That's not going

to go away.

particular, when we looked at our team,

we did a little token maxing experiment

uh, where we we got free access to some

tools and just gave it to all the

engineers. Um, and they they really made

use of these unlimited tokens. Um, and

and if if we actually look back for that

month, we spent 1.5 million dollars

worth of tokens. This is like a team of

I think 10 or 12. So it's roughly 10

times the amount we were spending on

salary for tokens in that month. Uh, and

so that was pretty insane, but we saw

great output. And so our goal was to try

and maintain the same level of output

across all these engineers, but

massively cut cut down on cost. And and

really our solution to do that was of

course to build an eval. Um, we built a

bunch of tools internally and then have

now exposed this as a product called Val

Smith. Uh, and so this is you know, what

we're doing in coding and I think it's

again a sign of what's going to what's

going to come in other parts of

knowledge work. Um, so what we were able

to do is look at the GitHub repo um, as

a essentialized way of seeing what work

has already been done and what work is

continuing to happen. Um, and in

particular GitHub repos are comprised of

PRs or pull requests which have a GitHub

issue and a resolution. And so what we

can do is unwind the clock and give

those same tasks to agents to see if

they could have equivalently done the

work of humans.

much clearer understanding of which

models were overkill, were way too

expensive, and did not need to be used,

and then which ones were actually

capable of getting the task done. Uh and

I think again, this is another place

where the intuitions that a lot of

people have may not be what's actually

born out in practice. So, I think you

may see, "Oh, we're using a lot of

close-source models. We should try and

cut costs and use our own self-hosted

open-source model." Uh you can see the

comparison here we have between GPT-5.6

Soul and Kimiko 3, and we see that 5.6

Soul is better. It's It's going 2%

better on this internal eval. It's also

considerably cheaper than than hosting

it through a very, you know,

inference-optimized uh stack. So, it's

$2.50 instead of $11, and it's faster,

half the time. And so, there's no real

reason why um this kind of intuitive

sense of using open source would be a

legitimate way to drive down cost. Um

and so, this has resulted in two main

differences. One, we can actually figure

out which tools we're going to procure,

what the usage limits of those things

should be across the organization,

but then also, we can give

recommendations for engineers and say,

"We think that for this GitHub issue,

you should actually use this particular

coding tool with this model." Uh and

this is able to drive the cost down

considerably. I think we're spending 90%

less, but still able to get the same

output.

you all We're We're continuing to expand

this approach and and take this model of

coding to other domains, knowledge work,

legal, finance, things like that. But

the key question is,

where are you tracking the history of

work that has already been done by that

firm? Some of these law firms have been

around for like 100 years. And so, where

Where is that I think quality of the

organization stored? Where are you

tracking the standard of excellence? And

so, that's today may exist in places

like email from the history of responses

that people have had with their clients,

may exist in SharePoint, may exist in

Databricks dumps, but I think this is

the key treasure trove that a lot of

companies have and don't realize will be

the major unlock for evals, and then the

unlock for AI adoption more

systematically.

they can enable adoption to happen, but

we've also seen that there's more

advantageous ways that they can be used.

So, first is right-sizing.

So, rather than just token maximizing,

how do you figure out what the right

usage limit should be across the

organization? Negotiation. Once you

realize uh you have actual leverage, you

can take that to the conversation you

have in bulk pricing.

Then routing. This is becoming a core

primitive in uh

in a lot of AI products, um but can also

be done by users directly. And so, they

can build intelligent routers that

actually figure out the right model to

use.

And then fine-tuning or post-training.

Uh this this can be the most reliable

way for you to actually mint the tasks

and then have an external eval set to

make sure that this approach is working.

Last thought I'd leave you with is I

think for a lot of companies, they have

a competitive advantage that has existed

in the heads of employees through things

like institutional knowledge. And what

evals allow you to do is actually

codify the competitive advantage and put

that down in words or in scripts in

language. Uh and that will actually be

the the biggest unlock for uh making

sure that a company succeeds through the

agentic era. Okay. That's all. Thanks,

everyone.