评估与优化 Agent Skills：如何量化模型先验与新知识增量

will be talking about skills and agents

knowledge of those skills.

where we've been

playing with the new stuff, new work

being done by the agents, repeating

those prompts over and over again, and

then we start packing those into the

skills,

creating repeatable, reproducible,

portable type of their

functionality

or capabilities that we can

teach agents to work with.

important, hundreds of stars on GitHub.

There are different types of their

skills. Some

teach models something new, bringing new

knowledge or capabilities to the model,

but

they're also type of their just steering

or process skills where we

do not bring any additional information

but just getting repeat

table

flow of the actions that we want agents

to perform.

be talking about new knowledge type of

the skills

and

we'll see what we can

get from the model's understanding of

those skills.

using them

basically

starting with their the feel of

does it work or not.

But there is also their more technical

way and more deterministic way for the

checking if skills are working or not

and what's the

impacts of those skills

are evolves.

evolves can be defined,

but at the sense this is task definition

that you want your agent to be able to

solve.

You put it in the place, ideally more

deterministic, so we can validate it.

And we can two arms. One where no skill

loaded and their second one where we're

loading the skill and providing this

additional information to the context.

multiple things. One is how many

evolves been solved.

In this case with no skill we're solving

three out of five.

With skill we're solving all five,

meaning that we get

two more tasks.

There also would be implications on the

tokens usage or time that it took

those tasks to be completed and other

factors.

seeing what makes sense what not.

And this is one of the main tools for

evaluating the skills and

saying that they not just

nice ideas but actually working pieces

of instructions and context for the

agents.

we have their working skill. It solved

some evolves.

But we know that

there should be a better way. We have

models evolving. We have

some

maybe cases there we want

to get skill more efficient, meaning

consuming less tokens. So what's their

their options there?

brute force

all the different combinations.

Another would be to think through what

makes sense so maybe use something

as the user feedback on the skills to

see where skills perform well there it's

not.

And with new models just running skill

and seeing how it works, what's the

options there?

And

how large those modification should be?

Just

lines, sentences, ideas.

we found to be useful is instead of

relying on their luck, we can put more

deterministic process here

using idea of their items from their

fact score.

We can split their skill and

instructions into those items

and

for each of those have

pair of question and answer that we

expect.

proposition in this case or contracts

there we put more

definitive expectation on their

action.

And in general, we will be getting their

some predefined amount of the items per

skill and we will have let say

25 or 30

items per skill.

again run two arms, one without skill

and one with skill.

And we will have judge

or panel of judges that will be

evaluating model answers. So we asking

those questions to the model that we

interested in.

We evaluating those questions with their

judge, ideally different family of the

models or

panel of judges.

We will ensure that the question do not

leak answers.

depending on the case can compare with

no tools or additional

information to check their internal

model understanding or provided with

their some tools like web access. So

it would be more practical.

And we

can get their simple forward counts

matrix.

Basically where we are checking that the

model

uh

knows or do not know something

and how this maps to their

skill impact on this.

option is where model was able to answer

this particular

questions even without skill or with

skill. This means that skill have no

impact on this and this was already part

of the model's knowledge.

actually model was able to answer this

correctly.

And by adding skill

we saw degradation in performance and

model capability to

talk or explain particular aspects.

model wasn't able to answer something

and with skill we start seeing that it

can answer those new

type of the questions and scenarios and

this is what we were aiming for teach

model something new.

option where

skills were written not

ideally or in some

might be not clean way

where model wasn't able to answer it

either without skill or with skill and

while this topic might be relevant and

important for the model it still sort of

misses some of those concepts.

to fix are

instructions or items that actually

hurts performance and

for popular skills those are pretty

rare.

So in their

uh

box below you see that it's just one

case out of the mostly 400

items in entropic skills and others

don't have those.

But for their smaller skills that are

less popular you probably having more of

those. This is what we are seeing with

some of our um experiments.

this is just straightforward the

simplest thing to fix

to ensure that your skill do not reduce

model capabilities and performance and

bring only prevalent

context there.

And we also can actually color this in

this scenario this is done by lines but

actually going with items you might have

there

different coloring for a single line or

a single sentence.

And this just gives your example there

visualization where

this would be helpful

details and instructions in terms

what new stuff you bring to the model

and what

is

something that model already knows.

we are running all those questions and

asking model

but still relatively this is cheap for

29 skills 14 bucks of there

sonnet and haiku

calls with their sonnet used as a judge.

there

couple ways how we also can check their

strictness of their some knowledge.

We have

lenient and strict ways of the checking

this where first checks the general idea

while the second one is checking for

exact details and you can think about

the difference between general knowledge

and understanding of the topic to the

understanding of the exact

keys parameters and exact values or

steps.

the difference is large

especially with their

larger frontier models where we see opus

dropping from 70% understanding of their

skill in case of the general

understanding

to

just 22% of the specific

details.

this is much our expectations. We see

that new models are

pretty well generalized and understand

concepts.

But it's still helpful to provide their

with exact details and data that will be

useful for their solving those problems.

comparing those three popular rappers

with the skills

across different models. So one of their

interesting observations here is that

smaller model have

in general lower understanding of their

skill

with their larger sonnet having better

understanding, which is

why we have bigger models.

models already understand a lot of their

details and context of those skills. And

you would expect that with the newer

models and more advances in their

better training capabilities, we will

have

more and more

in terms of the model's understanding.

But also we see that this pretty

significantly differs skill to skill.

it's not bad. Some skills are intended

to be directional.

But it's also useful piece of

information to see how those

skills bring new knowledge

to the different model and different

families and this should be run ideally

in the broader

set of the models.

just having their

this novelty piece of information is

just one of the signals. It's basically

initial vector for optimization.

At the end, we still have to run evolves

and validate that

all all changes that we

try to make with their skill actually

works.

would be we can look on their skill and

try to remove some or all of their

already

known pieces of information from the

skill and run evaluation.

It might come up with their same

scoring. Basically we will be getting

their same results model still capable

of solving those problems. Then great we

can reduce

skill size, save tokens, save context.

And instead of their

brute forcing and guessing what we can

or should remove

or add or how to fix their uh

degradations we can have a guidance.

been done with the docume which is a

framework for their comparing

everything related to the agents and

context

where it's is skill, agents MD, MCP

tools, whatever. And novelty is one of

their pieces of the functionality where

you can get this initial vector for

their skill optimization and evaluation

but

there are also uh evals. There is

assessment of the context impact,

routing, comparing their

uh descriptions, naming, bunch of the

things that related to their

skills where it would be helpful to

actually get their more deterministic

evaluations to make

conscious decisions in terms of the what

should be done with the skills.

So thank you for your time and please

go ahead and write better skills.