停止盲猜 vLLM 配置：在 RunPod 上基准测试与自托管大模型

My name is Chris Bella. I work at

RunPod. I'm a forward deploy engineer.

Today I'm going to be talking about

benchmarking VLMs. Kind of like the

struggle that people have when like

especially like a new model comes out.

Just some like learnings that we've had

like internally at RunPod and with like

customers.

So today I'm going to be showing you

essentially like a dashboard that is

built on top of like VLM.

It's using our like pods actually. So to

give you like a little more context of

like what RunPod does is we're in the

new cloud.

We make it really really easy for you to

rent GPUs

and

starting from like R&D to like more

service production workflows. A lot of

times what I see with customers

especially with VLM is like someone

wants you know the latest model. They

don't know how to like spin it up or

they they want to start going to

production and they don't know the the

knobs to like touch. They're they're

just using whatever like the base config

and I'm guessing a lot of you probably

seen like VLM recipes before.

You know it's a good start especially

but like especially in this world we're

always moving fast.

So how do you get to more production

level

style of code and at the same time not

just like vibing it. It's more like how

do you actually have data? How do you

show you know your manager like your

staff engineer that hey this is actually

better. This is actually working

correctly. So

I've seen a lot everyone's just

guessing. Like you know you're a lot of

times I see people are using the wrong

VLM version. They might they're probably

not using like the nightly version.

They have the wrong size for their

context window.

It really depends like

I I think it always is a step back. It's

like what are you trying to achieve with

your your LM model? Are you is it going

to be a coding agent? Is it more kind of

like you're just

um you know, doing summaries, like very

very basic tasks. That's usually the

first question I'd be asking before you

you know, getting Kim K 3, having you

know, a million long context. Like

that's not always necessary.

Uh and at the same time it's like how

many users are going to be using this?

Um

a lot of times people are like going

everyone's trying to always

get like the the smallest amount of GPU

for the cheapest price. But if you're

going to be going to a more concurrent

users, you need to start thinking about

like how am I going to be utilizing the

KB cache?

Uh so all this is there's all these

little like optimization knobs that you

need to like figure out before you go to

production. And a lot of times we're

trying to move fast. So um

Yeah, like like I said, like RunPod like

we like have very much like, you know,

we make it very very easy. It's like a

VM. Uh it's dockerized. Uh you can bring

your own image. But at the end of the

day like um

like who here has used like v l m

before?

you know, like when you're looking for

like like especially like Kim K 3, like

if you you use like a version of like,

you know, 0.27, it's not going to be

working. Like

like we're moving so fast the code is

not getting as fast. Like you need to

have the right version. And a lot of

times that turns into kind of like a CD

CI CD process that

um especially if people who don't

understand what's happening uh just

start struggling. They they start seeing

errors and they're like, "Why is this

not working?" You know, it

the the layers within it are not they

haven't been written or they're it's

very in development code.

Uh and at the same time that's actually

kind of a problem is like Kim K 3 comes

out, you know, the next day I would not

put that on production just yet. Uh

you're going to be seeing a lot of

probably hallucinations. Uh it's just

not ready in that state, I would say.

Um so what I propose um so uh I created

this thing called it's called BenchPod.

It's going to be a template. It hasn't

come out yet.

Um so it's essentially built on top of

like uh v l m. Um it's a whole metric

system. Uh you can benchmark very easy.

And a lot of you who are familiar with

like v l m you can benchmark already,

but this kind of

um centralizes everything, so you can be

able to see everything, be able to

change the config very quickly. Uh

I I notice is that uh especially with

people who use like our uh our pods is

you set up a config, but then you have

to restart the image. You have to wait

for cold starts to happen. It's all this

loading, and you're not moving as fast.

You're waiting like 30 minutes like for

Kim A I 3 to like download the weights,

load into GPU. Uh it's kind of wasted

time. I mean, if you have an agent, you

know, you really don't care, but um

that is a pain point especially when

we're trying to move fast. And at the

same time it's like the cool thing about

Runpod is you can find these machines.

So, if you have that question, it's like

I want to run Kim A I 3 on a B300, we

have B300s, and you can start going from

like not just like reading about it. You

can actually start testing it and seeing

if, you know, there's truth to all these

metrics.

So, yeah, so like bench like like I

said, like bench pod is more like

um it's it's built like a of a a docker

image of um

um v l m open a b i. We're going to be

having every version already. Um

we just have a little like like payload

that like pretty much creates this whole

fast a p i application. Uh you can use

it for your agents. Uh the cool thing is

you can serve uh you can

pretty much specify which uh models you

want to have. We have a whole watchdog,

too, so it like self-heals, too. So, uh

if you were actually starting to go to

more production levels, uh you can

actually uh you don't have to worry

about it. And then like I will show you

uh like

this past Sunday uh Kim A I went down on

our instance, and it uh self-healed.

And, you know, we have we have right now

like four nodes running it, so none of

like our internal users saw any problem.

It was a production ready um didn't have

to like think about it too much. And at

the end of day, yeah, like it's

it it makes it much more sim- uh simple

process to

uh have production work workflows.

going to show you Kim and K3 because

we've been using that a lot.

Um

I'm guessing a lot of people here are

very familiar with Kim and K3. Um

so, you know, 1 1 million contacts. Uh

one thing that's interesting on 1B 300,

uh can't get to the 1 million. You got

to use half of it. But one one thing

I've

noticed especially with our team, like

we're using it a lot for coding

uh agents.

Um that is very much

um

enough to do long task. Um It's

utilizing KB cache uh very well, too.

So, um in terms of speed of like time to

first second, um

it is very very very quick. And like in

the demo, I'll show you too with like my

coding agent running this Kim and uh

model on RunPod.

So, this is like our dashboard.

Um

It has a lot of metrics. You know, it

can it look looks like a lot, but uh

essentially, you know, we're always

thinking about like the GPU, like, you

know, have a lot of like GPU uh metrics,

you know, and what CUDA version you're

on, which is very very important. You

know, if you're running Kim and it

doesn't work on a 12 8 CUDA, it works on

13.0.

Uh and really it's what we have is like

the best we can we we can look at

benchmarking, look at all our data, we

can like playground with it. So, like

before you even put it into your like

coding agent or uh just hitting the

OpenAI uh response, you can start

testing real quick if this is working.

And just a lot of like more CICD process

of like, you know, how long are my cold

starts happening?

Um how are how are my metrics looking um

through actual traffic?

So,

So, really like I'm going to show you

real quick like RunPod itself. So,

um

this will So, we don't have this set up

yet. Like I said, um it'll be up

probably at the end of the month. Uh but

really all we have is just like a Docker

image. We can be able to

uh put that in right there.

If it loads correctly. But uh Uh, yeah,

so right now there's no B300s, but uh, I

have one running right now.

That's the fun thing about working in

this industry.

Uh,

you know, uh, supply can be uh,

dependent. Uh, but

[laughter]

But yeah, it's

in in the sense of that like, you know,

we make it really easy. You can put

storage volume. Uh, you deploy it, you

know, we have a part two so you can

deploy when you're available. So, you

know, if you were seeing the screen like

this, it will deploy whenever

it get it get started.

Uh, so that's like literally the

process. You click that button, benchpod

like starts up.

Um,

and I can show you like right here.

I have it on the wrong one, but I'll

show you guys here. So, like this is

like this is our actual like Kimi

running currently.

Um, so a cool thing is like

you know, you you start like it has a

whole air scan so you have actual data

to know like what is happening. Um,

cool thing is we have live metrics, too.

As like things are running. Uh,

all of this is also like backed up in

like storage data. So, like it's all

um,

um, backed up no matter what. Um, it's

all face fast API calls so your agent

could easily just call all this data.

You can actually just serve everything.

Like you don't even have to use the UI.

I I think I'm a big proponent of like as

we go to agents it's like I still want

to have like some awareness of what's

happening and having this gives me a

good like like eagle's eye view of like

what is happening with my uh, Kimi K3 K3

instance. Uh, but yeah, like you can see

here like I was going to show you like

uh, this past weekend we had a failure

with a GPU. Uh, we were able to fix it,

but uh, being able to just do a cold

start time like 9 minutes on it like K3s

pretty pretty good.

and the cool thing here is that we have

the serve so you can

start out experiments and essentially

like

apply them. It'll restart real quick.

Um, you can it's literally just like the

config YAML file from like VLM. And um,

so like for this like little demo what I

tested was we were just changing the max

number of batch tokens uh, that were

sent through.

And I kind of wanted to show you guys of

like, you know, what what kind of like

speed ups would happen with this. Um

I'll go right here.

So, I run these before because we're not

going to spend like 20 minutes just

looking at

benchmarks. So, these benchmarks are

running like concurrency. So, they're

they're running um

you know, starting with like just one

concurrent user and then eight, 64, 192.

And the cool thing is like we can be at

we can actually compare multiple runs

against each other.

You can kind of see like even just

increasing the match the the number of

batch tokens doesn't really change that

much the like time to first second.

But, interesting thing that I did notice

of increasing that is like the inner

token latency actually decreases quite a

bit when you increase that

batch tokens.

Which means it's like the request

itself, if it's especially running like

a long

like job essentially,

it's actually much faster finishing the

job than the base like config that like

you find on VLM recipe. The cool thing

is is like you can add this to an agent

and just start going crazy with it. And

you can start getting those answers much

much quicker,

have the data, and move from this if you

have it like on a pod, you can start

moving to, you know, a multi-node

cluster.

you're going to be like scaling up and

down

that that this is a perfect way of like

actually getting data. And just to give

you like a context on our serverless,

like serverless a lot of people will do

the same thing. They they just put

whatever the base config is and then are

struggling as like they're waiting for

workers to start up.

This actually gets you the real data.

You're now ready to go to production.

Makes it really really simple.

And just gives

It's like I always say, it's basing

everything in facts. And

I can like show you guys right here.

Like this is actually like Kenny

writing. So I was just like, what is a

bench pod?

I misspelled it. Sorry.

But that this is little command instance

running through it. Um

yeah, and Um you know, it is pretty

pretty quick. Like like I would say like

there's like a probably like 50 users

right now on our internal one and we

most of us are getting off cloud and

just using the and we have other

instances with like GLM, uh deep seek,

uh quant 3.8. Um but yeah, like I think

in the future we're looking to more at

like smart routers. Like how how do we

host all this without being reliant on

someone else?

Um

Yeah, so let me go back to our uh

thing real quick. So

just talking about it. If anyone wants

to screenshot that. But uh

Like I said, like you can you can

specify the experiment yourself too. So

if you were like, I'm going to have like

500 people using this all the time. Like

I want to like see how well this is

going to be. Um

But at the end of the day, like I said,

it is

how do we go from R&D to production? And

I think this is where like bench pod

will make it a lot more easier for

people to use.

Um

So like at the end of the day, like what

I want you guys to like know is like

um you know, pick a pod, get like a GPU

machine. You have your model, compare

configs. Don't just start with like the

VLN recipe. The VLN recipe is

proven, but it's not production ready.

Um

and then

test. Have your agent tested. And then

whatever, you know, based on what you

need, if it's ITL, if you need TTF,

TPOT, like metric, like optimize on

that. And then ship that winner. Go to

service, scale out, and now you can like

um

Now you have a working LLM model that

can be used internally, can be used for

our

Um

Yeah, it's it just makes it very very

simple.

one thing I would say especially with

like Kimmy that I've noticed is um

it's not always necessary for a team.

Like like a big thing that I think like

most people don't think about is that

um

you can use a smaller model. You know,

it it's a B 200 right now like a node.

That's like on our on our platform like

that's going to cost you 40K a month

essentially. And it might go up, it

might go down a little, but you know,

it's a very powerful machine and I think

when you're thinking about like how

you're going to be um

serving stuff if you want to be doing

this internally or for yourselves is

like what makes sense the best. And just

because it's the the frontier model,

maybe it doesn't make sense.

Um

like I said like like I run a deep six

on a 4 H 200s and does a really good

job.

Kimmy's more for like long tasks that

I'm trying to solve something like very

very detailed.

Um but like I said, the other thing is

like making sure your like tooling

choice is like on there. That is a big

thing.

Context like I I'm not going to be going

through too much about like context like

wikis and all that, but the ability to

have all that and like AI governance is

the next step that we are going to be

going with RunPod, but just something to

be always be thinking about especially

if you're going to be self-hosting your

um

your LLM models.

But yeah, is there any questions? Like

And we'll also be at the booths. So like

downstairs. But yeah, add me on LinkedIn

if you want to talk more. I'm always

exploring every new model like every

week. So

always have some like gotchas that I've

been learning through the through the

time I've been doing all this work. And

yeah, also scan the card QR code so you

can get

you get some credits and you can test

out RunPod. And and said, the template

will be out like at the end of the

month. So you can go play around with

it.