自托管大模型推理的准入控制与拒绝机制

Uh before we move on a quick intro about us.

So I am a senior software engineer at Audible where I lead some of the MLAI infrastructure.

Uh I am also writing an open-source handbook called as LLM inference at scale. It has received over 300 plus stars by now.

I am also maintaining an open-source framework called agentship which helps you ship reliable and secure agentic harnesses in an ultra fast speed. Uh perv would you like to introduce yourself?

Uh apart from this if you would like to connect with us our LinkedIn QR codes are right there on this slide. Now back to you Hershel.

Second, we try to dive into admission control where the decision to say no lives and what it needs to see. Third, we will go through some of the proven strategies, the different ways a system can actually make the admission decision. And fourth, we will try to dive into the benchmarking where you will learn how to prove that your policy for admission is working or it's not working.

So let's get started with the motivation. Um and for this everything today uh let's try to understand what is inference. Everything today happens during this inference itself. So inference is simply a moment when someone sends a prompt to the model and gets back an answer.

So the step one is like a user prompt comes in. It's a sequence of words. Uh for example, it could be like failing a fix a failing test. The model which is running on your GPUs writes the answer one token at a time. A token is roughly like a word as I mentioned. Uh it writes the first word then uses that word to pick the second uh then it uses both to pick the third and it keeps on continuing like that and it cannot write all of them at once because every word depends on the previous one. So those once all the words are generated they stream back to the user uh who ask for the result and that volt prompt and the answer out is like that one request. Now I want you to hold on to one small detail like model has no idea how long this answer is going to be. It only finds out whether it writes when it only finds out when it writes the first last token.

Keep that in mind. It will matter in the few minutes.

So a model never sits on the internet itself. Every request travels through a short pipeline. It starts in your app or your agent. Then it reaches your API gateway which checks your API key counts how many requests you are sending. Then a router picks a replica which which is just one running copy of the model and finally the requests land on the inference engine. The common some of the common inference engines are the VLMs, SGLAN, Tensor RT etc. engine is what actually hosts your model.

It actually serves your model uh and it loads the model weights into your GPUs. It groups many requests into one batch so the GPU stays busy and for every request it keeps something called as a KV cache. In plain words, the KV cache is the memory the request builds up when it try when its answer is being written. Longer the answer, bigger this memory gets.

Keep this picture in your head. Every box here makes some decision but one decision is missing and we will find it out in the few minutes.

Every deployment has a ceiling and it comes from the three targets that pull against each other. Quality is how big your and precise your model is. Latency is how fast the first token arrives and every token after it. Throughput is how many requests your GPUs can serve at a same time. You know the old saying good, fast, cheap, but you have to pick two. It's the same here. You cannot pick all the three. You cannot prioritize all the three. So you can't So every product has to pick two. For an example like an interactive chat, you would want to go with the quality and the latency because users are more sensitive to that. It's quite obvious to the users if the quality is down or the latency is high. For something like an async agents, you would want to go with a higher throughput as well as quality. If quality is less, you will have to redo the task and redoing is very very expensive because these are the background and the asynchronous tasks. And the high volume real-time systems such as the autocompletes they would try to favor latency as well as and the throughput in favor of the quality because you don't need an expensive model for doing the autocompletes for you. But whichever corner you give up there is still a maximum load you can carry while hitting your SLO your service level objective. Let's say that on this deployment it's eight requests at once. In a normal minute, you never notice that limit. Now, picture the 40 requests arriving in the same two seconds.

It's such a high load.

And this is not like hypothetical. It has happened to like biggest providers in the world. Here are the two public postmortms two years apart.

On November 8, 2023, OpenAI after a day of unusual heavy traffic, maybe not 40 requests, it would be some million requests, their routing servers ran out of the memory. They started failing their health checks. They added more replicas, then had to roll them back. The way out was to cut incoming traffic and ramp it up slowly. In January 21 2026 enthropic the sonnet 4.5 tire gear got slow first then it started returning errors over about 90 minutes. The cause they gave was simple not enough compute for the demand.

And now let's look at what each team decided as their next set of the action item. Neither of them says we need a faster engine.

Open AI's followup is graceful load shedding. Enthropics is detecting capacity limits earlier.

It's the same idea in the different words which is decide on your request sooner

to make sure which request gets in. And there are more incidents which we have linked in the appendix. If you are interested definitely go and read through them.

We have a client, we have an admit layer, the Q router, and then the engine.

It's never the client talking directly to your engine or client talking to your queue directly. There is an admission phase in between.

Admission is a decision made before a request takes any GPU memory and it only has three possible answers.

Admission decides whether to start now or you want to hold the request. You want to accept it but make it wait. Third is the reject. That means not right now. It can throw different amount of errors here such as 429s and we will dive into this bit more deeper in the second part.

this that is the decision that was missing from our pipeline.

So why does it have to happen before the engine and what would happen if you will actually accept the request what would be the cost of that yes so let's try to take a look at this like two phases of the inference like prefill and the decode. So first comes the prefill. That's the engine reading the W prompt, building the request KV cache. And then comes the decode. That's the engine writing the answer one token at a time. With every token, the cash grows a little more. All of that memory stays reserved until the request finishes.

So a yes is not a moment. It's a commitment that lasts the request whole life. And here you can look at the example.

There is a request A, there is a request B. Once you admit a request A which is a short answer request and then you admit a request B which is a long answer request. Now when you admit a request C which is again a short answer request, we don't know how longer it would take because there is a longer request which was already in the memory. Should we even take that request C or should we not even take the request C. So there is a cost associated of like admitting every request because admission could still result in missing out your SLOs's.

Another impact apart from the SLOs's is your memory.

Here is what it does to your GPU. So every running request it keeps growing your cache as it writes. So your total memory keeps climbing. When the memory gets to the full engine has three ways to cut. Either it will aait or like throw away your request. Either it will put it to your waiting state that means it will swap or it will just say okay I won't be able to serve and I I will probably return you an error. Every one of those throws I mean every one of those causes a certain delay in processing of the request. and missing out your SLOs's again while also wasting your memory.

So why is it like our problem like we have con servers from open AI anthropic bold teams which are serving their own infrastructure for the inference.

So it's a problem for us because there are teams who want privacy. They want to have their own models hosted on their own GPUs and they want it to be hosted on their own GPUs due to the cost perspectives but they have to also protect the burndown on those GPUs.

So they need to effectively decide which request to serve across their inference ecosystem.

On top of and on top of that like the traffic arriving at the door has also changed. Earlier there used to be a person that was chatting. He would send one request at a time. There would be a seconds of reading in between and then there would be a slow answer. But now coding agents they have taken over. There are tons of calls that these coding agents they send along with the retries, pauses, um like the undo calls as well and it there is such a overwhelming traffic that your cluster has to handle and what that means is cluster has a responsibility to take a wise decision of which requests to accept and which uh requests to reject.

And this is exactly that visualization. You ask your agent that fix an o test. It runs all your it tries to run multiple tool calls that is reading of the files like around there could be like 14 files. There could be like around 10 tests that it has to run and it would combine together to make 40 50 calls for just one request and all these 40 requests they require like reaching out to the LLM getting back the response. Uh and in between some of these requests would be long running some of the requests would be short prompt request. So we need to decide effectively where which one to pick and which one not to pick.

Yeah. And that was the like reason why we need to worry about admission control. Why we need to worry about handling the traffic right way across our infancy ecosystem.

Uh now moving on to the next slide. Uh that is slide number 18. So in this slide we will be covering three questions. Where does this decision from? which request to admit and which to reject sits back at your stack and the another question is what does it need to see to make a good call or judgment on the request and what does it returns back to the client. Moving on to the next slide. So here uh we see the pipeline again which each layer's jobs is written out. The agent sees its own call and decides when to retry.

The gateway sees API key and request count and enforces a rate limit rate limiter for each and the router which sees replicas or healthy and which one to pick. Now the engine sees its KV cache and the current batch and decides the order of request to run and what to evict and print.

So every layer decides something. Now ask each of them one question. Is the room for this request?

The agent says not my job. The gateway says I cannot see it. And the router say not my job and the engine says I only find out when I have already said yes.

The admission box is missing in in between.

That means keeping your own modified copy of BLLM or SG lang and updating it every time original changes forever. So an ingress is the only movement that works with the engine exactly as it ships.

So that's where we will stay for the rest of the talk. So now moving on to the next slide.

Now moving on to the next slide.

uh to make the call well the policy needs two views at the same time from the request and from the engine. So from the request how long the input is, which client or tenant sends it, which agent session it belongs to, and a deadline. And from the engine point of view, how long uh KV cache is, how long the oldest request has been waiting in the queue, how often the prompt is already cached, and how how loaded each pool of replica is. Now look back at the ST. The gateway has left side and the engine has the right side. Nobody has the both.

And that's really why admission has to be its own layer. Now moving on to the next slide.

If the input is too big to fit in the memory headroom just refer it. If a client or tenant is over it share reject it with 429. If one agent session has too many calls in flight defer with extras. If waiting would make a request miss its deadline anyway reject it from the engine side. Uh you can see at the bottom half of bottom half side. While KB cache is under high water mark admit it. When the oldest request has waited longer than your target defer it. If the prompt is already cached admit it first because it's cheap to serve. And when every pool is full, reject it with 529. One rule of thumb, pick the signal that changes in second, like a QH, not a one minute average. A coding agent sends 40 calls in 2 seconds. A minute average won't even notice. So that's what the policy reads. Now what does it say back? Now moving on to the next slide.

That's the commitment from part one just in the form of an HTTP response which is why the admission decision has to happen before 200 never after it.

Now the client side if we are writing the agent here is how to handle 200. Watch the stream not the status code. First time is properly start your timeout clock when the first token arrives not when you send the request under the heavy load prefill alone can take few seconds. Second, if you decide to give up, actually close the connection. The server only frees the KV cache when the request finishes or the client disconnects. An agent that times out but leaves the connection open is still holding the GPU memory. And the third one, uh, a stream can fail halfway through, right? And that that's no status for that, right? and you're you're left with partial answer and you have to decide whether to retry or carry on. In short, 200 promise to start. It's not a promise to finish. Right?

The second answer is dipper which basically means not yet. The server sends back 503 with retry header. The header tells the client how many seconds to wait. The policy picks that number from the engine. For example, how long until the running batch finishes and freeze the enough cash and you differ when capacity is coming back soon? It's a burst, not so not a trend, right? And for example, like if you think about it like Q age is rising but the system isn't saturated.

If the capacity isn't coming back anytime soon, a dipper just kicks kicks the can down the road, right? And in that case, we can just reject it with the honest answer.

Moving on to the next slide. On the agent side, handling a defer is simple. Sleep for retry after time and a little random jitter, then retry again. The jitter just means a small random extra delay. So client don't like don't call or like don't at all come back at the exact same moment. They will have to wait.

Look what the server sees from 10 retries without retry after every agent picks its own delay. And that's usually that's usually zero. So they all come back at once. That's how 40 calls turns into 80.

With retry after and jitter the retry spreads out across the window serves asked for.

Okay. So if if we see the uh reject reject returns 429 or 529 the third answer is no. Uh and there there are two different codes for it because there are two different reasons. A 429s means too many requests. This particular client or agent session is using more than its share. The policy saw something like per client load, the number of concurrent calls or a kota. The problem is the caller and then the 529 means the provider is overloaded. Enthropic uses this code. Every replica is sold. So nobody is getting in right now.

The problem is pool not the caller.

Now uh handling a reject. So how should agent react on 429? Reduce your own load. Cap how many calls that agent has in flight at once.

And on 529 retrying harder only makes this worse. Waiting longer switching to different model provider or park the task for later. and one either and on either one keep a retry budget. Use expansion exponential back off with jitter which means waiting longer after each failed try and after few tries tell the user what's going on because a reject that gets retrieded at full speed is how you end up in one of the postmodm from the talk at we described at the starting.

Now over to you Hershel

Thank you P. Uh so this is the module three where we will try to learn about the different strategies uh that have been rolled out for the admission control. So far we have seen that no layer in the today's stack asks where uh whether there is a room and that layer is ours to add and now we give it a brain. So we will look at why counting requests isn't enough. Breaking a policy down into four different choices. See what four different research paper chose and finish with a decision tree that you can use onto your own ecosystem.

So let's start with what most team already have which is counting. Yes. Counting the number of the requests. A simple idea is to rate limit at the gateway which gives API key a fixed allowance set in advance. It's great for keeping the clients fair, keeping your tenants fair with each other, but it does nothing to protect your GPUs because it doesn't change with the load that is happening onto your GPUs or your cluster. A fixed cap at the proxy gets closer. It counts how many requests are in flight on each replica. The problem is that the quick crep and a long planning to call count exactly the same while the real capacity of the GPU changes with every batch. So in practice it mostly cues and clients timeout ends up doing the refusing.

Neither one knows whether our 40 calls are cheap or expensive.

The cost of air policy does though and it estimates each request will actually cost what each request will cost in the KV memory and the tokens using the live state from your engine. And it says no before the request actually takes up the memory. So the short version is count is not a cost.

And whatever it is, it has to actually move when the system gets busy. So three is the rule which means how does the signal gets turned into a decision. So rule is used for that and that can be anything from a simple threshold that takes microscond to a full optimizer that takes seconds.

Fourth is the action. Once you have the rule that gives you a decision what do you do with it? What does your how does your client gets impacted through those actions? That's the admit defer and the reject that per just talked about in the module two. Each choice limits the next one. For example, a deadline signal only works if your request actually carry the deadlines. So let's go through the first three.

Choice one. Do you admit a single request or do you admit a poll sessions? Say you have two agent task 20 calls each room for almost half of them. You can uh point you can see the top row. So if you admit per request about half of each task gets in but neither task can finish because each one is stuck waiting on the calls that didn't make it. You have used all the capacity and completed nothing. If you admit per session, session A gets all of its call and finishes. Session B gets one clear answer. Come back later. And this picture is illustrative, but the logic holds.

Our 40 calls are one session. Coding agents send sessions and yet most published policies admit only the single requests.

So which metric do you watch for what particular problem? This is the question that we are trying to answer. It depends on the problem that you are trying to prevent. If you are if you want to keep the cash from overflowing, you definitely have to watch your KV pressure.

If you want to avoid your wasted recomputee, you have to watch out the KV used together with the cash hit rate. A full cache is fine if it's being reused.

If you want to keep your weights to be shorter, you have to watch for your queue depth and you have to take actions based on that. If you want to meet your latency targets, you need to watch your deadline slack. Take actions based on that. If you need to plan for your long answers, you need to predict for the output length.

And look at the last column fails when it tells you when each signal stops working. Look at the first row. KV pressure fails when the cache never fills up.

Remember that row? It comes back to bite someone in the part four.

So what are the four ways to take that decision?

That is how do you convert your signal into a decision from the cheapest to the most expensive. Here are the ways a threshold just compares one number to the one limit. Example is a policy named clued weight that was uh created somewhere in 2026. It takes tens of microsconds per decision.

Second is a dual gate which needs two signals to agree before it acts. Example is a paper called concour uh which checks like KVUs and the cash hits under 100 microsconds.

Third is an optimizer which solves like one big planning problem across all the cues at once. Example is a QLM. It's thorough but it takes about 5 seconds per decision.

And the fourth is the formal bound uh which does the math on the worst case. Can this request still meets its deadline? The example is the Kronos on which uses the like time to first token and it uh provides you like a microcond level latency.

And now hold that cost column up against our agent. It sends 40 calls in 2 seconds. A 5-second decision shows up after every party is over.

So each purper used a different model engine and a workload. So please don't read this as a leaderboard.

The question that matters for this room is does any of them fit a coding agent and the answer is no. We placed every paper on the two axises. One axis is what it optimizes deadlines or the throughputs. Other is what it admits single request or the VO agent sessions. A coding agent needs a policy that admits bold sessions, works online with live traffic and runs engine you haven't fought and nothing sits there cleanly. Conc doesn't need a for but it was only tested offline.

So in practice you will pick the closest match and build just build your own. And here is how you can choose which polic like which kind of a policy you want to try it out.

So here is that decision tree.

First the question that you have to answer is does your request carry any deadlines? If yes, set up a some sort of a formal bound on that deadline. This will take some microsconds level of latency.

If there is no deadline requirement, the second question that you can ask is does the KV cache fill at the peak load? If your KV cache fills at the peak load, then you would want something like a way to detect the cache prefixes used. So you can use some sort of a dual gate mechanism or a KV threshold mechanism. Um and then if your KV cache gets filled at the peak load, it does not gets filled at the peak load, then you what you could do is you can set up a Q level of threshold on it.

And that gets us to the benchmarking like once you have a policy, how do you benchmark?

So I will be handing it over to pop.

Four is a score. Then they run the same trace twice. Once with the policy and once without and then you can compare your result. The score is where the most people go wrong. So let's look look at that. Now moving on to the next slide which is uh how how you score. The score we recommend is called goodput under admission. So you can see the formula on the screen. The top of the fraction is the request that met their SLO their latency target. That means the answer started on time and it never stalled for too long between tokens.

The bottom of the fraction is every request that arrived including the one which got rejected. Why does the bottom matter so much?

So try this. Uh refuse 95% of your traffic and serve the remaining 5% perfectly. If you only count what you admitted, your SLO attainment is 100%. Looks amazing, right? Just ship it. But no like good put under admission says 5%. Shedding looks three when you only account the request you kept. It isn't.

The second thing to get right is the workload. A benchmark only tell you something if it looks like your real traffic.

So replay agent traces with their fan out retries session not uniform random arrival.

Describe load as multiple of the capacity you measured and sweep from half of it to double it.

Set your guideline as multiple of your latency with no load. And remember a benchmark that never goes past capacity never actually tests the policy because the policy never has to say no. So you have to make sure that you reach at that point.

Now move moving on to the seven uh seven things every admission result should disclose.

Signal liess is one of the seven things every admission result should disclose. Here is the full full list that you can see on your screen.

The load as a multiple of measured capacity deadline sets relative to latency with no load.

Signal liveless which is no paper report.

How many times you repeated the run and how much the result varied not one lucky run. Where each parameters comes from, turned on this setup, borrowed from somewhere else or left at the default.

Whether the rejected request count in a good put denominator and the full configuration and the full configuration is model, engine, version, hardware, workload, SLOs's and baseline. None of this ask anyone to change how they evaluate. It just ask them to say what they did. Use it as a checklist when you read on on a paper and when your benchmark and when you benchmark your own policy. Doing all seven by hand is painful. So we built a tool that records all of this uh as as a default.

So if you also see we on the right side uh it's uh in in a benchmark it's a harness that runs any policy against a real engine on a real GPU and reports what the policy cost both sides calls the same function policy decide. So the policy you benchmark is exactly the policy you deploy.

We build it because across those 16 paper no two share the same baseline engine workload or SLO admitf makes a head-to-head comparison possible.

Now moving on to the next slide.

So using its pive command. So you can see on the screen we have five commands. So infra up starts a real VLM on a real GPU. Infra smoke checks that it's actually serving. Then we have bench runs every policy several times each. Then we have bench computer tell you which policy won and how by how much. And then we have infraown which which basically means it should shuts all all off so you stop paying. And that's my favorite one.

Uh and so now provisioning and benchmarking are separate commands on purpose. Loading a model takes minutes and every policy has to face exactly same deployment.

If you are reprovisioned between policy, you would change the very thing you are trying to hold a constant.

The deployment is constant. The policy is variable.

Um so here what we did is every decision is recorded with the state it saw. Every run leaves behind a folder you can open inspect. There would be four files as you can see on the screen. Those are manifest JSON summary JSON decision JSON and uh out outcome JSON. So what each does is uh you can take a look deeply once you run the results. Now we we will be looking at the shading cut. So shade shading 37% cut tail latency. So here are the four results how they will look like on a real GPU. The setup small model we used quen 2.5 uh and it's a half billion model on a A10 GPU. The engine was kept out at four sequ sequences at a time. Eight request at 14 per second two repeats.

If you look at the first row with no admission control, the P95 time to first token was about two two and two and a half around 2 and a half seconds and P 45 sorry P95 means 95% of the requests were faster than that and it swung by about 700 milliseconds either way. Now the last row if you see with a queue up policy that allows at most two request waiting 60 63% of requests were admitted and the P95 dropped to about around 1 second that's 2.5 times better with the exact the good same put now if we look at the signal matter more than the threshold but this This is the result I love most. Same run. KV cache usage peaked at half a percent. The queue got 24 requests deep on a small model. KV cache is huge compared to the four short sequences. So it never comes close to filling. All of the pressure showed up in the queue. Instead, a KV cache pressure policy would have botched a flat line and admitted everything. that that's the row of choice from two. So

now here and you you can try it today. You don't need a GPU. There is a small fake engine that speaks the three endpoints admit puff uses and it gets busy under load just like a real one. So you can test the whole setup on your laptop. If your right policy is your own package, admit perf discovers it automatically. No change to the repo and contribute contributions are very welcome. New policies, workload, engine adapter or matrices.

So if you want to remember like one slide just make it this one that when you are accepting the request in your inference clusters you need to have a mechanism to say no early something that works at the ingress something before saying yes to the request before that request goes into the GPU before that request takes up your memory.

and per added clearly. There are a couple of error codes that your admission policies return. You need to handle those requests and those errors.

And with that we would like to come to the closure. We would like to thank you all for your patience, for your time. And here are like two of our projects. It all started as part of the LLM inference at scale handbook. We and when we were doing the survey, we found the gaps and we prepared admit puff. So here is the source code for it. Feel free to go to it and like experiment with it. Raise the issues, contribute to it. idea is to build your admission policies, benchmark them and be confident that it uh helps you keep your cluster stable. Yeah.