投机解码实战指南：LLM推理加速原理、权衡与工作负载评估

My name is Sheila. I'm a DA at Akamai, and we're here today to discuss the topic of speculative decoding and how you can figure out whether it's worth enabling for your workloads.

So, usually when you ask an LLM question, there are usually two phases that happen. First, the model reads your query, processes all the incoming tokens, and then creates what's called a KV cache, which is essentially the model's working memory for the rest of the query, and this only happens once. And then you have the decoding phase, which is when the model starts to actually record the response. Since each token depends on the previous tokens, this process usually happens one at a time. And so, imagine you're deploying an application that uses a large model, like the 70 billion Llama model. Usually it requires even up to hundreds of direct model transfers, and that's a very expensive process, right? And this can cause significant latency for your custom applications.

Which begs the question: what if you didn't have to use a larger model to handle token generation? What if you were to use a smaller model to make some guesses to speculate on multiple tokens, which is still an autoregressive process, but the whole point is that because it's smaller, it's a faster process. So, it would be a cheaper process. The output remains the same because the target model, the underlying model that has accuracy, still has to validate the tokens, and it does that in one straight pass. So, this is the idea of speculative decoding.

So, as I said, you have a small model that generates tokens, typically you can set them to three to five for each cycle. The target model will process the predictions, i.e. it will be able to scale, i.e. approve or reject them. And then for all rejected tokens, the target model will recalculate the token for that.

Good. So, this process is wonderful. It can save you a lot of money in terms of generating logical conclusions, but it is not free because you are now hosting two models. You need to consider the amount of memory required to accommodate the second model. Not much since it's the smaller model, but you'll also have to allocate extra space for the KV cache for both models. Yes, you would usually consider running speculative decoding when you have extra space on the GPU. For example, if your workload is highly parallel execution, your GPUs are probably already busy trying to fulfill all the requests. Therefore, it makes no sense to implement speculative decoding, but if you have extra space on your GPU, it's worth considering, but I'll talk about how to evaluate whether it's worth it.

And the first question is choosing a rough model. So, usually when you decide whether to turn it on, can the draft model predict as accurately as the target model? And then another question is whether it is as fast. Therefore, you usually choose a model that is definitely worth only a fraction of its cost. Well, those are two major decisions that need to be balanced, right?

I'll just leave it here. So, for my situation, I will explain why I chose these two models. For my demo configuration, when I ran it, I had access to a single GPU. I was working on a Blackwell GPU and wanted to run both the base version and the speculative decoding configuration in one. So, to be able to do this, I had to choose a model that was small enough to fit because I had to split the GPU usage between both servers. So, this is what it looked like. For example, the basic version takes up about 16 GB of weight, and the draft model takes up 2.5, so there is a huge amount of space left for caching KV.

And then when you think about choosing a model, some of the things you usually consider are, first of all, as I said, it has to be smaller, usually 10 to 50 times smaller than your target model. It should have the same tokenizer. Ideally from the same model family, otherwise, you know, you would have to manually convert between different tokenizers. Hmm, what else? Yes, these are the main things. I seem to be forgetting one thing. Hmm, and yes, and as I said, cost and accuracy.

Um, okay, so, oops. So when I ran this, three questions, speculative decoding, there are some workloads that benefit from this, compared to others that don't. Usually, since the goal here is to get a smaller model that can predict almost as accurately as the target model, it helps in some use cases where you have a workload that is more like highly structured, like coding use cases. For example, writing JSON or SQL queries. It would probably get more use than if you were using it for a very creative use case, like writing poetry, brainstorming, or something like that.

I think I have it. This is a fast app. I'm not even sure if this will work, but so here I have just two tabs for the first demo. Like I said, this is just to focus on how the acceptance rate works depending on the type of task you have. So the first step here is structured output. When I click on this, it starts at the same time. Wow. Oh my God. I'm just waiting. Technical problems. I'm not sure why my demo isn't working. Hmm, let's see. Oh, do I need to reconnect? Give me 1 second. I think I might have to restart. It looks like I need to restart the VLM servers and my system. Just give me a few minutes. Is this enabled? Oh, hmm? Should I? Yes, I think I need to. Oh, it shows up. Okay, here we go. Yes, I said it was a demonstration.

So when I launch this, I'll redo it. So you can see, as expected, with the speculative approach on the right, it was able to generate much, much faster, 1.6 times faster. The key thing to note is how high the adoption rate is, i.e. the ratio of the number of tokens accepted to the total number of tokens generated by the model project. Another metric here, you can see that with speculative mining enabled, we can generate a lot more tokens per second, which improves throughput.

And then if I run this for another case... So here's another use case. As expected, the same, but in this case, I think the main indicator to look at here is the low acceptance rate. And as I said, it's because it's more...there's more variety in how the next tokens will be generated, because, first of all, the temperature is set high, it's a creative use case...so what else? I think that's about it for this case.

And then another situation where...oh! Oh yeah, that's exactly what I meant...I should have posted it here, just to clarify when it would be useful. And then another use case would be the length of the context. As I mentioned earlier, there are two parts to token generation. Speculative decoding is intended to speed up the generation part, not the part where the model reads the query. So typically, if you have a use case where you need to provide a lot of context to the model, like for RAG applications or analyzing a lot of documents, in those situations, the model will spend a lot more time on the part like building the knowledge base caching, especially if you're using it for fast work where the generated tokens are much smaller than the context, like the input data.

Yeah, so I think the key here is how to think about what your app does? Is it highly structured? Do you have enough video memory to even consider this? Do you use more creative scenarios? Are you using a lot of parallelism? Or not? As if smaller party sizes would benefit from this. Yes. I'll leave this slide here. This is basically a summary of what I just went through.

And if anyone has any questions, I'll take one or two questions now. Wow, I can't hear you. Oh, yes. The question was, are there any specific tools I recommend for speculative decoding?

So, I implemented this using vllm, a service mechanism. They have really good documentation on how to get started with it. This is the simplest of the simple versions of speculative decoding. There are a few others, such as the n-gram method, Eagle, which is more like an architecture that is more advanced. But for starters, yes, I would say you can't go wrong by just reading. When I was doing this, there wasn't a lot of content on the topic, but I found a few blogs like the one where you type in "speculative decoding," one of the first blogs to appear on General Compute talks about it in more detail. There are also two research papers published by Google, and what other company? Yes. But it's more like something deeper.

Yes. But I'm also trying to maybe create more content on this. Maybe a video. It was a little difficult to get through that. I'm a little confused, but I would definitely like to maybe post a blog or make a video with snippets. Yeah, on YouTube or somewhere. But you can definitely find resources. Okay, yes. Okay, I should look into that. But I don't have one, I should add a QR code for that. Good.

Oh, as I said, I'm from Akamai, and if you're interested in learning how to replicate this, you can find it on our website, our GitHub, GitHub, Akamai developers. Yes. Yes, we have a lot of different resources here. I'm focusing on optimizing logical inference. We have other people who are focused on building AI-based agents. And on how to use our managed Kubernetes service, how to use Akamai features. So if you have time, please check out our GitHub pages, and we also have a Discord channel that we are currently trying to create. So it would be great to chat with some of you. Thank you.