Harshal Jain: Good afternoon everyone. My name is Harshal Jain, and he's Tanmay Shah, and we would like to welcome you all to this two-hour workshop on LLM inference.

Harshal Jain: The goal of this workshop is to understand this domain from the first principles and dive deeper into it, to understand what's going on throughout the industry. A bit of background about us: I am a senior software engineer at Audible. I have been building media data platforms for the past five years, and on the side, I have been writing this open-source handbook on LLM inference. Next to me is Tanmay, who is a senior quantitative modeler. He recently completed his PhD, and he has been actively doing research in agent verifiers and world models.

Harshal Jain: So a quick show of hands here: who here is brand new to LLM inference? Okay, great. And who here has deployed models in production, been tuning them, and serving production traffic? Okay, great.

Harshal Jain: So this workshop is targeted towards the beginner and intermediate levels, and all of the slides and exercises are in the repo. I will share that soon. Here is the quick agenda for the workshop: we will start with the problem statement. We will try to understand a few of the pain points around LLM inference, then understand what causes those pain points and build our foundations from there. Then we will dive into two kinds of optimizations that we do: model optimizations and serving optimizations. Then we'll start learning about different serving engines available to deploy our LLM inference solutions in production, and we will showcase some benchmarks and a decision chart on which engine to use.

Harshal Jain: To understand the pain points, first we need to know what LLM inference is. Probably a lot of us already know this, but anything that you ask your AI to do—whether it be generating video or audio, analyzing text, analyzing your medical reports or tax bills—all of that is LLM inference.

Harshal Jain: This market is approximately twenty billion dollars today. SemiAnalysis recently shared that if you want to model Google search queries with LLMs, you need a massive capital outlay, and query cost has to be less than 0.5 cents to keep your search business profitable. On the other hand, Business Insider mentioned that your AI has to be put on a diet, and everyone has to start auditing and budgeting their token usage.

Harshal Jain: And all of this is happening, why? Because your hardware is limited, compute is expensive, and your inference is expensive. Because of the growing usage of AI, this causes costs to rise rapidly.

Harshal Jain: This stat is an older stat from OpenAI, but it is still true. If you look at the training cost of GPT-3, it was around 4.6 million dollars, which was a one-time cost. But the inference cost is a recurring cost because it is an operating cost that scales with every user that comes in, every token that comes in, and every session that is initiated.

Harshal Jain: There are basically only two ways to counter this: one way is to reduce your token usage. Alternatively, you should try to optimize your inference solutions as an inference service provider for your customers and for yourself. Because of this, we have been seeing a lot of new solutions coming out every day. The idea is that we will try to build those foundations that will help us understand and evaluate whatever ships next.

Harshal Jain: To get started, we will do a quick short demo of what the different pain points are around inference. This is the repo—you can pull it or open it on GitHub. It's called LLM Inference. A bit of background here: four months back when I didn't know anything about LLM inference, I started learning it and saw that a lot of resources were scattered. So we started putting it altogether in one place so that it could benefit people.

Harshal Jain: Let me actually get out of this slide show mode and probably go to this extended mode.

Tanmay Shah: Okay, great.

Harshal Jain: In this repository, if you see the README file, there is a link to the slides. It's a folder where you have the PPTX and the benchmark report in there, and you can download it. Then for demo purposes, we have a couple of Jupyter notebooks. We collaborated with a platform providing GPU instances like RTX 6000 with large VRAM. We have already set up these notebooks so that it becomes easy to experiment with, and all of the assets are preset for you.

Harshal Jain: We will start with a simple demo. When it comes to inference, you need to do inference on a certain model. For the workshop purposes, we are using a simple Mistral-7B model. It's a small model of around 15 GB in size. We are going to load that into the GPU, and we will look at some of the GPU stats as well. I'm not running the cells live because I don't trust Wi-Fi at conferences, so I will be going over the results that we ran previously.

Harshal Jain: The first thing that comes to mind is: what does my memory consumption look like when I do LLM inference? I load this model, and I see 15 GB used here, leaving roughly 87 GB free. Now when I run inference, what I notice is that the more input tokens I pass, the more memory I need. It increases slowly, but it's still increasing. Imagine if you have a context length around 4,000, 16,000, or 32,000 tokens—this memory could grow really large, and you could hit out-of-memory issues. Definitely, this is problem number one: memory increasing with the increase in tokens.

Harshal Jain: The second problem is that the Time to First Token (TTFT) can be very slow. It is measured by the metric TTFT. When you measure TTFT against the input size, you see that the longer the context, the slower this TTFT becomes. So now there are two problems: your memory increases with token size, and your TTFT increases with token size.

Harshal Jain: The third problem is throughput. Throughput is about how many tokens you can serve per second, and how many users you can serve per second. If you take a naive implementation on your local system, it executes sequentially. If you send five requests, all five requests will be catered sequentially, so your requests take significant time to complete when you have multiple users. These are the three main problems.

Harshal Jain: There is a fourth problem. I haven't described it here; probably we will build that intuition as we move forward. But let's remember these three main problems: Memory, TTFT, and Throughput.

Harshal Jain: I will go back to the slides.

Tanmay Shah: Okay, perfect.

Harshal Jain: Is this visible?

Tanmay Shah: Yeah, looks good.

Harshal Jain: Within that repository, if you look at the workshop folder, you will see the README, and the README has all the links to the slides and demos. Does that work?

Tanmay Shah: Perfect.

Harshal Jain: Let's start working through the foundations. Let's start understanding the reasons behind those pain points. For that, we have to look at this inference pipeline: We get an input text, which could have any number of words. You convert those into tokens. For simplicity, you can assume one word equals one token.

Harshal Jain: Then you convert them into embeddings, and then you send them to the transformers. There are 32 layers of transformers in the case of Mistral-7B, though different models have different numbers of layers. Then you generate a new token, and that token goes back to the input, then you generate another token, and that keeps on going. Now, in this entire pipeline...

Speaker 1: You would see like 95% of your compute is taken by these transformer layers. So it's worth looking at what goes within this transformer layer. Within this transformer layer you have more layers: you have a normalization layer, you have an attention layer, you have a feed-forward layer and all. And attention layer is the one I think that has been very, very famous — "Attention Is All You Need", I think that's very well known. So attention is the most compute-intensive layer, and we need to understand what goes within that attention layer.

Speaker 1: So what does attention do? If you have an input text, it needs to find the attention scores of every token with respect to all of the previous tokens. And to do that, what it needs to do is it needs to project every token into a key, query, and value space. So in simple terms, just understand this: if you have 10 tokens, then it needs 10 different query, key, and value vectors; if there are 100 tokens, you would need 100 key and value vectors; if there are 1000 tokens, you would need 1000 key-value vectors. And so the number of key and value vectors increases as you increase the input size.

Speaker 1: And if you calculate the KV size per token for Mistral 7B, it comes out to be 131 KB. This is because you have to multiply the size: one vector is like 128 dimensions, you have to multiply it by the transformer layers (32 layers), and then you have to multiply it by the KV heads. For Mistral 7B, it's 8 KV heads, it's not 32 because it uses a different kind of attention mechanism, which we will talk about for sure. But yeah, so the KV size per token is like 131 KB.

Speaker 1: Now imagine if you have 4k context, so that size becomes like half a GB (0.5 GB). If you do like 16k context, that size becomes 2.1 GB. Now multiply that by the users: assume you can serve multiple users together at the same time within that GPU, you could have like 42 GB with a 4k context and 80 users at the same time. If it is only let's say 24 GB, you are already running out of memory. So you cannot serve that many users with that much context.

Speaker 0: To visualize this, look at a GPU memory.

Speaker 1: So the GPU memory has model weights which are pretty fixed — these are pre-trained weights. There is like an overhead that is also fixed, that changes but it does not change that much, overall you can assume it's fixed. And then there is leftover memory. This leftover memory is what's being used by your KV cache, like key and value vectors. So assume you have one user, you can only store that many key and value vectors or that many tokens which can fit in this entire memory that is left. So we can show this with a simple demo too.

Speaker 3: Okay, great. Okay, great. Let me see if I can actually run this... okay, great, yeah.

Speaker 1: So you would see the GPU is allocated. So here we are just trying to confirm the memory based on the math and based on the intuition that we have built. So the model memory is, let's say if you have 7 billion parameters and you are doing 16-bit precision, your total memory comes out to be 14.6 GB, which you can basically verify with the math. If you do all that math, that comes out to be 14.6 GB. Now comes the KV size, so this KV size is like 131 KB per token. And if you do that math and you try to visualize this...

Speaker 3: Okay, and then let's just visualize this, okay great.

Speaker 1: Yeah, so this is the memory chart. So if you see, as your context increases, your memory keeps increasing. Then another thing to realize is, as your users increase, then also your memory increases. So if you want to serve like 160 users on a GPU, you can only support lesser context length. So there is always a trade-off between what context length you can serve versus how much cost you can save by putting multiple users or concurrent users into a single GPU. So you have to always pick that trade-off, and we will go through that in a couple of slides.

Speaker 1: Can you repeat please? I'm sorry, I cannot hear you.

Speaker 0: Do you mean different tools, different product names for... wait a second...

Speaker 3: Okay, so let me go back.

Speaker 1: So that was about memory. We need to understand why we had a slower time to first token when we increased the context length. So for that, we need to understand the two phases of inference, and those phases are the prefill and the decode phase. I think you have all seen a lot of articles, but we just wanted to explain it.

Speaker 1: So when you send these input tokens, what you want to do is you want to build those key and value vectors that I mentioned for all the tokens. Then you want to compute the attention scores of every token with respect to the previous tokens. All this operation that you do is very matrix-heavy, it's a very compute-heavy operation. And we all know GPUs are very well suited for a heavy compute workload. So we call prefill to be compute-bound, and it does take some time to complete. So whatever time that this phase takes to complete, that's your Time to First Token (TTFT). So if you have more input tokens, you have to generate more key-value vectors, you have to do a lot more attention math, and because of that your TTFT becomes much slower.

Speaker 1: Whereas once you generate one token, you need to keep doing this to generate subsequent tokens sequentially, one after another. But in that process, every time you have to use the key and value vectors of all the previous tokens, which is same as the prefill where you were building key-value vectors. But in the decode phase, you are only computing the attention math for the single new token. And that is why it's less compute-oriented, and it's also called memory-bound. We will see shortly why it's called memory-bound.

Speaker 1: So in a classic timeline, you would see prefill and decode phases like this: time taken by prefill, that's your time to first token; and then time taken by every decode step, that's basically your inter-token latency. So that's the core metric that you need to worry about: what's the time being taken by your decode step?

Speaker 3: Okay, now why does that decode step take time and why is it being called a memory-bound operation?

Speaker 1: Let's try to understand that. To understand that, we need to look at how the matrix math basically works on the GPU on a high level. GPU has two kinds of memories: you have High Bandwidth Memory (HBM), and you have Shared Memory. The High Bandwidth Memory is larger in size, but has lower bandwidth — by lower bandwidth, I mean you can transfer data out of it at a lower rate compared to the shared memory. The shared memory is smaller in size, but it has very, very high bandwidth, which means you can transfer data in and out of it very fast.

Speaker 1: When you have to do matrix math, you have to pick the data in chunks from the High Bandwidth Memory, put it into the shared memory, do that math, and write back the result into the High Bandwidth Memory. For the prefill phase, you have to do this matrix math only once. But for the decode phase, you have to do matrix math again and again because you are generating each and every token sequentially. And so it doesn't matter how fast your decode compute is, because now you can only transfer your data out of the High Bandwidth Memory into shared memory at a certain speed since you are limited by the HBM bandwidth. And so that governs your token ceiling — at what rate you can actually generate tokens out of the decode step.

Speaker 1: If we look at this in the roofline plot: there is a left section which is called to be memory-bound. Mathematically it's governed by the arithmetic intensity. Arithmetic intensity is the number of FLOP operations that you perform per byte of data being transferred. For the decode step, since you are transferring a lot of data — like the key and value vectors of all the previous tokens and the model weights — but you are doing much less computation because you are computing attention math for only one token, its arithmetic intensity is very low. But for a prefill phase, you...

Speaker 1: You are transferring the data once, but then like you are doing this heavy computation. So its arithmetic intensity is very high. So now you know like in terms of mathematics why arithmetic intensity of Prefill is very high compared to your Decode, okay.

Speaker 1: We are loading the model now. This is like the Prefill cost. So what we are basically doing is we are getting the input text, and then we are trying to generate. In the Prefill step, the amount of time it takes, we see like as we increase the size of the input tokens, this Prefill is increasing, and this is the reason why your TTFT increases. And then like your Decode time: the Decode time is like on average it stays about the same. And so assuming you ignore that cold start, your Decode time is approximately around the average line. It is still impacted by the input size; it's not like it's a constant time, and it is because it still needs to pull the Key and the Value vectors from the memory for all the previous tokens. So there is still basically a small increase in time that you would see with the Decode step. And then this is the classic Roofline plot, okay.

Speaker 1: Now let's try to understand like the throughput dimension. You want to understand how many users you can actually serve. And I think we saw like a diagram of the GPU memory where we saw okay, there is some memory that is free for the Key and the Value vectors to grow. So assume like you have just a single user, what's the total KV size that you have you can basically support is defined by context limit. The max users that you can support is whatever is your GPU availability, like whatever is the memory that is available in the GPU, you divide it by the Key and the Value size per user. And when you do that, it comes out to be like your concurrent users.

Speaker 1: Now assume like your GPU is fixed, your model is fixed, so your KV size per token is fixed. There are only two dimensions that are left here, which is context and your concurrent users. If you want to serve more concurrent users, you have to reduce the context length. If you reduce the context length, you could impact your quality. So these are the two dimensions right now that we are trading off.

Speaker 1: But can you actually serve the max number of concurrent users in an ideal [scenario]? Probably not, because every business has like latency SLA that we have to meet. So if you remember like in the Decode step, I said that time for the Decode still increases if you have more inputs, it also increases if you have more users. So ultimately inter-token latency also gets impacted if you have like a higher batch size, and your TTFT also gets impacted.

Speaker 1: So now there is a third dimension you have to worry about, which is like your latency. So the three dimensions that you have is like quality, latency, and throughput. So it comes out to be like this trade-off triangle where you have to choose between them.

Speaker 1: So for a premium chat application, you would want to prioritize definitely the quality, and you want to prioritize like the latency. You would not want your users to wait for a larger latency. You can always sacrifice the number of users you can support on the GPU and probably take that cost in form of more instances. And like if you consider an async agent workload, you would want to prioritize definitely quality and throughput, because these are the long running tasks and you would want to serve as many concurrent tasks as possible, but with a very, very higher quality.

Speaker 1: And often like we think like okay, the GPU is like a very expensive GPU that might not be a good fit for us, but it turns out that could actually serve you the lowest cost per million tokens. But you really have to trust your kind of calculations on the max users that you want, and you really have to make those estimations correctly.

Speaker 1: So for the capacity calculator, there is like a link to the Colab. Because I was facing certain issues with the environment, I had to migrate out the whole widget library and I didn't have time, so I just picked Colab. So what we have done over here is we have tabulated some of the GPUs with their VRAMs, bandwidth, the TFLOPs, and the cost per hour.

Speaker 1: Then we kind of like built this simple capacity calculator. This is just a KV visualizer where when you increase the number of tokens, you see KV size increases; and when you increase the number of users, your size is increasing at a much faster rate. And then in this capacity calculator, if we have like a model which is like a 7 billion parameter model that we selected, we set the precision to be FP16. Now the way we basically go by the GPU decision is you have to fix one dimension first which you care about the most. For premium chat dimension, latency is definitely the one, and then for async workload, the minimum batch size that you want to serve from like a single GPU that is the second dimension. So you want to fix these first.

Speaker 1: So I will go about like in a premium chat application: I can go ahead with like 10 milliseconds latency, a minimum batch size I'm okay with probably 7 concurrent users on a single GPU. And then like my context limit is very important to me because I want to focus on the quality as well. And so I do see like some of the GPUs: the H100 80GB it's like [a few] dollars per hour, but like MI300X is around 10 dollars per hour. But if you do all that throughput math that we shared in the mathematics before, you could find like a cost per million tokens that could be significantly lesser. So you need to do such calculations by fixing those dimensions, and you need to decide your GPU to like reduce your kind of inference costs. This is at least the first step that you can take towards optimizing the inference.

Speaker 1: So now the next thing is about the model optimization. So we are now basically have built that foundation where we understood some of the pain points, reason behind those pain points, why those were happening, how we could like address that GPU capacity thing. We need to understand what can we further do about it. So it is about model optimization, and I would like to invite Tanmay, and he can talk more about these model optimizations provided he has worked on this during his research times, okay.

Speaker 5: Hi everyone, mic check. Am I audible? Yeah, okay. So, hi, I'm Tanmay. I work as a senior quant modeler and also I'm an AI researcher. My work focuses on agent verification and right now building world models. So for this model optimization, before we start model optimization, I created a research template so that it will be easy for us to understand all these complex things.

Speaker 5: So our template is simple: first we will identify the problem; second step we will solve the problem using two algorithms. First algorithm is called Ostrich Algorithm: whenever we see a problem, just like an ostrich puts their head into the sand, same thing we will do whenever we face a problem, we will just ignore it. So this is an important algorithm we should follow. Second one is called World Cup Algorithm: for example, we don't know who will win this FIFA World Cup, so what organizers did they break the 48 teams into 12 groups, then Round of 32, then Round of 16, then Quarterfinals, then Semifinals and Finals. So what they are doing is breaking it into smaller problems and the useful results are moving forward. So same analogy or same algorithm we will use to understand this model optimization and all those things. So yeah, let's start: so I have one H100 GPU, I have to use this open source model...

Speaker 5: It is called GPT-OSS 120 billion parameter model. Right now, they have trained on BF16, and the weight is 240 GB. What should I do? This is the problem we have. The first thing is that we have 240 GB, and 80 GB on each H100. And I have to fit only in one GPU, not in multiple GPUs. So what can we do? I think the simple thought is that just compress it, but how should we compress it? That's another challenge.

Speaker 5: If we compress to FP8, then it will be around 120 GB, but our GPU H100 is still 80 GB. So what I think they did is that they compressed it further into MXFP4, and I think the size is around 65 GB, right? So this is something we can do to compress. And we are assuming that there is no loss in compressing a bigger model into a smaller size.

Speaker 5: Second thing in this slide: we have used Mistral 7B. 7 billion parameters is such a small model. So if multiplied by two bytes, the weight of it is around 14.5 GB, which can easily fit into an H100 or even an A40. So next, what we can do instead, like Mistral 7B, instead of keeping it at FP16, we can apply different techniques like INT8, INT4, or NF4. Basically, we use an algorithm and believe that there is no quality loss, but somehow you also have to mathematically prove that by doing some kind of testing on external benchmarks to see whether it is working or not. And this comes under post-training quantization. One can also do this during fine-tuning; one can also do this kind of quantization, which comes under quantization-aware training.

Speaker 5: Let's move to our next problem. We have huge matrices. Just imagine a $1000 \times 1000$ dimension matrix $A$ and another matrix $1000 \times 1000$. If you multiply these two matrices, the number of operations will be $1000^3$. And this is a problem in terms of computing. We want our matrix multiplications to be fast, and it should save memory. So what should we do? We have a giant matrix.

Speaker 5: Let's take this one: $4096 \times 4096$. What should we do to solve our problem of speeding up things and memory for $4096 \times 4096$? First thing is that we can just break the block vertically; it does not matter how you choose it. So let's say we have 4096 columns, we will break it into a group of 128 columns each vertically. So we will get 32 blocks if we divide 4096 by 128.

Speaker 0: Then what will happen by doing this thing? If we just divide this one vertically, then we can use multiple GPUs to speed up the process. So this kind of thing is called multi-head attention.

Speaker 0: So what else can we do? We have a big matrix. As I have mentioned, our main problem is sizing. So what we can do is that instead of having all those 32 vertical blocks, we will throw away 31 blocks, and we will assume that one block is sufficient enough that all the queries can handle those blocks. Our loss will be almost negligible, and we come up with this algorithm, which is called Multi-Query Attention.

Speaker 0: As we can see right now, we are at two spectrums. One is Multi-Head Attention where we split it into 32 blocks and use different GPUs or do parallel processing. And at the same time, we are throwing away 31 blocks and calling this Multi-Query Attention. So at both extremes, now we should come up with a middle ground. Something where instead of throwing away all 31, maybe we can group some of the blocks together, so that we can assume similar blocks will attend to similar kinds of queries. This kind of technique comes under Grouped-Query Attention, which is very popular right now. Even in Mistral or in other models, this Grouped-Query Attention works.

Speaker 0: Right now, we have understood: we have a big matrix, we can divide it the way we want and do some mathematical calculation to prove that loss is almost negligible. So what else can we do after this Grouped-Query Attention? See, we have big matrices: one is Key and one is Value. Let's compress that matrix into a latent vector and then come up with some algorithm to reconstruct from the latent vector to our original matrix. So this kind of strategy comes under Multi-Head Latent Attention. But again, it has some problems with RoPE, because RoPE is position-dependent, whereas this is position-independent. So one needs to also include some index for keys so that one can map it.

Speaker 0: But again, the main problem is: why are we multiplying all those big matrices? Because that's how this attention mechanism works, that each token will pay attention to every token. So how about we don't pay attention to all the previous tokens, and only pay attention to the important tokens which are important for us? This kind of field is evolving. So this concept is used in DeepSeek Sparse Attention.

Speaker 0: Next one is FlashAttention. So in FlashAttention, currently almost everyone uses FlashAttention. But way back in 2022 or 2023, that's how it worked: Query and Key matrices were in HBM. It loads them into Tensor Cores, does some calculations, and then writes them back to HBM, and this process goes on multiple times. In FlashAttention, what they did is that instead of multiplying the whole matrices, they divided the bigger matrices into small tiles and only put those small tiles to SRAM, so that it can process multiplication fast and just keep track of some three variables to dynamically calculate online Softmax.

Speaker 0: So this is just mathematics. If we have Multi-Head Attention, say if it is 540 KB, then it depends upon how much grouping we want. If instead of 32 KV heads we only want to use 8 KV heads, we can get a compression of 4 times. And for Multi-Head Latent Attention, this formula depends on model to model, how many layers your model has. In the original DeepSeek paper, I don't remember the exact dimension, but they used a latent vector with 512 as dimension and 64 for RoPE index, and then they showed that it is 56x more compressed than Multi-Head Attention.

Speaker 0: So this is the trade-off diagram. Here, we have not talked about Linear Attention or Mamba. The main problem is just all this matrix multiplication. Right now everyone is using attention. Suppose in the future if we don't want to use attention, and rather than generating tokens sequentially just use diffusion models where we can generate everything simultaneously, all these algorithms will change also. But here they have two more: Linear Attention and Mamba. According to this slide, if we are not compressing anything, MHA is just parallelizing the process, so the quality is good. And then Grouped-Query Attention, which almost every model is using, along with GQA and DSA-like approaches. This provides good quality and high throughput in the attention mechanism.

Speaker 0: Uh throughput is okay. And for grouped-query attention, it depends upon your use case also. Though, though quality is almost similar to multi-head attention, but use case also matters a lot. Multi-query attention is just one extreme there. I don't know why, but we are just assuming that we only need one block and all the queries will attend to that smaller, a smaller block. So quality is not that great for, for, for MQA, and this multi-head variant attention. So if you have tried some this DeepSeek models, so I think they are doing great job. Yeah, in, in quality wise. Besides that, sliding windows. So all these are some techniques, which yeah, all these are some techniques like I just slide the windows, all those things. And instead of you're, instead of multiply everything, so linear attention is just saying get summarize everything first, uh, and then look up into it. And then Mamba, this is just a state space models. Yeah, okay.

Speaker 2: All good. Thank you, Tanmay. So for the model like optimizations, we also have like two notebooks here. So there will be I have to go to this. Okay. So for the quantization, uh like the demo, uh this is is this already run? No, let me just run this. Okay. So we are loading the model,

Speaker 1: Which is like a Mistral-7B also.

Speaker 2: So this one is like with the FP16 baseline, wait, did it run? Okay. So it's a two millisecond then this ran. Okay.

Speaker 1: So so ah this time it's fetching that model with the FP16 precision, it's gonna take time.

Speaker 2: Okay, yeah, it's because it's downloading the weights from the Hugging Face.

Speaker 1: Ah yes.

Speaker 2: So Colab is like running online because it needs to make the network call through the Hugging Face. And like it fetching, I don't not like, but it's taking time to download.

Speaker 1: Okay.

Speaker 2: Um okay, okay. So here we see like the memory sizes, like 15 GB around approximately with the FP16 precision. We are trying to do the 2-bit compression as talked about with the INT8, let that download. Okay. So we do see like your memory size is now like 7.5 GB. What that means is now you have a more more memory for your KV to basically grow. That means you can either serve higher context limit or you can serve the higher concurrent users there. If you do the like INT4 basic, you are doing the 4-bit compression. So that with the 4-bit compression, it would be more lower. It would be, I think, around 3 to 4 GB, you have 4.5 GB. And yeah, so this is it. So this is just a basic plot of like. So these are the theoretical numbers. We are not doing the like any throughput test here, but usually you would say, like a memory increases. You would also like a bit of higher throughputs from some of the benchmarks that we started. We saw like the INT8 compression, it does have like a lower throughputs. Okay.

Speaker 2: And then there is like a demo on the like the attention mechanisms. So for the attention, okay, I have to run this. Okay. So it is run. Why why does it say no GPU detected?

Speaker 1: I should say the GPU should be detected.

Speaker 2: Okay.

Speaker 1: But this is surprising.

Speaker 2: I guess it's not like able to detect the GPU for some reason. Uh, we do have like a GPU here. Okay, never mind, yes. But that like basic idea here was more like, as you try to move towards like compressing the computation, like by using different attention mechanisms, like moving from the multi-head to the grouped-query attention. And then to the MQA, you would start seeing some optimizations. Um I think yesterday Tanmay, you were doing some bench marking. Um I wanted to correct this part. Um so it wasn't like 56x. It was for MQA. Basically the demo had had a mistake of like a computation where it did not multiply the number of layers. Ah ah so apologies for that. So this MQA is like a 14 to 18x savings were in comparison to like your multi-head attention.

Speaker 2: So now that we have understanding of the pain points, the foundations, the one side of the optimizations, which is the model optimizations. We want to talk about what can you do on the like, the serving side. So the first thing is, we saw like when you perform like a simple decode step, you are pulling it. You are basically pulling the model weight, and then you are recomputing the key and the value vectors for all the previous tokens, even though you already computed the those vectors for the tokens. So there is definitely like a lot of compute wastage. And if you kind of analyze the time complexity of it, it would come out of be $O(N^2)$. And the way we resolve that is like a classic trade off against the memory. You can maintain a memory of those vectors against the tokens, and you can reference that memory. So that memory was called as like KV cache. And like the flow looks something like this. And then based on this KV cache, there were like four optimizations that were really possible.

Speaker 2: The first one is about the PagedAttention. So what's the different? What's the problem today? So when you send like multiple requests as the input to the GPU, these requests are in a batch, every request is allocated like a continuous memory storage. Let's say of, I'm just taking an example, like let's say 2 KB. However, like your request needed only, let's say, 1 KB. So there is like a 50% of that memory fragmentation, and this fragmentation basically leads to the memory wastage. That means there was a space in the memory where you could have served more request, but you could not because you were looking for that contiguous block of the memory. So an inspiration to here was being taken from, like how the OS works, like you maintain a logical memory, and you basically have a physical memory. So in the logical memory, it would still feel like that the KV vector for, like every token is like a contiguous, but it will be mapping toward different physical address. So that really helped like saving a lot of memory. And it was only possible because you they consider like memory as a set of blocks, and you would be dynamically allocating those blocks as the request need as the like new tokens comes in, and they need that kind of memory.

Speaker 2: The another lever is like when you are sending multiple requests in the batch, GPU is like taking those requests, but it does not accept the new batch, unless all the request in that batch gets completed. So the diagram looks more like a PagedAttention. But here it is more about like when is GPU available to take the next batch. So there is a time period where GPU is like sitting really idle. And you want to like resolve for that. And for that, like the idea was like, okay, let's do that continuous batching. So the continuous batching also really helped like throughputs. Now you can serve more requests, pretty quickly, keep making sure like GPU it's always like occupied, and it's not like sitting idle. So you are saving on that compute.

Speaker 2: The third is the like, like prefix caching. So you remember like the KV cache helps you save the computation for a single request across the tokens, but what about like you have the same tokens across multiple requests? How do you basically save against that? So the prefix caching was was introduced,

Speaker 1: Where the vLLM exactly counters that.

Speaker 2: And then the third is like we talked about, fourth actually. So we talked about quantizing the model, but you could also you can also like quantize the the KV weights. That means now you, you need like a lesser space for your key and the value vectors. That means you can serve more key and the value vectors in the memory. And that means like you can serve more tokens. That means you can serve more context, context limit. And that means like you can serve more model quality. And all of this is like already present in the vLLM, you don't really need to reinvent that wheel. And you can like deploy this vLLM in production, and you could see that basically growth.

Speaker 2: So next, we have like a benchmark that we did. So this benchmark was let me see if I have that here, the demo docs. So doing this benchmark takes like around 1 hour because you have to continuously stop and like restart that vLLM servers and you have to load the models and all. So it does take a lot of time and doing that testing. But I can like really tell you here what we are doing. So we have kept the model as same like the Mistral-7B. And then we have like the set of input questions that we are sending, consider them as the prompts.

Speaker 2: then we have couple of helper functions here like checking the server is up or not. the server is the vLLM server, then there helper functions to get the vLLM metrics. ah and i will talk about like what those metrics are. then there are like a lot of of the benchmarks and all. and then you have to measure that KV usage and know these are the like helper functions.

Speaker 2: so the baseline very simple like we have hugging face baseline. this is a raw like sending the text to the LLM. getting back the response. we see some results here. we saw like hugging face has a throughput of like around 51 tokens per second time to first token was like 54, and then the inter token latency was 19.

Speaker 2: this was all run under H100 um and then we start like a very default vLLM server. so by default, vLLM provides you the paged attention continuous batching and the kv caching. so three things are present by default. and when you try to compare those benchmark, you see your throughput is like almost 50x. you are able to serve more tokens per second, then your time to the first token that also rises. and then your the inter-token latency kind of goes down? and then your KV cache usage and the versus context increases for sure.

Speaker 2: now when you apply the prefix caching to it. so with the prefix caching, you see like your throughput increases more, your TTFT decreases your intertoken latency is approximately same. and then the your kv cache usage, what is the usage? it's kind of going down. the versus the context. it's not going down. it's approximately same. I think this is also approximately same. it's like not that um big of a deal when you apply the like, kv quantization on top of it.

Speaker 2: so so you see like a throughput is like almost similar. your time to first token is similar. your token latency is similar. but then your KV usage actually goes down. this is because like you have quantized your key value space, and then there is a concept of speculative decoding that, that he will talk about. so when you try to benchmark those, so you also see like there is a uh like a bit of like the less KV usage, although like the the results are approximately same.

Speaker 1: 然后三十的时候呢，还没有还是到少呢。

Speaker 2: so yeah, i mean overall, like these other like the metrics across probably, i should zoom out okay that's all zoom out now.

Speaker 1: 对对，

Speaker 2: great. so yeah, this is the like vLLM benchmarks. um it's a production default. by the way, we will also share that decision tree uh when we try to talk about like the other engines.

Speaker 1: 嗯，

Speaker 2: so yeah. so we should talk about like, what are some of the other inference optimizations we can do on top of it and what what some of the other solutions that came out. so i would like to again invite Thami. he is going to talk about like some of these optimizations. i'm sorry, i'm so sorry, i didn't enable the slides. what was the okay? great which one the speculativedecoding?

Speaker 0: yeah. thank you, Harpal yeah. so so all this are like speculative decoding, all these are the so so what we say, uh, different flavors of same kind of soda. so this this technique comes under decoding accelerators.

Speaker 0: so first one. so we're only talking about this speculative decoding, but there are other variants like self speculative eagle Medusa like that's one eagle algorithm. personally i don't think speculative decoding works, because main problem is alignment.

Speaker 0: okay, so let's to start with what is a speculative decoding. main problem is that in transformer architecture, all these tokens are generated sequentially one by one by one. how about just use a smaller model and let a smaller model to generate maybe let's say, four or five tokens. and this teacher model or we can say, according to our world, dcup algorithm can say, referee so referee, we decide how many are tokens it accept. and this loop keeps on going on.

Speaker 0: and our assumption is that there are certain domain where this kind of things will work, like maybe in maybe in coding, where almost there is no creativity, each code or syntax is almost similar. so maybe it can help it. but based on personal testing, i didn't find this speculative decoding useful at all, but other techniques like uh self speculative decoding, where teacher model also have one head auxiliary head, and it will do similar kind of things what this base model or small model is doing it.

Speaker 0: but then does eagle came eagle 1 2 3. i don't know how many versions are, but it is just saying that instead of generating tokens, let's train a small model inside, train a small model and just take features from one of main models layer. so that instead of generating token, it will generate these features. so eagle is is better compared to this other kind of technologies and then another one is Medusa, which is just saying that just generate all those tokens in parallel.

Speaker 0: okay. so here, so here in this slide, yeah okay. now we come to uh now we will come to this one prefix caching. so i don't know whether people are using this one static prefix caching or not, but things is that. main problem with prefix caching is that, sometimes we type and make a small kind of mistake. and this standard static prefix caching is basically, it takes a prompt do some hashing. and then next time when user ask similar kind of question, it will try to match the hash. so if hash is equal then it will, instead of recomputing all those K and V it will just take it from this storage.

Speaker 0: but you know that sometimes we make a mistake or maybe we can just change a word or letter something like that then we have a very higher cash miss hit rate. so that's why this one Radix Tree. so Radix Tree is becoming very popular and also because of agent. so i think almost everyone is doing agent, and most of the computation is going during test time inference kind of thing where we keep on asking same kind of questions.

Speaker 0: and for example, you are an expert software engineer multiply by 200 times. this kind of loop keeps on going inside this agentic kind kind of things where it is necessary to store similar kind of things in a Radix Tree. so Radix Tree is just a advanced version of this prefix tree, where we will just collapse a node if it does not have any branch, and for this kind of work will keep on repeating same thing. this Radix Tree helps a lot and SGLang use this kind of algorithm for prefix caching.

Speaker 0: okay, yeah, then there is another thing when is TensorRT-LLM. this is very confusing when i first started, i was all i was just confused for this TensorRT-LLM. so yeah, so TensorRT is just standard SDK kind of thing. TensorRT-LLM is just an inference engine, just like vLLM SGLang. but problem is that it is related to Nvidia. they optimize each and every layer. and every problem, as i mentioned in our algorithm, they just break everything and optimize everything at hardware level also.

Speaker 0: so for this workshop, we also did some benchmarking like which is best. so our setup was something similar. so we did two kind of testing. first one is without agency testing, where we use ShareGPT dataset and just asked those questions using vLLM and SGLang. okay, yeah, ok.

Speaker 2: and let me just okay great.

Speaker 0: okay. so for this workshop, we use H100 and about first testing we take questions from ShareGPT and put it into vLLM and SGLang, and we found. actually, there's no statistical difference between which one is better. so both have almost similar kind. so both are fulfilling similar kind of request per second TTFT and latency.

Speaker 0: so but only difference we have seen during agentic branch. so what we did was that we ask that similar kind of question that you are the best software engineer in the world, just solve the problem of traffic congestion in this city kind of thing. then we put this into LLM, LLM generate some output. then we did another round two also. so once this LLM generates this output, then in round two, we have specially mentioned that provide review the proposal and give ratings from 1 to 10. so this are two prompts.

Speaker 0: this loop keeps on repeating it. um what we found is that for this kind of um workflow where everything is standard, all those prompts and context engineering comes into the picture. if we do proper this agentic branching, then i think this SGLang is 3 to 4 times better. but again, this depends upon the different setup.

Speaker 0: Maybe if you do it, you may get different results. Okay, yeah. So I think did we upload it? Yeah, GitHub. Okay.

Speaker 2: Yeah. So the video is also in the Drive, it's the same link as the slides. So a quick summary here. On a standard API workload throughput, you would see like vLLM and SGLang would be the same. So if you have like a standard workload, definitely go with vLLM, it's the production choice anyways. But what was also saying, when you try to make it like agentic workloads, that is where like your SGLang really shines, and it kind of provides you all the benefits. So yeah, keep vLLM as a default. But if you have agentic workloads, probably try to move towards SGLang, and if you're not happy with vLLM.

Speaker 1: But okay.

Speaker 2: I'll let me okay. And then there is like a comparison that is done at the 120B like for the GPT-OSS 120B. This is a benchmark that was prepared by Clarifai. So there is like a blog link here. Okay, yeah. So they did the similar benchmark, and they included like TensorRT-LLM. Definitely, you can always go through these benchmarks and try to understand which basically suits your use case. As we mentioned, TensorRT-LLM tries to optimize the hardware side as well, having the peak hardware performance.

Speaker 2: Okay, yeah. And then in terms of when you want to pick your engines once you figure out between vLLM, SGLang, and TensorRT-LLM. There are some new engines that are popping up, NVIDIA Dynamo for short. They are also for the agentic session routing. Hugging Face is always there, it's a simple server. Then there is an engine that was recently proposed by Stanford, they are for multimodal. So definitely you could explore those.

When you try to give a quick summary, we start with the baseline, try to find what model could fit our use cases. You could pick DeepSeek. You could pick—don't pick Mistral 7B, I mean it's not good, but yeah. You pick your model and you want to have a smaller memory, and you want to try to fit that bigger model into smaller memory so that you could save cost on the GPU costs. So you can do all those quantizations, then you can apply all those serving optimizations by using the right serving engine under the hood. So that can really provide you that throughput that you really want.

Speaker 2: Now something that you can do after going back home, because we cannot actually go over all the material here, is definitely reading about some of the source information, like different attention mechanisms, different engines, and try to just read the different benchmarks which are present online as well.

And then there are a lot of in-depth guides for the next phases of it, which is like learning about some KV eviction strategies. The world is moving towards having a separate KV cache engineering domain, so you want to understand what's going on in there. So KV cache, KV eviction, cache compressions, hybrid memories. There are a lot of solutions that are happening around there. So always try to stick to those foundations and the first principles, and try to see which solution basically solves what problem and whether you actually need that problem to be solved for your use case.

Then there is distributed LLM inference, which is a different pain point altogether. You would probably need like a two-hour workshop there as well to go over all the internals and do all the hands-on.

Speaker 2: Yeah. And this is something we are trying to propose for the AI session in New York, which is to dive deeper into the advanced sections of LLM inference. So this workshop was more for the beginner and the intermediate level. So in this form, we do have feedback as well plus also the interest. If you think we need certain improvements on certain sections, definitely give that feedback as well. And if you want to see this workshop in New York, definitely feel free to enroll your interest. Oh, how I should push it.

Speaker 1: But let me just check. Um.

Speaker 2: Good. URL works, right? And not that QR code. Okay, probably I forgot to link those two together, redirect. Okay, good. Yes, if you can give that feedback, that will be fine. And yeah, I think we would like to wrap this workshop there. And I'm sure like a lot of you would be having a lot of questions. So we can take all those offline, we can meet and we can talk about those questions.

Speaker 0: Questions, yeah.

Speaker 2: Thanks everyone, thanks for joining. I think it was really meaningful.

Speaker 0: Yeah, thanks.