Ziv Ilan: [music] Okay. Hope everyone are awake after lunch and nice to meet you all. I'm Ziv. I'm in the AI labs team in Nvidia based out of Paris and working with different frontier model builders across a lot of domains. Of course, diffusion is one of them and we'll hear about a couple of examples of work we do with them. Um, we have only 20 minutes. So, obviously it's kind of a mix between going deep and going very high level. Each of these topics will probably be a full day or full um, conference to cover. So, I'll try to cover everything I can within this time frame, but feel free to reach out afterwards either through LinkedIn or I'll stay here a few minutes afterwards.

Ziv Ilan: So, without further ado, diffusion models. I assume everyone are here knows about it. Anyone doesn't know what video gen image gen how they work on a high level, denoising? Perfect. Okay. The idea is that, of course, unlike auto regressive um, architectures LLM, the idea is that you have a lot of iterations to denoise the image or the video. Usually between 20 to 50 steps and um, we see, I think the last year, an influx of very good high quality models both for image generation, what with flux two, um, video generation, um, LTX two, one, um, Google with the nano banana and and the later generations and we do see a lot of more practical use cases for that. And the main challenge once we have some interesting use cases is how to make it actually usable, right? We know that it's cool to generate videos or to generate images, but now if we talk about a developer context or a enterprise context, this should be uh, fast, okay? We want it to be mature, we want it to be scalable and these are usually the challenges that are hard to solve as this ecosystem is is not as mature as the auto regressive LLM VLM ecosystem, okay?

Ziv Ilan: So, we try to borrow a lot of the concepts we see work very well for LLM and we gradually kind of distill them, if I'll steal this um, terminology, into the world of diffusion models, okay? We'll cover a few of the topics here, but again, it's every day we see more and more research, um, in this domain and I expect this world to be even more mature in the next AI engineer. Use case and enablement, real-time image, real-time video is obviously the the holy grail, okay? Imagine how many new use cases where it's uh, world models for for robotics, for um, you know, computer games, for content generation. It opens a lot of new avenues for companies and developers to use it. And the big challenge together is, of course, the latency, okay? It takes a lot of time to get a first image and then to obviously get a high quality, okay? If we talk about 1080p or 720p uh, content out there. And to bridge this gap, I'll talk about three concepts. Of course, it's not um, covering all the the ways you can optimize your video gen image gen models, but I'll touch on quantization, caching, and distillation. It's not necessarily the the order you'll deploy it yourself, okay? Usually you'll start with distillation, then do some quantization, then some caching, but I started from the simple to the more complex, okay? Simple is usually quantization, okay? For those of you tried it in LLMs, concepts are quite similar. Okay, then we'll talk about caching and distillation.

Ziv Ilan: When we talk about quantization, we have two approaches, okay? Post-training quantization and quantization-aware training. Um, I'd say in many cases, of course, we do want to use the more simple approach like PTQ, but we know that at least for that to maintain the image quality, the video quality, it's a little bit more complex with the diffusion models, okay? Uh, we also know that um, this type of models are more attention heavy, which means that the impact of doing quantization is not as impactful as the LLMs VLMs, but it is still quite a low-hanging fruit when we talk about taking advantage of the more advanced um, features of Blackwall, for example, and and more modern uh, compute. In this example, just a work we did with Black Forest Labs on flux two, you can see that using usually dynamic quantization, okay? You don't want to use we can use static, which means that we'll compute all the range of all the different parameters up front, deploy it and use this static range for the quantization. In this case, we use dynamic approach, okay? Which means that some of the range will be computed on the fly, okay? Again, to make sure that the distribution is in line with the different uh, data distribution that you'll probably want to use when running these models.

Ziv Ilan: It's something that you can either do it yourself, okay? We recently released a good example in our TRT LLM visual gen repository, open source. You can start using it and see how it goes. What we also try to to do to again help the community to adopt it is also to help our partners to do quant- pre-quantized checkpoints. So, you can just go to Hugging Face, load the the quantized checkpoint and start using it, okay? If you don't need to fine-tune or to do some lower adapters afterwards, it's something again, it's quite handy and you can already see the impact. Of course, when we talk about quantization, the impact is both on the memory, okay? It will require less memory, which means you can run it on lower-end GPUs, whether it's consumer GPUs or lower-end data center GPUs, but um, also something that will help you in the performance, okay? So, this is one part of the toolkit. Again, a whole world sitting behind it to make sure that it's something that is effective. Just today I've seen one of the latest research coming from How Lab um, about attention FP4, which again, as I mentioned, attention is quite heavy for this kind of models. So, we do try to follow up with the latest research and make sure it's accessible for your you as a community.

Ziv Ilan: Okay, when we are talking about the second stage, caching, okay? KV cache is something that, you know, anyone that worked a little bit about in with LLMs, with auto regressive models, it's it's something that everyone talk about how to um, to use it efficiently, how to offload it, etc., etc. Again, it's a whole world. With the characteristics of diffusion models, it's not the same way, right? We don't we don't generate a token every time. So, it's harder to use this kind of techniques when we talk about denoising steps or, you know, getting again, making sure that we use the computation we had before in the way we'll generate future images or future videos. There are some T-cache is one example. Again, it's not um, a very strong example, but it's a good example to understand the concept, okay? While we are doing denoising steps, right? We talked about 20 to 50 steps. There are areas between the denoising steps that are pretty much the same, okay? So, we don't necessarily need to recompute them. What T-cache is doing is, okay, if if there's there was a minimal change or very small change between the denoising steps, it compares it and it understands, okay, now I don't need to um, to recompute for the next denoising step, okay? So, it's more general, okay? It does it for the entire pixel space or latent space, okay?

Ziv Ilan: More modern techniques of caching will do it in a more chunk-based, okay? Imagine that, I don't know, now we are in the classroom here. Most of you audience are sitting, staring at the the screen, so nothing much changes, but I try to be a little bit dynamic, so I'm uh, you know, you still wake up and follow me and which means that this chunk of the video doesn't necessarily need to you guys don't need to recompute. I do need to, okay? So, we'll isolate just this chunk and recalculate that, okay? Of course, you can define the threshold and and this is something that actually makes a lot of impact. We provided here some good examples of the expected boost you can get from using this. Um, but make sure that you try it, of course, and you maintain the quality, okay? Caching is something that if you don't do the right way, can have quite a significant impact on the quality of the image, okay? And as content creators, again, world models, etc., it's something you want to make sure that you maintain while you get the boost, okay? Um, so that's caching. Again, I encourage you to read more about different techniques. This is something that is already available in the TRT LLM visual gen I mentioned. Just a flag you enable and you set up the threshold. Uh, you can experiment with it, but also it's available, you know, in VLLM Omni as GLN diffusion and other serving libraries.

Ziv Ilan: Distillation, okay? And this goes to the you don't necessarily need 50 steps. Distillation is something we've seen, again, um, I I'd say probably the big bang for distillation was during the Deep Seek first release, how they managed to distill from a very big model to much smaller models and get, um, you know, I would say acceptable quality, but with a much with a much smaller model. In diffusion, the goal is not to get to a smaller model, okay? You'll still have the same number of parameters. This is more about step distillation, okay? Training the model, the student model, to generate as good quality images or videos, but by using much less steps. Okay? Instead of 50 steps, going to four steps, eight steps, in some cases one one shot. Okay? And maintaining the quality. Okay? And this is the big challenge. And imagine if you are able to reduce this significant number of steps, but maintaining the quality, it's something that can give you 10x, 200x improvement in performance. And if we go back to the real-time generation, this is something today it's probably the only way that can get us there in good quality. Okay? Um there is The next one, I think there's some demo we did in the last GTC conference a couple of weeks ago in in San Jose uh with two different um distillation techniques. Um and we got to a real-time generation. Okay? And this is something again that everyone are looking for, all the AI labs, and I'm sure also the bigger players, because this is um this means that we can actually get to again streaming, something that will open a lot of new use cases. Okay?

Ziv Ilan: So, how do we get it? Okay? We are When we talk about distillation, we always have a teacher model and a student model. Currently, we have two main approaches when we talk about distillation. Okay? One is trajectory based, which means we'll try the student try to teach the student how to follow the trajectory of the denoising steps as the teacher is doing. Okay? And the second is distribution based, which means we'll only look at the output distribution. Okay? We want the student to get to the same point at the end, but we'll let the student understand how to get there. Okay? And not by following the exact trajectory. Okay? The more common, I would say, and better quality technique these days is distribution based. And we also see a lot of ways that can be combined. These techniques can be combined. Um in the last FastGen video release, they actually managed to do kind of an hybrid approach that maintain the quality, but also got to a more stable um um training. The challenge and why I kept it to the last is that distillation is usually it's a post-training technique. Okay? Which means that you let's say if you do want it to work with your data, it's something you'll need to use some data for that technique, and and you want it to converge in a good way, right? Cuz otherwise, it will just um you know, garbage in, garbage out. Okay? So, it will require more compute, it will require more time, also more proficiency. Again, as it's an exploratory, still or research-driven domain, there's a lot of different techniques out there, and we expect more to come, but we are starting to see more mature techniques coming. And some very good examples shown in again in with the latest open-source um models. Of course, um closed-source model builders are also using this approach.

Ziv Ilan: So, FastGen is something that came out of our NV research group. Okay? It's an open-source repository. You can go There's a lot of different techniques there. Okay? It's not a distillation technique or method, but the idea is that because it's so complex when we talk about, you know, large models. Okay? A lot of the new video diffusion models are 20, 30, 40 B parameters, and we expect it again to get to hundreds of billions of parameters. It requires post-training, requires scale sharding across different GPUs. So, to manage all of this, we came with FastGen as a way to structure this process for you and enable you to focus only on the quality and, of course, you know, fine-tuning the exact recipe that you want to use. Like you can see, there's an optional training data here. If you're not using um well, you can always use open-source data, and it will work up to a point, right? And and we're actually happy about the results there. Um but if you want it to work for or if your use case have very specific data distribution, then we recommend you to use your own data for the fine-tuning. Um some of the results quoted here, okay? Um you know, the speedup, it's actually something we got not just The speedup doesn't come only in time, it's also in using much smaller uh much um less compute, I would say, to get to real-time. Okay? We got again in in GTC, as I mentioned, we got to one GPU of Blackwell B200 to generate near real-time video or real-time again, depends on the quality of the output. So, it does something that we highly recommend you to look into if you want to get to this point. Okay?

Ziv Ilan: We do expect again, a lot of the other autoregressive techniques to come and gradually um be relevant for the video video generation and image generation. We also see a lot of new model builders working kind of a transfusion or autoregressive diffusion approach. Okay? So, you use a diffusion to generate a um um a frame, sorry, but then it generates frame after frame in a autoregressive manner. So, again, we expect a lot more of these techniques to get into this domain, but it's still a lot of research-driven. So, make sure again, this is one very good example you can take a look at. And I think the best value about it is all of it is incremental. Okay? You can use this plus this plus this. You don't necessarily need to decide, okay, I'm doing only distillation or only quantization or only context parallelism, or again, there's a lot of different techniques out there, and they're all incremental. Okay? So, you can start with quantization, as I mentioned, which is the easier approach. If it's good enough for you, stay there. If not, okay, let's move to now multi-GPU, maybe do some context parallelism, maybe add some caching techniques. Okay? And then last and the most impactful, that's a distillation. And again, hope to see a lot of you trying it and getting into the real-time performance now. Try it yourself. Okay? All of it are open-source resources that you can use. We have added support for the open-source models as well, whether it's the one family, Flux 2 family, LTX 2 family, and other ongoing. So, hopefully, you'll be able We will be able to see also you guys contributing to this um and making video diffusion as good as we see with LLM VLM.

Ziv Ilan: Um I think I'm almost at time. So, if there is maybe one, two questions, happy to try and answer. If not, um we can let you 1 minute of breathing. [applause]

Audience: On average, what would you say is like are the requirements for you to fine-tune this model? Because access to GB200s are are not that easy right now. And in terms of data set, how big are the data sets that you've seen work well with some of these models?

Ziv Ilan: Okay, so the question was about the compute needed for that, and then from the data set needed for that, just for everyone to hear. Um what's good about distillation is that you don't need um GB200, right? You can do it with Hoppers, you can do it with H200, H100, B200, you know, B300. So, it's not necessarily that you need very big compute as you do for pre-training, uh but you still need to compute. Okay? So, it's not something you just, you know, take your um I don't know, just one instance and start doing it. Of course, it depends on on the size of the model, right? If your model is small, you have video generation models that are very small, 2 B, 4 B parameters. So, this will require obviously much less compute. On the data front, I think it's very important to make sure that one, you know how to evaluate. Okay? So, you can understand what's different if I just use just a general-purpose data set versus your specific data requires for your use case. And in such cases, we have seen differences. So, for the more general demos, we don't do use any special data set, and it works well. But again, if it's something that, I don't know, protein generation or something around that, that it will require, you know, something more specific. I'm at time, I think, but yeah, until they kick me out. Anyone other question there? Okay, we can afterwards, I think. Thanks, everyone. [music]