[Netter]: Agents can do three things. They can access your files. They can access the internet. And then now they can write custom code and execute it. You should really only let an agent do two of those three things. If you can access your files and you can write custom code, you don't want internet access because that's one vulnerability, right? If you have access to internet and your file system, you should know the full scope of what that agent's capable of doing. Otherwise, malware can get injected or something that can happen. And so that's a lot of what we've been thinking about is like, you know, how do we both enable this because it's clearly the future, but then also, you know, what are these enforcement points that we can start to like protect?

[Host]: All right, welcome to the Lean Space podcast in the Chroma Studio. Welcome um to all the guests here. Uh we're back with our guest host Vivu. Welcome. Good to have you back. Uh and our friends uh Netter and uh Kyle from Nvidia. Welcome.

[Netter]: Yeah, thanks for having us. I'm one of the engineering leaders and AR architects of Dynamo.

[Kyle]: Yeah, you're the developers, developers, developers guy at NVIDIA, open source, agent marketing, brev, and like dev tools and stuff in the focus.

[Host]: One of my favorite memories uh for NAD, like you always do like marketing stunts and like while you were brev you like had this surfboard that you like went down to GTC with and like Nvidia apparently liked it so much that they bought you like what was that like?

[Netter]: Yeah. Um, yeah, we we um our logo was a shaka. We we uh we were always just kind of like trying to keep true to who we were. I think you know so much stuff startups you're like trying to pretend that you're a bigger more mature company than you are and it was actually Evan Conrad uh from SF Compute who was just like you guys are guest amazing yeah um he was just like guys you're two dudes in a room why are you pretending that you're not U and so then we were like okay let's make the logo of Shaka we brought surfboards to our booth to GTC and the energy was great some palm trees too. They actually poked out over the walls so you could see the bread booth and no one else just from very far way.

My wife was uh at the time fiance she was in medical school and she came to help us cuz it was like a big moment for us. And so we we bought this Cricut. It's like a vine like a vinyl uh printer cuz like how else are we going to label the surfboard? So um we got a surfboard. We got a cricket and it was just like fine-tuning for enterprises or something like that that we put on the surfboard and it's 1:00 a.m. the day before we go to GTC. She's helping me put these like vinyl stickers on and she goes, "You son of a... If you pull this off, you son of a bitch." And so, uh, pretty much after the acquisition, I stitched that within the news of the acquisition. I sent it to our family group chat.

[Host]: One more thing before we move on to Kyle. Just have so many Jensen stories and I just love mining Jensen stories. Uh, my favorite so far is so. Uh, what is what is so?

[Netter]: So is actually I I think of all the lessons I've learned, that one's definitely my favorite. As you start to become a much larger organization, so is is essentially like what is the physics, right? The speed of light moves at a certain speed. So if flight's moving some slower, then you know something's in the way. So before trying to layer reality back in of like why can't this be delivered at some date? Let's just understand the physics. what is the theoretical limit to like uh how fast this can go and then start to tell me why cuz otherwise people will start telling you why something can't be done but actually I think any great leader's goal is just to create urgency.

[Vivu]: One thing I'm unclear about is can only Jensen use the so card...

[Netter]: Yeah. I think it's not so much about like get the [ __ ] out. It's like give me the root understanding, right? Like if you tell me something takes 3 weeks... The first principles. It's like what's the like why is it 3 weeks? What is the actual limit? If let's say you wanted to buy a new computer and someone told you it's going to be here in 5 days, what's the SOL? Well, like the SOL is like I could walk into a Best Buy and pick it up for you, right? So then anything that's like beyond that... we can kind of piece the reality back.

[Kyle]: I think at this point a lot of people are familiar with the term of inference. Inference at scale is becoming a lot more important. Uh we have these moments like you know open claw where you have these agents that take lots and lots of tokens but produce you know incredible results. Dynino sort of came about at NVIDIA because myself and a couple others were sort of talking about the these concepts that like, you know, you have inference engines like VLM, SLA, Tensor TLM. Um, and they have like one single copy. But when you're actually serving things at scale, you can't just scale up that replica because you end up with like performance problems. Dynamo is this data center scale inference engine that sits on top of the frameworks like VLM sang and tensor HLM and just makes things go faster because you can leverage the economy of scale. maximizing your cache hits or you want to employ new techniques in inference like disagregation.

[Kyle]: There are three things that determine whether or not something can be done with inference: cost, quality, latency. Historically models would be hosted with a single inference engine and that inference engine would sort of ping pong between two phases. There's pre-fill and then using that KV cache to generate new tokens which is called decode. Prefill is right now computebound most of the time. Decode is usually memory bound. Someone uh ExoLabs did a really cool demo where for the DGX Spark, which has a lot more compute, you can do the compute hungry prefill on a DGX Spark and then do the decode on a Mac and so that's faster. Dynamo actually has like a a Kubernetes uh component in it called Grove that allows you to do do this like crazy scaling specialization.

[Kyle]: We see models like Kimmy or GPTOSS. Kimmy 2 comes out, right? Kimi has more experts but fewer attention heads. Attention scales with the number of heads. If you have 64 heads versus 32 heads you do half the work. Deepseek's MLA (multi headlen attention) decrease the burden that KV has on the model which allows you to grow like longer in context. For context, the total context length of Deepseek is 128,000 tokens... fits into 8 GB and previously context like the llama 405b context of a similar size was like 40 or 80 gigabytes. I wouldn't be surprised if we do see the ability to like break through to like 10 million, 20 million, 100 million context through an unhobler showing up.

[Netter]: Coding agents have been so much more effective than general purpose agents. And I think a large part of that is it just has access to the terminal. It can write code and then it can compile the code and if there are errors it can fix it. I found this person uh his name is Carlos at the company. He wrote an Outlook CLI. I asked it to go through all of my emails... give me a summary, highlight any escalations... It was magic.

[Kyle]: Yeah, it's kind of funny, right? Like we like computing began with a terminal with a shell, but we said that it's not empathetic to humans. So we built these nice user interfaces and then now we have LLMs navigating our user interfaces. And ironically, we're not empathetic to the machine anymore. Just give the LLM access to the shell.

[Kyle]: This is the year system as model, right? Where instead of having like a single model be a thing, you have a system of models and components that are working together to like emulate the blackbox model. Under the hood, it's like a billion different models. And that's a lot of complexity, right? With Dynamo and with other libraries on media, we're looking to help manage that comp. We actually for CES, we just released the model router... the model router decides when to send queries to which one. It's no longer this like either or. It's use the best of everything that's available to you.