主持人: Hello. Good morning. How's everybody doing?

主持人: Whoa. Okay, that's the energy I'm asking for.

主持人: No way. You came back. Well, I also know some people just came here as their first day. So, welcome back everybody. And for people who are new, welcome to AI Engineering Miami day two.

主持人: Okay, some quick questions. Who learned something awesome yesterday that they can't wait to try very soon?

主持人: Oo, I see hands.

主持人: Okay, who made some new LinkedIn connections?

主持人: Oh, I see hands.

主持人: Who made more than five LinkedIn connections? Um, okay. I'll take note of that.

主持人: I hope you get replaced by AI.

主持人: Who's that person?

主持人: All right. And um, who got drunk last night? Don't be shy. Show me your hands. Who got wasted? Never mind. I'm surprised you made it back. Anyways,

主持人: Wow. Well, because I'm so excited about the agenda for today. So, no matter how much you drank last night, we have a great lineup of talks today. So, ranging from G2I, our organizer, all the way to Cursor. So, a lot of great talks. Actually, I'm super excited about one talk in particular. It's superfuristic, but I'm not going to spoil it for you. So, stay tight. And at the end of the day, I'm going to ask you to guess which one was the one that I'm most excited about. So just keep tuning in and find out all the talks that we have organized for you today. Okay. So I have a question for you all. Who is a worldclass engineer? Yes. Yes. That's the confidence that I'm looking for. Every single one of you actually sitting in this room is a world-class engineer. And also the person sitting next to you, even the people who don't like other people are also world-class engineers. So I want you to really take this moment to learn more from all the talks today. Really try to be that world-class engineer that you truly are and also talk to other people and make connections so that together we can all build a better future with AI. So that is the goal for the conference for today and I'm just so grateful that every single one of you is here and you are making connections and making that possible for us.

主持人: I forgot to ask one question. Who's loving the MC's this year?

主持人: Good, good, good.

主持人: Who's loving the MC's?

主持人: Thank you all.

主持人: Yeah. Well, we're going to keep the energy for our first speaker of today, David House. Iman, what do you think about David House?

Iman: I'm I'm really jealous of his jacket.

主持人: You'll see. You'll see.

主持人: Well, so David will be talking to us representing G2I, which is the organizer for the conference. He's going to talk about how we're transforming programming mindsets. So he's going to give us some case studies in agentic coding adoption. So relevant to every single one of us. And I can't wait to introduce David and let him show off his jacket. So give it up for David.

David House: Good morning everyone. Hi. So I am a software engineering manager but my background is actually in mental health counseling. Um I've was trying to figure out how to marry these skills together and recently I saw a unique opportunity. I've been hearing about all of these people adopting coding agents but there's also a lot of anxiety, a lot of fear, a lot of existential dread. So I wanted to learn more about that. Also this is my very first conference talk. So, please clap.

David House: So, there's a lot to learn. Um, how are people adopting these tools? And I I got into the weeds with different interviews, people on the team. I asked them questions about how they adopted the tools, what their experiences were, what their feelings were. And what I ended up learning about was a model of successful adoption. Our first case is Ava.

David House: Ava's first experience with AI was with chat GPT. She used chatbt for coming up with ideas for family events, kids activities, all kinds of things that she could review herself and trust the output. She was the only one who needed to sign off on it. But when it came to work, she held a very different standard. She didn't copy paste code. She didn't trust the output. And contrary to her work with AI with her family, there were other reviewers that she was very mindful of. She was very mindful of her reputation and she didn't want to ship slot.

David House: However, when she joined G2I, the project was very different. I'll tell you more about how the project was structured later but she joined a project where agent coding was the expectation from the beginning and the whole project was built around agent use. She learned that different techniques for working with agents could produce better output output that she could actually trust and she realized that a lot of things had to change about her development workflow. After months of using the prescribed workflow, she internalized those lessons enough to be able to create a sub agent for herself for a more specialized testing pattern.

David House: So when I watch YouTube videos about AI, I hear slop cannon and I hear skill issue. It's really easy to polarize between the two, right? But like if you're someone who's saying skill issue, maybe you were someone who said slop cannon. And may if you are someone who's saying slop cannon, maybe there's something for you to learn to get to the point where you can say skill issue. The point I'm trying to make is there's something in between those two poles. And that's what I'm really interested in. I think both are true, right? And when you learn how to work with AI, then you can start to differentiate how to avoid slop cannon and hopefully how not to be arrogant about it.

David House: Okay, I agonized over this slide for a long time, but now it's my favorite. Here's my claim for all of you is that for a beginner, an agentic framework should constrain their input. A beginner doesn't know when they're going to make mistakes. But for an expert, someone who has spent a lot of time with agents, someone who's internalized the development practices of how to work with them, an agent framework should actually amplify their input. So an agentic framework should shape the input of the human in addition to improving the agent's output. What I suspect is that these skills are not intuitive for any engineer. Agentic use was not part of any of our education. So we can't be expected to know it.

David House: Let's talk about what happened at G2I. So the case studies here are all from a very a specific project at G2I. Everyone was onboarded to the same agentic framework which relies very heavily on documentation and staged handoffs. It starts with a SL brief skill where the agent assumes a product manager persona interviews the user and the output of that conversation is a product brief. That product brief becomes input to a SAS spec bill where the agent assumes the role of a technical architect again interviews the user and the result of that is a technical specification that's provided as input to the slashcode or slashtestdriven development go where both documents ments act as very specific, very well-reasoned prompts with embedded judgment from the engineer. And then we close with a slash review skill to make sure that the agent covered all of the implementation as defined in the doc and then a draft PR skill to save you a little bit of time and spend a few extra tokens.

David House: But don't get distracted by the framework. I'm not here to try to tell you that this framework is better than any other. And I'd probably go as far as to say that most frameworks are pretty much doing the same job. The I break that job down into three things. Number one, revealing the hidden practice of agent coding. Number two, making the work of the agent reviewable, both its reasoning and its output. And number three, training the engineer in effective delegation with the agent. A beginner might throw one prompt to rewrite the whole codebase in Rust. But a more experienced engineer will know to take that step by step.

David House: Our second case is Lucy who has four years of engineer an engineering experience. Lucy got started with coding agents in a prior job to G2I. She had success writing a migration script from uh if any of you know about Vit and know about barrel files, you know exactly what how hard this is. The agent could automate the boring stuff and the agent could build a script that encoded a lot of judgment.

David House: Then Lucy's success on that project motivated her to give agents a try from the very beginning with a very complex personal project. But she ended up with a lot of duplicated code. Code that she didn't understand code that was too much to review and this was also a using technologies that she wasn't unfamiliar with. It was very challenging to work with and very demoralizing. When she joined G2I, the framework that we used told her something that felt obvious in hindsight, which is that you have to tell the agent to do the right thing. You have to tell the agent to write tests. You have to tell the agent to run the tests and make sure that they're passing. So she learned that the the practice of agentic programming is make is building the judgment you have about how to be an engineer into prompts and into the review mechanisms. She said that she does not take for granted that the agent will do what you want.

David House: So Lucy used the G2 framework for months, but she doesn't use the framework anymore. Now Lucy's method is much more interview focused. She'll spend a lot of time going back and forth with the agent, not necessarily using a strict prompt flow, not necessarily expecting a specific documentation result, but she trusts that during this interview, she's able to build in as much judgment as she possibly can. She's able to steer the agent during the interview, and she also doesn't necessarily trust herself to be a perfect provider of information. So the interview loop helps her remember the things that she had forgotten to say.

David House: Xcase Antoine. Antoine's been an engineer for 15 years. He's a startup founder and he is meticulous. Antoine is someone who has a reputation at G2I for being able to get anything done. No matter what it is, no matter how hard it is, it's something we have to do. Antoine can do it. So, we end up being very careful about where we direct his attention. Antoine's skill and attention to detail meant that when he was working with agents, he was very focused on the results. His early experiences with agents were very disappointing. He realized he needed to be vigilant and reviewing the code as output taught him that he needed to be very cautious. So he swung from early exposure with agents to hyper vigilance that diminished the value that he could get from coding agents.

David House: But when he joined G2I, the TDD skill significantly improved the output, which improved his trust, which allowed him to trust the agent to do more, which gave him better results, which improved his trust. And he's still skeptical, but he's tuned his skepticism to be more context specific. He now continues to use the agentic flow, but to hear him talk about it, he he can't necessarily describe how the way he does it now is different than the way that he did it prior. He talks about the the hundred nuances in ways that his judgment built over time working with agents, reviewing their output has built skills that are essential to how he writes code today.

David House: Our third case here is Dale. Dale's been an engineer for about four years. His first experience with AI was using chat GPT for coding. Um I was actually working with Dale at the time. We were both both working in a PHP codebase. Neither of us were very good at PHP. And I was like, "Hey Dale, you can use chat GPT to explain collections. You can use chatgpt to explain Laravel. Uh it will it will have infinite patience for you and infinite time. I had lots of patience, but I didn't have a lot of time. So he asked her for help with code uh understanding it but not actually writing it. Similar to Ava, Dale was very concerned about reputational risk. He didn't trust he also didn't trust that the agent had enough information. So with chat GPT he knew that the agent and intuitively he knew that the agent couldn't read the whole codebase. So he was unsure always every single time whether or not the output was misaligned based on the available context. He was also more junior at the time. This is a few years ago. So he didn't trust that he could actually validate the correctness of what was being output by Apache PT.

David House: When Dale joined G2I, he was a little hesitant. But it was clear that it was clear to him that he needed to figure this out in order to be successful. He had some early successes with agents, but went a little off the rails and ended up with a 10,000line PR. He figured, okay, well, the agent can do the whole epic, right? So, let's just have it do the whole epic and then learned very quickly that that's not necessarily a very good idea. He learned to narrow the scope of his delegation and make sure that he could that the agent wouldn't get away from him. Now, differently than I think Antoine, differently than Lucy, Dale has moved away from the framework, but his way of prompting today is very very similar to it. Instead of using the skills to generate a document that is then used as input, Dale writes the entire document himself. And he encodes the same guard rails and judgments that the framework made sure to include by default. He knows now intuitively to include them himself. And he can then use his judgment to be even more precise about the guidance. So then he spends 20 minutes in his own words writing this prompt, but he trusts the agent well enough and he trusts his own delegation skills well enough that when he hits enter, he's very happy to walk away and come back to a PR that he'll review as though it was a PR from his teammate.

David House: So I think the point I'm trying to make is that successful adoption is about internalization of these practices. The engineers when they started working with agents described feelings of disempowerment. They described feelings of sort of reacting to situations, not necessarily steering the output. But as the engineers learn more about agents, they learned how to be the ones in the driver's seat. You could say they learned agency.

David House: But what about junior engineers? When I hear talk people talk about junior engineers, I tend to hear one of two things. I hear if you find a uh a junior engineer who has already learned AI that you're set. It's fine. Thanks. But if you don't, then juniors are screwed. So again, it's another one of these polarities. Either you find someone who's already learned the skills or you don't know what to do. So I think we can do better than this.

David House: Andy is a member of our team, fresh out of college. His whole experience with AI was being told not to use it by his faculty, but his tutor tutors encouraged him to be a little more nuanced. His tutors had to coers him into using AI like a tutor because he was spending a lot of time at the at the tutoring center. He was open to having AI augment his understanding, but all throughout school, he wrote every single line of code that he submitted for his assignments. When he joined G2I, it was a total culture shock. Andy started to reexamine his hesitancy when he saw senior engineers with much more experience than he had finding success using agents. but he didn't necessarily trust himself to be a guide. And for good reason. He hadn't spent a lot of time in the field. He hadn't built a lot of software. He hadn't made the mistakes.

David House: But what he found made him much more successful was when he brought a senior engineer into the loop of his work. So if you remember we have this agentic framework at G2I where you use skills to do document handoffs. He would take the documents before he gets into any code, put just the documents up for PR and get reviews from senior members of the team. that allowed the senior engineers to encode their judgment into the documents which made the docs more successful, made his implementations more successful, but also served as a teaching opportunity and a mentorship opportunity.

David House: Also, Andy is looking for the next his next gig. So, let me know if you want to meet him, but don't wait because he's already interviewing. So, don't give up on juniors. Andy became a steady contributor in three months on the team despite being the only junior engineer on the team. Junior engineers still need mentorship. But agents expedite that because they can serve as tutors for the engineers when all the senior engineers are busy with their own implementation tasks and agents expedite that mentorship as well. Because the documents that the junior engineers create can serve as highlevel high leverage review artifacts where then the implementation can be much more successful building the confidence of the junior engineer. To quote one of his colleagues, quote, I was shocked to hear this was your first role out of school. You fit right in alongside folks with 10 years of experience. That's incredible.

David House: So many people are convinced that of software engineering doom. Maybe we're on a timeline where the profession of software engineering becomes irrelevant. Maybe that's true. But what if everyone trying to predict the future got it wrong? what we do today matters. I want us to look back on how we're approaching this work and how we're approaching our field from the standpoint of opening doors for everyone. I don't want to give up on the hesitant. I don't want to get give up on the conflicted. I don't want to give up on the people who haven't had that experience of being shown how to be successful with these tools. I'm not interested in leaving anyone behind. So you can try the framework if you want, but again don't be distracted by the framework. Use Beam Aad. Use RPI or the acronym that add more letters to RPI that I can't remember from yesterday. Krispy. Yes. Try a framework that slows you down so you can review the output.

主持人: Thank you, David.

主持人: So everybody wants to run the latest model, right? The fancier one with more parameters and that ends up being very costly and we have tremendous debt and that's latency debt. The next speaker is here to save us from that debt. Please welcome to the stage Sarah Chiang, head of DevX at Cerebras.

Sarah Chang: Sweet. Well, good morning everyone. I um me and my team, we actually just came from California, so it's it's a nice 6:30 a.m. talk. Um, but it's great to be here and uh I'm excited. All right, so recently I've been learning a wide variety of new vocabulary words. Coitating words like reticulating, schleing. Some of you know where this is going. And the reason I've been learning all of these words is not by choice, but because every time that I try to use claude code, I get these words for like 35 minutes. And I also get, you know, writing and waiting and thinking, but we already know those words. And if any of you guys have been playing around with LLMs or using them recently, you've probably had a pretty similar experience. Slow code generation. We write massive prompts. We submit massive diffs or honestly we hit enter and we go on our lunch break and we come back and pray it's done. It's frustrating. It kills the sparks and joys of AI coding. And today we're going to talk about why this is all happening.

Sarah Chang: So my name is Sarah Chang. I am the head of developer experience at Cerebrus and we are building the world's largest and fastest AI processor. And a big part of my job is introducing developers to fast coding for the very first time. And it's a very exciting job because oftentimes we get a lot of oo and a's and oh my god this exists. And outside of cerebrus I also create content on YouTube and Tik Tok and Twitter.

Sarah Chang: So before we dive into everything that is frustrating and going wrong, let's talk about all the amazing things that are going right. So in the last few years we have seen models get so much bigger. We have gone from models that are.3 billion parameters to over 1 trillion parameters. And a lot of these are open source models too. And as you can imagine this also means we're getting much smarter models. Context windows are also getting much bigger. If you so desire, you can shove millions of tokens worth of context into your model. And what's exciting is that we're actually seeing developers use this context.

Sarah Chang: So this is a study by Open Router and A16Z. Open Router is a unified API layer for LLMs. And they did a study where they sampled 100 trillion tokens worth of real world tokens across different use cases and capabilities. And what they saw is that input token links have gotten three times bigger, sorry, four times bigger in the last year alone. And on the flip side of that, output tokens have also gotten so much bigger. So the number of output tokens per prompt has also increased by 3x in the last year. And we can see this is because not only are our models outputting the final answer, but we're also outputting reasoning tokens. And so reasoning tokens are the intermediate tokens that a model will output when it's thinking step by step before producing the final answer. And we can see this coupled nicely with the rise of reasoning models as well. And so we've had a lot of incredible reasoning models over the last year. And so our models are getting smarter. Our models are getting bigger. All of this is wonderful news.

Sarah Chang: And this, my friends, is quite literally the best technology that humans have ever built. And so now let's look at this best technology in action. This is Opus 4.5 doing a very simple HTML snake game. Sorry, poor game. Awkward. You all wish you could go leave for a snack, but you can't. And this is if this was in Claude Code, this is where we'd be learning new vocabulary words. This goes on for minutes and I won't bore you, but I'm bored. You're probably bored. And as it turns out, if you've been playing with LLMs, you've probably had a very similar experience.

Sarah Chang: But unfortunately with all of this exciting innovation, there's a very unexciting side effect. And I like to call that side effect latency debt. So while we've been innovating on all these models, they've gotten bigger, they've gotten more intelligent, we put more tokens in, there's more tokens out, we have accumulated a hidden cost while optimizing our models faster than our infrastructure. And what I mean by that is that now if we look at all we've improved on so many fronts but now let's look at speed over the past two years we look at the Gemini families we look at the Claude families the Sonic models our speed has stayed the same we have always stayed between 50 to 150 tokens per second and now remember that even if your speed stays the same if the number of input tokens increases and the number of output tokens increases then the total total time that it's going to take for your prompt to complete is also going to increase. And when that increases, the developer experience decreases.

Sarah Chang: And so now, this is the exciting part. So now is when I get to tell you that all of this is about to change. In fact, it is changing. We've accumulated so much latency over the years. And now we have a solution. So about a month ago, we at Cerebras and OpenAI partnered up to release Codex Spark. It's the first state-of-the-art coding model that can generate code at,200 tokens per second. And to put that into perspective, that is 20 times faster than anything you're seeing here. And not only that, but Codeex Spark is just the first of many future models that we can expect to be a step function faster than anything that we've been frustrated with before. And so not only are we unlocking new capabilities and workflows and use cases, but this also fundamentally changes how we as developers can interact with the coding model. No longer do you hit enter and you leave, but now you can actually sit down with the code coding model. Treat it like a pair programmer. Verify the outputs, steer it. No longer do you have to generate machine levels of code. Develop technical debt. Now you can actually verify your code. You don't have to create bad code. And so now let's look at that same graph again. And let's look at codec spark. It's so much faster that we had to adjust the y-axis. And remember codec spark is just the first of many future models that we can expect to be a step function faster. And at this point it's a complete regime change. We are entering a regime of fast inference. And now we are entering a regime where the coding model can code faster than we humans can keep up with. Now we are the bottleneck.

Sarah Chang: And so why is this happening? We've been accumulating this latency debt for so many years. But what is suddenly happening right now that is making all this possible? So the exciting part is that it's because the entire inference stack is getting optimized all at once. And many of you in the audience are probably contributing to this in your day-to-day work. And so let's break it down. Let's take some moments. Let's start with the very bottom. The hardware. Hardware is the physical reason that speed is possible. It is the inference and training everything. The physical device that all of the workloads are on. With hardware, one of the biggest considerations is what I like to call the memory wall. So during inference, we have memory movement, but we have the movement of weights, KV cache values, etc. between the memory and the actual chip. On traditional hardware like the GPU, all of this memory is stored off chip. And so we're constantly moving all of these values between on and off the chip. And that memory movement is why the hardware is accounting for 50 to 80% of the total inference latency.

Sarah Chang: What we're now seeing newer approaches to is we're thinking about how can we reduce this memory movement as much as possible. This is an example by Cerebrris building our own version of the AI processor where instead of storing our memory off chip all of our weights and activations KV cache values we're storing them on chip in a distributed SRAMM so that every single core that's actually doing the computation has direct access to all of the values that it needs.

Sarah Chang: Even more exciting the commercialization of disagregated inference. This is why Nvidia bought Grock for $20 billion a few months ago. And this is also why Cerebrus and AWS just released a partnership to serve the Cerebrus wafers scale engine and the train AWS tranium together. What disagregated inference is so in traditional inference we have two steps. We have prefill and we have decode. In prefill is when we're actually processing all of those input tokens and we can do this in parallel. We're embedding them and adding them to our KV cache. So this step is computebound. Our second step, decode, is when we're actually generating every output token token by token. This is sequential and this is memory bound like what we were talking about before. And so we have two different steps with very fundamentally different requirements. And we've always been serving them on the same piece of hardware. Well, no more. And now we're splitting it up so that decode is happening on memory optimized hardware and prefill is happening on commute optimized hardware.

Sarah Chang: And now let's move up the stack. the model architecture. There's so many ways that we have adapted our models for the hardware that we have to run especially well on the hardware that we have. We think about model size, shape, the layers, the dimensions. And so isn't particularly new, but this is a great example of one way that we're doing that. So are mixture of experts is where instead of activating the entire model for every single token, we only activate a small subset of experts. And what this gives us is the intelligence of a much larger model for the compute cost of a much smaller model. Again, we're always thinking about memory, just like the hardware thing we're talking about before. But more recently, what we're seeing is that models researchers are building on top of so we're seeing a lot of exciting new advancements for how we can make our models fit even better on our hardware, especially for speed. And an example here is reap router weighted expert activation pruning. mouthful where instead of activating again the entire model, we're looking at a specific use case. We're seeing what experts are activated and we're just asking and removing and pruning the ones that we don't need.

Sarah Chang: And now the final layer of the step is inference optimization. Again, this is where a lot of you are probably working on. Philip gave a great talk from base 10 yesterday. This is where companies like B 10, modal, together, fireworks are all working. And one of the biggest considerations at this level is the KV cache reuse. So by storing and reusing previously computed token representations, we don't have to recomputee attention over the entire sequence every single time.

Sarah Chang: And so all of this are investments that we've been making and all of this are investments that we're slowly starting to see come to fruition. And it's incredible. We've had this latency debt that we've accumulated for years and now look at where we're at now. So now I want to show you what a state-of-the-art model looks like on a state-of-the-art AI optimized hardware. Again, it's the same two-player pool game, but this time it's running on this. It's running, this is the wafer scale engine 3, Cubris's own version of our hardware, and so it's not running on GPUs. And this is Codex Spark inside of Codex. And it's done. There it is.

Sarah Chang: And if you still don't believe me about light and sees real, you probably do, but just to be, you know, sure, let's look at the money. Let's look at where the biggest companies in the industry are putting their money over the last six months. Number one, Google versus Nvidia. Google releases the TPU four times faster than the Nvidia GPU. This is Nvidia's biggest customer investing in the infrastructure level. We have Google and Anthropic. Then what happens? Anthropic comes in and buys those TPUs, tens of billions of dollars worth of it. As I previously mentioned, Grock had been Nvidia bought Grock for $20 billion. This was Nvidia's biggest purchase ever. Grock had been building the LPU, their own version of a faster AI processor. And most recently, and most excitingly, not biased at all, OpenAI buying Cerebra's hardware. And this happened a few months ago, Codex Spark was the first model that they released in this partnership. And again, we can expect many future models to be the step function faster latency. And so we can see all this memory movement because what's exciting is that latency debt has been slowly accumulating. We saw the biggest players not only recognize latency debt but invest in latency debt. And now we're seeing that investment pay off.

Sarah Chang: And so now as we think about these new eras that we're entering and we're thinking about entering this new regime where the model codes faster than we can keep up with, I want us to take a little trip down memory lane at previous eras that we have. So in 1985, it's a history lesson now. We switched from custom hardware to CPUs. Why? Because our workload changed. We went from scientific simulations and mathematics to needing highly programmable creating highly programmable software apps. In 2003, our workload changed again. Now we have machine learning and graphics processing. So once again, our infrastructure changed. We went from CPUs to GPUs because now we cared about high levels of parallelization. Where are we now? We're in 2026 and once again our workload is changing. We are now we have multi- aent systems. We're building 500 plus agent swarms, human in the loop, agentic workflows and once again our requirements change. And what's exciting is that we've already seen people invest on this level. We've seen people invest in the infrastructure. And now right now is when all of that is paying off.

Sarah Chang: I cannot communicate enough how exciting it is to be a developer right now. not this year, this month, this day because we are right in the middle of a step change as we enter a new paradigm where the coding and developer experience is going to get so much better. And the thing that I want to emphasize is that it's not just about faster models, but it's about the fact that the developer experience is going to change. The way that we as developers interact with the models are going to change, and our ability to not no longer produce bad code is going to change.

Sarah Chang: So, thank you guys so much. My name is Sarah Chang. Um, if you have any credits, if you need any credits or have any questions, reach out to me. Um, my handle is Milks and Matcha across the board. Thank you.

主持人: All right, that was a great talk from Sarah. Okay, so our next speaker is Le. And then fun fact, I think Le might be the most likely to survive if there were a power outage here because he has all sorts of gadgets with him. He even has a flashlight. Uh he has a pointer, a laser pointer, and Keith's LLM models are deployed locally on his mobile. So even if there's no Wi-Fi out here, he will be able to make it out alive. So I'm so excited to hear how he's able to do that. So Le is going to tell us about ambient generative AI and how he deployed latent diffusion models on his mobile phone MPUs. So welcome on stage, Le

Le Kalinowski: Hello. Okay. Um I'm um I'm so surprised that people are still so much inspired by nature. Um actually uh we are just trying to mimic u leaves mimic birds uh uh mimic how they they just fly. So and uh yet we also try to mimic one of the most fundamental physical law uh which is randomness.

Le Kalinowski: My name is Le Kalinowski and I am a physicist. I work in Kstak. It's a Polish uh Polish company and today I want to I want to present present to you my uh research original research about the deployment of the the diffusion models on mobile. So um this is a part of the big bigger project uh which is related to hyper user experiences. um and I'm work in the research and development center that that means just try to innovate the the part of the user interfaces and try to uh push the boundaries um push the boundaries of the implementation of AI in that area.

Le Kalinowski: So randomness it's uh really well known and used across many of years in algorithmical computer science. So um one of the most um most famous um algorithm is the Monte Carlo method when you can just for example calculate the area of a of a circle or you can just calculate or simulate the chain reaction um in um in nuclear physics. Um and today um I just want to uh tell you uh why I just simply do this and um where I just get my inspiration to uh build that type of of project. So currently the biggest world labs in AI works with randomness. Uh what it means just try to optimize and build LLMs which are fast and cheap. So for example uh Google just introduced Turboquant and at the core of those algorithm there there is randomness. Um some kind of other examples uh methos cost like $120 spend per 1 million tokens and that's is really expensive model. This is why we are just looking for the fundamental laws and try to in be inspired by them to build um to build better, faster and more optimal models. Um but um and this is the reason why I just wanted to try to deploy the diffusion models on a mobile application in the really best possible optimal way.

Le Kalinowski: So I started to um I I started to think how I can do it because as you probably know from yesterday and Google just show it the diffusion models and the picture generation is really really difficult and computational heavy that means you need to have the huge GPU to generate takes a lot of time. So um uh this is couple of constraints that means you have have two on your mind the the termal uh profile of of the of the program and actually I just get an idea maybe I can use the um NPU neural processing unit on the smartphone to deploy there directly in a diffusion model and then try to uh run out um diffusion um diffusion inference uh there but it's not like enough. Uh the main idea was also to get rid of the full um full uh pipeline of the change text through the embeddings and uh building the uh you know full prompts through it. So I just you know throw it out and get an idea to use direct numerical value uh from an ambient sensor. Of course um here is a here's an example that means it can be like different sensor it can be accelerometer it can be something different but um but actually it works.

Le Kalinowski: So what I did, I just design um designed a full pipeline with the zero cloud um API calls to the you know outside um outside um not local LLMs um and I um build a full measurable system to to check it out and prove that the fusion models are possible on the local mobile devices. Um here's a here's a pipeline that means the most important thing is the latent update and as you probably can see here uh there's a stoastic pert perturbation scale and the randomness here is implemented uh one more thing I don't know if you just all know what the diffusion really is because uh the diffusion is made by uh adding noise to our data sets for example we have uh you know data set uh with a lot of the you know I don't know maybe dog pictures and we are just adding layers of noise there and then coding like in transformers um the model that means we are just minimizing a function and at then we have a huge noise of the latent uh space representation and to get an inference from other side we are just decoding that that noise that means decoding that randomness and for that we also use like in regular LLMs we also use the transformations that means we are decoding the the decoding the noising um uh the information there and we are doing it iteratively that means all of the informations requires really lot a lot of requires a lot of computer power um so um the the the logic is is quite simple

Le Kalinowski: That means the first ambient sensor then mapping after it conditioning that means the full pipeline to um to get those data u mapped to the latent space and the latent update function runtime session and and the full telemetry uh um to to measure all of the let's say results. Uh here's a a bit detailed view of it. Actually the full project was um full full uh application is designed in the um um in in the blocks that first block is an input and and the coding layer that means this is not that simple to um to get the raw data from from the sensor and map it to the latent space. So the the full first um full first mapping is the that we are getting the emmed light sensor then building hises on the top of that u build profit and the full normalization and then building the latent vector. So um I also build a full fallback that means to be sure that you know I'm just running out of memory there if some kind of thermal uh you know incidents rise and I just cut off the whole application because I don't know don't didn't want to lose my phone um and um the here is the API and the native um Android uh bridge which gives me up opportunity to deploy the model directly on the NPU And I also used an onnx runtime to do it.

Le Kalinowski: The uh the next um next part is the full diffusion line where Android native session manager and here is the model and this is the unit unit various auto um encoder that means the part where we are just encoding the let's say prompt then we have the unit d noiseis loop that means the core of the diffusion process with the 8, 16 and 24 steps and then we are just decoding it to the tensor and RGB um picture. Um at the end I also collect you know the the pictures itself the all of the artifacts and the meta data. Um and uh at the end we have the gallery and full telemetry.

Le Kalinowski: Uh okay. Um I think this is the uh um the funniest part uh of my today's presentation. Uh let me show you first the simulation on on on a on a computer just before I show you working application on my phone. Okay, this is how it looks like. It's not maybe um impressive at at that stage but uh give me a second. Okay. So I can change the the simulation of light and get an instant inference on the device here. So let me also show you the whole measurement whole measurement part that means we have full full um metadata captured ambient light smooth light stabilization temperature and everything which is um which is necessary. That means one of the most important parameter for me was um uh uh uh was u uh was the stabilization and the and the latency. Okay, let me uh let me show you right now how it works on the on the smartphone.

Le Kalinowski: Oh, here here's my private private phone. It's not not not sponsored by Samsung. It's my private one. Um actually let let me probably you can see my screen. I will switch uh switch off all of the internet connections to the flight mode. We'll be sure then um it's not the fake. Uh okay. And here is it. Maybe I would just write it up. So um I'm really glad that there's a lot of lights there because it's working. Um here we have a change in in the in in lux that means we are just capturing all of the data from from sensor and and this what you are just seeing right now is the full inference of the diffusion model deployed on the NPUs. It works really really fast. I know this is not a like a pixel wise representation because to do it really optimal um I need to cut off whole of the you know whole of the uh whole of the part uh of building real text to real text to to embeddings prompts and um what is really important as you probably can see it's it is extremely fast and it is really stable that means with this you can build some kind of other different applications and use it to I don't know try to build some kind of um different pictures or you can just even try to build an animation with this and this is not something which which will kill your phone at the at the end.

Le Kalinowski: Um okay let let me show you um briefly results of it. Uh so here here we have parameters I captured artifacts how many artifacts true this is the single experiment um u robust latency it's around 600 milliseconds that means really really fast the full inference it's it's cost not not that much uh it's really stable that means you can use that type of architecture for your um future projects but um I didn't want to finish on the let's say um weird RGB colors on the phone uh and I just forced that applications to build something um something uh nicer um and I uh decided to build just before conference I decided to build something more and give me a second I will only uh only just uh will find an application for So I decided to generate a heart um and try to force uh with some geometrical tricks and and constraints and not try to force the model to generate a real heart with the with the model. And actually that works. So um I'm I'm really happy that uh I was able to do it. uh um and I'm really happy that um that I can show you um the recent progress on the on the hyper user experiences uh today to you. Uh to wrap it up, uh I think you probably should think about the NDPUs and try to use it whatever possible to optimize and speed up your applications because all of the modern mobile phones already have it. But I think not everyone just use it in the in the modern development. We can for example can build an animation with this. Um and it's a it's a really good option. So I think um I just point you out that you can use it as an unconventional um application and it just uh it just waits to be discovered in your projects. And if you just want to learn more about Kalstack and our work because we just um in our incubator and R&D center we already just open source quite a lot of let's say artificial intelligence products project projects like for example evaluation benchmark for re react native applications um then grab a grab a code and take a look there. Thank you very much.

主持人: Great. Thank you, Le. Uh, quick note, uh, for our coming up speakers, please show up at the GTI booth at the expo hall so they prepare you to bring you to backstage. And, uh, I promised Le and David that I would, uh, compare their jackets. Well, he has a suit. Um, those of you who liked David's jacket, please make some noise.

主持人: Okay, those who like his,

主持人: I think we have a winner. Uh, all right. So, we have a break right now. Um, please, um, get your coffee, but be back here around 10:55 as we have the next talk starting then. Thank you.

主持人: Hey everyone, welcome back. I would like to take a moment to thank our sponsors uh without whose support this would not have been possible. Code Rabbit, Mintify, Cerebras, Sentry, Cloudflare, Tailscale, Modem, Ampify, Ozero, Google DeepMind, City Furniture, and Encrypt AI. Let's hear it for our sponsors, shall we?

主持人: Our next speaker, I asked him in the back, uh, what's the craziest thing you do? He said, I code all day, but I have a foot pedal. It's like a productivity hack apparently. A foot pedal when he presses it, it's it it starts the voice commands and he can he can talk. Turns on the speaker I suppose.

Tis: Oh, no. Whisper flow if you know what that is.

主持人: There you go. I don't know what that is.

Tis: It's like a voice detect.

主持人: You have to explain that uh during your talk then. Um so he does that. Also, he tries to take, did I say it? 210 grams of protein a day.

Tis: You're giving giving a home out.

主持人: All right. Uh it's my pleasure to welcome Tis on the stage and uh he he's going to talk about a lot of inference optimization. He's worked in Tesla autopilot before and now he has his own startup and uh his focus is inference optimization and how to make uh code gen more optimized. There you go. Thank you.

Tis: All right. One sec.

Tis: Hey, Dad. Dude, I swear I I practiced this before. Sorry. One sec.

主持人: Yeah, I'm not seeing anything from HDMI. Hey, hey, hey. What are we doing? I think so.

主持人: Okay, as Tis is setting up, uh here's a reminder also with the background music that there is a afterparty today. So, uh, if you didn't get a chance or you're held back from getting wasted last night, tonight is your chance. It's at 7:00 p.m. at Thrilled Show. And with that, I think Tis is ready.

Tis: Cool.

主持人: Take it away.

Tis: Yeah. Hey everyone. Uh, I'm Tis. I'm the founder of Morph. Uh, and today I'm going to talk to you about sub agents and specialized models.

Tis: So, you might remember this. Uh, this happened like last year where Trump went to the White House. Uh he was demoing a Model S from Elon and he said, "Wow, that's beautiful. Everything is computer." Uh I thought that was great because it's actually accurate. Like everything is actually computer, right? Like even the Windows inside a Model S is like a mini computer.

Tis: Um and so what I'm getting into now is um this is uh Andre Karpathy's sort of software 1.0, 2.0, 3.0 paradigm. Uh so if you remember software 1.0, this was humans writing code that programs computers. Software 2.0 0 was um weights programming neural nets which is sort of like a a flexible fixed function computer that you can do things like detecting stop lights. Uh software 3.0 was humans prompting program uh using prompts to program LLMs. Uh but a new primitive has appeared. Uh so now we have sub task and composer as a as a background um search tool and then cognition with Swed. Uh so what I think is the next sort of step past software 3.0 is what I call software 3.5 which is sort of agents prompting other agents. Um you still have the human at the top prompting that main agent but you sort of have some people call this agent swarms. I think that's a two vibe cody of a term. Um so yeah and so essentially what a sub agent is is super simple. The simplest definition is just using a separate context window for a task. Uh this has been proven to be useful coding agent performance.

Tis: Um so yeah so it's sort of this third brand new primitive. I think the first two were system prompts and tools. Uh the question becomes should sub agents be specialized models or should they be general models? And I think the right answer is that it depends. Uh but the interesting part of this talk is when it's actually specialized models. So first the the cost of what of using a frontier model. So frontier models are really good uh especially at coding and reasoning but they take a lot of compute and they're super super expensive. So they're not really great for every task.

Tis: Uh so why why should you even bother considering a specialized model? So from a macro perspective, we have this huge compute scarcity crisis right now where um like there's no H200 or B200 capacity across the board if you look at GCP or AWS. In fact, right now I even have agents in the background trying to find some capacity. Um but basically this is a macro pressure to like pressure people. This sort of macro pressure pushes us to specialized models because then you can use less compute to do a task to like the same accuracy. Um this is just a graph saying the same thing. Basically what you want to do is the minimum compute to do a task well and this sort of just arises from this this compute constraint.

Tis: And so our answer at Morph is uh we basically do specialized model and specialized inference. And so we think stuff like code search and context compression should move to specialized models. Uh whereas stuff that still needs frontier compute like coding, planning and reasoning should stay frontier.

Tis: Uh so one of the things we can do when we specialize a model is that we can do things that you wouldn't want you wouldn't do to a general model. For example, a frontier model lab right now they typically won't train on a parallel tool calling harness because you get this chronic forgetting problem where when once you do RL on this parallel tool calling harness, you start seeing regressions on uh sequential tool call performance. And so that's an example of a trade-off we could make that uh a lab might not want to make. And so that's what we do for our code search model. Our code model can do up to 12 parallel GPS, reads or list directories uh up to six turns. And it can do this very fast on a inference engine that we make that's designed for this workload which is super pre-fill heavy like up to 80k of token input and like around 200 tokens output.

Tis: Uh so another model we have is our our compactions model. It's basically a model that's trained to compact context at 33,000 tokens per second. Fast apply is our diff apply model. And yeah, so we build these models because I think coding agents spend most of their time um doing things that don't actually need frontier compute. And this is like code search, compaction, and applying code edits. And these happen thousands of times, but just aren't actually stuff that needs frontier compute.

Tis: Yeah. So how do we make them good? So good for us doesn't just mean like building something that gets like a good like F1 score for example. uh you have to train these to be a good sub agent, not just a good model. uh so what that means is uh for us is that when we do RL we train to sort of minimize this uh this randomness that you get when you introduce a specialized model. So like let's say you have opus at the top calling one of our like our code search model. One of the problems when you when you have this sort of setup with a separate model is that opus might have an assumption that your model is more capable than it might be or uh that it works in a different way than it actually does. So during RL we have to train to like minimize this randomness. Um so that that's like one dimension in which which you have to cover when you like make a specialized sub agent.

Tis: And so how do we make these fast? Um so one of the the most common ways people make models fast is called specive decoding. Speculative decoding is sort of it's sort of a thing first of all because you have this like massively parallel machine that's a GPU. Uh and you have this like super sequential architecture that is the LLM. Uh so at the top there you see traditional LM inference where you just have to generate one token at a time regenerate KV and all your all your stuff and then to generate the next token. Uh with spective decoding you sort of switch this uh so you have something some heristic to like make like a guess of like let's say six tokens and then you have your massively parallel machine the GPU just verify that those six tokens are correct and that time is linear to generate one token or verify those six tokens. those both take the same time. And so traditionally with spective decoding, you can have a you can have a smaller model like a very very tiny model like a 0.7B do these these like these predictions and then just have the big model that's like three trillion verify them. Uh but you don't actually have to have a small model. You can do it heristic base. You can do it whichever way you want. You just need some sort of way to get those guesses.

Tis: The another common method is disagregated prefill. So that's basically prefill is everything that happens in the time to first token. decode is everything that happens after that. Uh so a common thing we're starting to see happen now is u disagregating that those workloads into separate chips. Uh this is sort of a thing because Nvidia's NVLink is so good. Uh Nvidia is starting to do this now with Grock where Grock is going to be doing pre-fill uh and traditional Nvidia GPUs will be doing decode. It also works if you just use Nvidia for both. Um but yeah

Tis: Uh another way is if you uh write kernels. uh kernels are notoriously hard. It's a very fast changing field. One day it's QDSL, the next day it's emojo, the next day it's uh Qile and next day it's back to QDSL. And so uh this one is the hardest one out of all the three. But these are the the three main dimensions in which we we do our inference optimization. Uh for the really technical people out here, I think where the future is going is actually like disagregating further where you take attention operations out into machines. So you're going to might even have three or four separate chips each specialized for like prefill, decode, attention. Um, yeah, a lot of first principal thinkers would think this is really stupid though, like Jim Keller and these people probably hate this because intuitively you don't you don't need you shouldn't need like four separate chips to do this. You should be doing this all on one board, but reality plays out different in theory.

Tis: So yeah, getting back to the topic, uh, specialized models are a great way to actually have a product mode because everyone is sort of building the same thing, right? You have cursor with u multi-agent with background search agents and composer. You have devon parallel sessions uh and their sweet model. Uh and across the board you have cloud codeex winds surf and I guess grock. But uh a way to differentiate is being faster and cheaper. Uh and being faster and cheaper can let you do lots of things like you can go through more tokens. you can uh like like if you can run 10x more tokens, there's a lot of features that arise from that that I think a lot of people don't get. Uh and so we work with around like 40 plus uh companies and they're they're production agents. So I'm going to just go through some of the lessons I've had from working with these companies.

Tis: So the main one uh this used to be hotter of a take uh like a month ago and today it's like a super lukewarm take, but uh everything becomes a coding agent. And so that's like marketing agents, uh, like a customer support agent, the agent writes SQL. Marketing agents, they do things like call Apollo APIs and, uh, send emails. And basically across the board, we're seeing like like our initial customers were just coding agents like the Versel Vzeros of the world, the lovables. Um, and then you start seeing like this shift into different companies like Zo Computer, sort of like a personal assistant uh, that writes code and does stuff for you. And um yeah, basically everything is becoming coding agent. It's pretty obvious now. Uh and from everything becoming coding agent, you start to see a lot of tools you no longer need. You can start deleting stuff uh because you can just with code.

Tis: Uh code is essentially the feature that builds all features. Uh and so you're only bottlenecked by how how good the your your base model is at writing code. Uh, another interesting fact is that this is sort of consistent across almost all of our customers is that when you double speed, you roughly double conversion rates. Uh, given that you don't actually hurt accuracy. So, if you take the cohort of people that didn't hit like a a tra like a trace back or get a new error, you will double conversion rates within that cohort. So, if you can double speed without hurting accuracy, it's very, very valuable. And again, context link matters. That's why you should use sub agents.

Tis: Uh, yeah. Yeah. So I'll go into some good practices about building good sub agents. Now uh so if you're building a search sub agent, you should really just it's very simple. You just put natural language in like if you're doing code search, it would just be like where is authentication logic sub aent at the end would output file paths and line ranges. That way you keep the output tokens that that sub aent is generating very small. So this can be very fast. Uh and this leads to improvements across the board no matter which sub aent you use. For our search sub agent, we sort of see like a roughly 3% sweet bench pro improvement across the board, but it's it's a similar improvement if you use cloud code search sub agent or um any of them.

Tis: Uh so for task agents, which I think you should still be using a frontier model, not a specialized model, uh you should be sharing the prefix cache. So, uh, basically when your main your main model like Opus for example wants to, um, like spit out like three three three task agents, uh, those should all share the same prefix so you get the caching, it'll still be fast. Um, if anyone's looked into the cloud code leak, they they actually have this thing called team create where there's birectional communication. I think that's actually really cool as well. Um, so basically what happens is your main agent like Opus can do a team create. It'll make a folder with a slug. Uh it'll have a JSON for each agent and every time it writes to those it can send messages by writing to a JSON file for each agent. Um and each one of those task agents will be continuously pulling that JSON for updates. And uh basically you have this sort of birectional communication and there's no real protocol. This is sort of like a hacky way to do it. There's some other I forget who's trying to make like some protocol called ACP which is like a to standardize some RPC format to do this.

Tis: Uh I I I don't really care how it's implemented. I think the the the thing that's more interesting is this actual birectional communication between main agent and sub agent. Cool. Yeah. And so hopefully you got out of this talk that everything is models. Uh if you like working on inference optimization NRL, please email us.

主持人: All right, that was a great talk by Tis. And next up, I'm going to introduce Fidian Aentuity. I'm sorry, that's not his name. Well, so the next speaker is Rick Bllelock and he is a Floridaian who is the founder of Agentuity and he's gonna tell us a little bit more about coding agents and how they're eating the world. So welcome onto the stage Fidian aka Rick.

Rick Blelock: All right, let's do this here. Extended. I can't see the screen. Which one do you guys see? Cuz I don't know what I'm seeing.

主持人: You don't want to see my nose.

Rick Blelock: How about now?

主持人: Perfect. All right.

Rick Blelock: That's what I want to hear. Blow that up. All right. Welcome to Miami, everybody. And uh it's been a fun uh fun conference so far. our AI engineers, one of my favorite conferences of all time and uh super happy to be here. I'm glad you all are here, too. So, my name is Rick Blelock from Agent Tui and today we're going to walk through the last couple years in the evolution of AI agents. how right now in April 2026, it seems like coding agents are not just writing all the world's new software, but they're actually slowly starting to become that software as well. So, by the end of this, I hope that you have a clear understanding of where we're at and why a coding agent is a new fundamental to software.

Rick Blelock: So m much like you can tell where a software engineer at is at in their journey, you can tell where someone is at on their agent journey or even their level of psychosis, their agent psychosis. Um I'm afflicted by that. I'm sure many of you are as well. But anyway, you can tell where they're at by how they describe an agent. You'll hear things like, "Well, it's just a workflow, right? It's just a fancy RPA. It's just a few LLM calls. Ah, agents just a software loop, isn't it? Isn't that what it is? Isn't it just a newer chatbot that we were building those back in 2015? Isn't it just like a better chatbot? Or coding agents, you know, they're not really ready yet for software dev because it makes mistakes. So, you hear things like that all the time, right? So, you kind of can tell where people are at in this journey and it things have happened rapidly in the last two years, right? And so we're going to talk through that.

Rick Blelock: So let's jump back two, three years ago. We said we wanted agency out of agents. And then we built systems really obsessed with control. We were calling them agents, but most of them are really out they were just kind of prettier workflows or chatbot++ kind of thing. And you know, we tend to shoehorn our past experiences into everything. That's a human thing. We all do that. And so we're we're so used to solving problems a certain way with computers. Decades of solving problems with computers a certain way. By the way, that includes non-engineers, by the way. Um how non-engineers, just nontechnical people think about using computers. So it's really hard for us to start from first principles when it comes to software engineering and using computers to solve problems. But there, you know, a few years ago, there was a few people starting to poke around at that and try to figure out, you know, what what does agency mean and software and non-determinism.

Rick Blelock: So, if you remember, how many you remember autog4 launched? So, I don't know, maybe 50 AI years ago, something like that.

Rick Blelock: And so, um, it uh it became the number one trending GitHub repo. It did. It it be I don't know it had like 160 or 170,000 GitHub stars. It was the fastest growing repo GitHub repo of all time. And um I don't know if you remember some of them. There was spin-offs. There was Chaos GPT which was a spin-off. Everybody remember that? It was tasked with destroying humanity. I don't know if you remember that. But it was popular because it offered a practical glimpse into what like this kind of agentic future could be like. It was also popular because it was a spectacular failure from a user perspective. Um, it was stuck in loops. It burned tokens needlessly. Lots of bad hallucination as you can imagine with the models at the time. And uh, if if you really remember in hacker news, there was a dev that just replaced the loop, the feedback loop with static code and it got better results. I don't know if any of you remember that.

Rick Blelock: So, um, Aram from Tom's Hardware, he said, uh, it was too autonomous to be useful. Um, so, you know, like there's there's these like stamps, these glimpses and uh, snapshots in time where we see that and we go, "Oh, some people's brains get hardwired to that moment in time." Go, see, agents aren't good. They're not ready to be. But there was there was people there was a glimpse of hope, right? Somebody was trying stuff. Didn't work out. It's fine. But we were getting somewhere with it.

Rick Blelock: Then around that time, frameworks, you know, they started popping up that simulated agency. A lot of them were actually still brittle deterministic pipelines. The uh 2023 agent architecture were really just a lot of linear chains. Step one, step two, step three, a lot of determinism. And we were all trying to figure out what we wanted to do when we grew up with this, right? You know, there's Crew AI, Langchain, Autogen came out, NAN. I mean, there's a list of them. Some of some of them attempted some type of agency, but kind of like autogen because of the limits of the models, there were problems. So, we all fell into this trap of what customers wanted right now and not kind of like where the models were going to be in six months. And what they wanted, they just wanted a better chatbot. They wanted determinism. So we were using agents that were for non-determinism, but we were using them with all this determinism in there. Um, so that's what that's kind of when the frameworks tried to solve some of those problems.

Rick Blelock: And you know, the curious thing is the enterprise at the time the expectation was these projects, these prototypes were around six months, which I mean, if you've been in this AI engineer stuff for any, but six months is like an eternity. I mean that is probably about four AI years. I don't know. Um, so six months to like implement this thing with this framework and then all of a sudden the models get better and the of course the frameworks get better but then all of us start asking well wait a minute what do we even need these frameworks? I mean they're all kind of complicated anyway. Do we even need them now? And then even anthropics own advice this is back in December of 2024 which again doesn't seem that long ago. I'm going to read a quote. We recommend finding the simplest solution possible. This means this might mean not building a gentic systems at all. Well, that's interesting, right? So, just a couple years ago, h it's not really it's complicated all this like orchestration theater and all this stuff going on.

Rick Blelock: And um let me go back here real quick. So like you think about that orchestration theater. I don't know if if if if there's probably I'm sure several of you on here that have gone through that and the complexities of all of that that's happened and then all of that gets removed in the last six to nine months, right? Go from chatbot tool caller workflow orchestrator multi- aent theater. or you know remember cursor added sub aents and then Claude wasn't very good at using sub aents so they removed it and then all of a sudden Claude's good at using sub aents again and cursor adds it back um and the advice one day was don't use sub aents sub agents are bad and now the sub aents are good um and they are good that all that's changing all this turmoil and you know we're preaching agents need agency and then customers are like well this just is just RPA this This is just this isn't agentic.

Rick Blelock: And then during all that time, Cognition Labs demoed Devon. How many how many of remember the demo of Devon when it was demoed? Marketed as the first AI software engineer remember that interesting. And of course, you know, early days there was the hype cycle that maybe outpaced the product a little bit at the time. Personally, in my opinion, I thought it was kind of the right balance of a little bit of controversy and vision casting coupled with a decent product launch. That's what I thought. But fast forward to today, and it's honestly, it's one of the most capable agentic uh engineering products out there. And it's free from the nitty-gritty like messaging of workflows and frameworks. I kind of thought it was a breath of fresh air. I I I met Walden, one of the founders co-founders in Salesforce Park last year and I was I was telling him like I really want to put Devon more in our in our flow and I I know I want to integrate it more and I was thinking traditionally like frameworks and stuff like that like how do how do I stitch this together? How do I put it in this? And after talking to Walden it was like well just give Devon some scripts and some access to your tools and it'll just it just do that. So it it was, you know, free from that MCP theater and workflow theater. And it was the first time that I started thinking, oh, I should treat this coding agent for more than just code, but managing our demos. We did Friday demos um every it's it's on our YouTube. It's public demos. And Devon helped us. Devon literally would tell us what we would demo. I can't remember what I worked on yesterday. Um so it was the first time I was like, "Oh, well, there's something here." interesting um and I appreciate that about them.

Rick Blelock: So all this is going on and meanwhile we have customers and real businesses around the world saying what are the use cases for agents? What are they? Trying to understand how they actually fit inside their company. I mean I can't tell you how many times I've had conversations with business people like what are the agents? Is it this? Is it that? and kind of only devs kind of understood the complexity of the orchestration and the frameworks and it kind of muddied the waters and maybe made it look like a little bit of snake uh snake oil salesman kind of stuff to some of the business people um or getting accused like ah you guys are just crypto bros that are using AI now or whatever you know you kind of get a lot of that

Rick Blelock: um but then use openclaw as an example open claw stitch together enough things that it helped normies understand what an agent specifically agency can be in software. What Devon helped um with developers understanding an agent use case for our world, OpenClaw helped non-devs understand agent use cases for theirs. You know, as engineers, we like to bash OpenClaw, right? It's all these these scripts and these complexities. It's insecure. But meanwhile, you have non-technical people buying Mac minis. You can't buy a Mac Mini right now. I mean, I'm sure this was not on uh Mr. Cook's bingo card that he would sell out of Mac Minis. Um, and they're copy and pasting instructions into an app they've never heard called Terminal. And they've never used it. They don't know what they're doing. Why are they doing that? Because they understand the use case. they they're willing to suffer this weird pain because they understand the value they can get from it. And the models are at the point where, you know, it doesn't feel super brittle. So all of a sudden, autonomous agent makes sense to a large group of people that didn't get it before. You know, before I was like, autonomous agent, they're like, "What the heck are you talking about? This sounds like crap, right?" Um, and sitting on top of all of that, what we're talking about is a coding agent. That's what we're talking about as a coding agent.

Rick Blelock: So here we are 2026 and if I say a coding agent is a universal software primitive. A lot more of us understand that this is actually pretty true. You know code isn't just a developer arch um artifact. It's not just a developer artifact. It is a coding agent isn't just a harness for developers. It's not a coding agent can implement every other type of agent. And even a a normie who can't describe what's going on when he builds multiple agents that help him manage his business. A coding agent can build a chatbot. It can do rag. It can orchestrate other agents. It can automate its own workflow, not just yours. It can create a database, stuff in it, and use it to answer its own questions.

Rick Blelock: Try that sometime. Take the canonical Titanic data set and um have it put in a database. It'll create a model, stuff things in there, know how to um update it. You know, we attribute intelligence of animals um as if they if they can build complex or compound tools, they have high intelligence. And although I know some of you will want to nitpick on what intelligence means when it comes to a coding agent, we are there with coding agents. Coding agents will build their own tools for themselves if you have the right instructions for them.

Rick Blelock: So in 2026, convergence is I mean this is the inflection point for us. You know, sandboxes are giving agents a body. Model intelligence is now giving them the brain, right? we got these good protocols that we're starting to kind of figure out the nervous system. Whereas, you know, a year ago when I said agents need a different approach to infra, people were thinking, well, why does a framework need a different infra approach? I don't understand that. But now with coding agents now, they're like, oh yeah, it does. It does kind of make sense. We do kind of need something a little more purpose-built for a coding agent to run all the software to be all a lot of the software. Now it's a little more understandable.

Rick Blelock: The coding agents still need a lot of this. You know, at a aentuity, we had to adjust our thinking. A year ago, started a developer horizontal developer cloud platform, and now it's kind of more like an agent platform that's for agents to run to for them to run. Um, and all these things here that they need. Um, you know, it's a place to act, a place to persist, to live, a place to observe, to integrate, to manage. And there's still like gaps in our in what we can offer, right? I mean, the the boomer clouds, if I can call them that, are are trying to sell the infra from a decade or two ago, right? They really are. And they have this veneer of cutting edge, but when you get in, it's all still kind of hacked together. Honestly, kind of like our personal coding agent setups. It's all kind of like just, you know, stitched together. And ser services honestly still built for stateless web tech. and a coding agent, you're like, I'm try to shoehorn this into this. I mean, the story is always the same when I talk to people. It's like, oh, we started at serverless, try to deploy it, but guess what? My agent runs for five minutes and serverless times out in 30 seconds and then and invariably after like a bunch of experience experiments, you end up on EC2 and you're running like we were software in 2008 over and over and over and over and over again. That's what we're trying to do at agentuity is like, you know, make it so agents don't feel like they're running in uh 2008.

Rick Blelock: But with that in mind, put a pin in that. With that in mind, back to openclaw and use cases for normies. Normies don't understand or know all the things that I just mentioned about the infra about what's needed for this stuff to work. Yet still, Mac minis are being sold at Apple and they're copying pasting and they're getting outcome. And this is the mind shift that I think we have to make. The coding agents, they are a fundamental to software. Now, non-engineers already understand this. They they do. They they they maybe can't articulate it in all these crazy ways that we can, but I think they probably understand it faster than some engineers grasp it. You know, we're busy poking holes in the fact that coding agents make bad mistakes, crazy, stupid, idiotic mistakes.

Rick Blelock: Um, and you know, we're quick to point out agent psychosis, which I have, a lot of us have. Um, but then the non-engineers, they're using this stuff and they think when we when we're like sitting there complaining about, they're like, "All right, are you using the same thing I'm using?" Because I'm running my business on this is what they're thinking. You know, if you're at the if you're at a party and you're wait and this guy's talking about how he's gotten done all these things and you're just waiting for the pause and you're like, "Yeah, but coding agents, they make mistakes. You you're here. I'm here. You see the difference?" You know, um if if you're that if you're there, they're they're going to look at you and go, "I think you have psychosis. You have software engineering psychosis." That that's what's going to happen. I mean, just as an example, so there's um there's a man I know, he's in his mid60s, very successful entrepreneur. He built a manufacturing company that did a bunch of stuff for Toyota. And now he started this construction company in Texas. And he was paying tens of thousands of dollars to HubSpot a month for this company. And he's like, I'm just going to build it on my own. I'm going to use coding agents and I'm going to use coding agents to run the little bit of HubSpot that I use. And he did. He did it in about three months. Um, another example on Saturday I had up in Jupiter, Florida, I had um I had breakfast with this guy. He's 24 years old. He runs a wind window cleaning company and he runs his business with a coding agent, his marketing, his sales, his sales estimates. So, you know, if I told him, ah, coding agents make mistakes, he's like, yeah, but like I'm like way up here in productivity. What are you talking about? There's an exe a former uh CIO of a large financial firm that's in here today and he uses it uh for his board meetings. So my point is the foundational agent primitive is already a coding agent for normies. They maybe they don't realize it, maybe they can't articulate it. And what kind of agents do you think are going to be tasked with working on the org chart, the focused work of of things in the future? And I think it's coding agents. If you peer just in the near future, six months, 10 AI years, coding agents will be used to create and manage other agents to manage certain parts of the org charts work.

Rick Blelock: I mean, we're doing that with our competitive marketing. We have this this uh it's called GEM. It's this set of agents that manage we have about 50 competitors or frennemies we call them that we monitor. And um these agents are constantly watching. I won't tell you what they're watching, but they're watching all sorts of things, all sorts of signals. And um we we know like everything about where we are, where they are, the messaging. This company's probably about to pivot because they changed this messaging a lot in their site map, whatever. Um so it's managing a sizable part of our marketing stuff and it's and it will be used by non-engineers to manage their work and their organization. leaders will direct coding agents and it's not intuities by the way not in two. So next time you're debating what framework should we use for building an AI agent, how it should work, pick a coding agent. It's kind of like Ben Ben at the last talk yesterday. He he talked about those different versk and pi and open code SDK. Um try that out. Try using a coding agent instead. You know the good ones they have an SDK. They have server side handling of things. They have on demand sandboxes that you can pick up resume uh where where they left off and they have observability built in and that's also what we're trying to heavily do with agentity. Um you started as a developer platform kind of more like hey agent is a is actually a customer a persona of our platform. There's all this inbound interest from normies and people that want to run this stuff at scale. They don't know how to. Um, and it's all built on like coding agents, multiplayer coding agents that watch each other, agents that tell each other what to do. Um, so that's where we're going with this. So yes, Mark Andre and he said software ate the world and now I think coding agents are eating software. Thank you.

主持人: Okay, how's it going everyone? Having fun? Great. So, let's say I want to build my app. I have a lot of data and uh about transactions, places, and there are relationships between them and uh I want my agents or or my app to query this information. I have to store them in a database. I can dump them in a blob storage or or maybe SQL database. But is there a better way to store all of this information especially if there are relationships between the entities? And the answer is of course yes. The best way to capture these relationships is through graphs. And maybe the better way to store them in a database is to store them in a knowledge graph. And the next presenter is going to talk to us about how we can optimize our um context engineering by leveraging these knowledge graphs. Please welcome Nia on the station.

Nia Mlin: Thank you. Thank you everyone. Hey, hey, hey. All right. Yes. Good morning everyone. How are you doing?

Nia Mlin: Okay. Okay. I see some energy. I love it. I love it. Well, I'm going to wake you up. Okay. So, good morning everyone. My name is N Mlin and today we're going to get into effective context engineering techniques for artificial intelligence. So, as I like to do, by the way, um I'm a teacher in a past life. So, I like to get um lots of different ways that people can remember the concept that we're going to be talking about today. Not just sit here listening to a lecture, but actually conceptually understanding what's going on. And I like to make sure that people from the highest chief security officers, highest chief information officers, CTO's as well understand the concepts as well as maybe a junior developer or an intern. All right. So, um, let's get right into it. And I'm gonna start off with a question. All right. I'm gonna get us all engaged. So, how many of you all, and this is going to be a yes to pretty much everyone, but how many of you have shipped an agent to production? Lots of people, right? Lots of people. Okay. So, all right. Of those of you who've shipped an agent to production, how many of you have had that agent then do something in production that made you question your career choices? Question your career choices. Right. I shouldn't be an engineer. I'm so sorry. I brought down prod. Customers are crying right now. They are so angry. It's happened. It has happened and that's okay.

Nia Mlin: Right? So let's get into a scenario. Right? So this is actually real table stakes. This happens every day. Right? So a compliance team is going to run an agent pipeline, right? And that agent is going to retrieve separate chunks. So that chunk is going to be for example Jessica hypothetical person works at Apex Global, right? And then Apex is actually on the sanctions watch list. Okay, that's interesting. And then Jessica has actually requested a $25,000 credit increase. So all three of these chunks have been received, right? And all three are sitting there in our context window. So the agent YOLO is going to approve it. is going to approve this credit line increase. There seems to be nothing wrong with Jessica's request. She has excellent credit. She's a 850 credit score. There's no problems. We're just going to yolo and approve that. We're just going to vibe bank real quick. Y'all don't like vibe banking? Y'all don't want vibe banking? that agent is going to approve that request not because the agent itself or even the model is bad but because the connection between these these facts is actually never represented anywhere in vector space. Right? So it wasn't just the data, it wasn't the retrieval, but the model actually had to do a guess and actually it guessed wrong. Okay, it guessed wrong. So, nobody caught that until the lawsuit came.

Nia Mlin: Until the lawsuit came and your scene, your your skip level is now hitting you up asking you who wrote the agent, why did the agent answer this way? Why did the agent actually approve this request? And what are you going to say, right? Everyone is listening. You are sweating bullets over here. You are stressed. Okay. But this actually this tutorial does keep landing on my desk, right? And in the next 25 minutes, I'm going to show you all exactly what happens and the techniques that we can use to fix this. Prove it live. So like I had mentioned this this scenario always happens. You say you don't know and you end up fired. Can't stand it. Can't stand it. Need to pay these bills.

Nia Mlin: So what's happening here is that text similarity when we're when we're relying on vector search it finds documents that have a similar meaning right structural similarity on the other hand find entities that have similar connections so almost is actually building the second one and I want to give you this upfront so that you know exactly what I'm trying to argue here every retrieval pipeline that we're using right in the room we're using normal naive rag hybrid search ranking everything like that. It operates on one dimension. This text similarity piece. How close is this chunk in meaning to even the query itself and that work that can get you like 70% of the way, right? It could even get you 80% of the way. But there's a second dimension that almost no one is building upon. And that is that structural similarity piece, right? not do these two documents the same thing but actually are these entities connected in the same way. Right? So when we're thinking about a credit line increase for a corporate banking client then and then we're also thinking about this concept of everyone listed uh at the employer is listed on the federal sanctions watch list. That doesn't mean the same thing, right? So your vector search is actually never going to surface those things as being connected and and they're not going to look at each other. But the relationship, right, the relationships between Jessica, between her employer, and then those sanctions as well, that isn't a meaning problem. It's a connection problem. Right? So right now everyone um or right now nobody really is using an agent pipeline to see connections. And that's the vital dimension that we're missing here.

Nia Mlin: And so even research backs this up, right? We've all seen this like this is fine. It's going to be fine. We're all going to be fine, right? And MIT's NANDA initiative, you can feel free to take a screenshot of the QR code to read this report yourself. I always um like to tell people to read the white papers. Don't just listen to people who are up here speaking. But in their 2025 Genai Divide report, it found that 95% of AI pilots are delivering no measurable P&L impact. Right? 95%. That's a huge number, right? and we're building agents, we're scaling them into multi- aent systems, we're investing months of engineering time, and then the project is killed because it actually doesn't clear the bar for quality. It doesn't clear the bar for privacy. It doesn't clear the bar for security. And it doesn't clear the bar for transparency either. So the central gap that MIT had identified is that systems do not just retain feedback but the systems that are that are making sure that they don't retain that feedback are adapting to the context or they're not even improving over time and Gartner themselves as well is predicting that 40% of agentic AIA projects will be cancelled by 2027 right and the models here are not the problem the context is the problem.

Nia Mlin: So, better models don't fix fractured context. I want you to repeat that with me. Okay? Better models don't fix fractured context. Period. Right? And that's where all of this all has in common. Right? these these um these atentic systems are going to retrieve via your text similarity search and then the text similarity it actually has a blind spot right of the size of the building when it comes to actual relationships better models don't fix fractured context they just reason over these broken pieces

Nia Mlin: and let me show you what I mean right so on this example we have a human view of an right there's three ways to represent this apple on the left. So on the right actually we have a vectorzed view of an apple. Right? This is just an array of floatingoint numbers. It's a way that most of our embedding models they uh understand the textual reference of an apple. Right? This is what a vector search is going to operate on. It's not human readable. It's very opaque. And then in the middle we have the human view of an apple. You see an apple. Your brain processes all of the relationships implicitly instantly, right? You don't even think about it. You just know an apple is round. It's fruit. We're not talking about the company. And then on the left, we have a knowledge graph view of an apple, right? The apple connects to a tree. The tree connects to a body. The body has a fruit. It has a stem. Every property is actually a node, right? And every relationship is explicit. So a human can read this actually and understand exactly what the model knows about an apple. This is both human readable and machine readable as well.

Nia Mlin: And that's the thing, vectors capture meaning, right? They they know that things are close to fruit and red and food, but it's often a black box. You can't inspect it. You can't ask, okay, why does this model know that this apple is round? You can't audit it as well. A knowledge graph is going to capture that structure and capture that meaning and it's going to be much more transparent, more queryable, right? More auditable example that we needed before to then overcome AI's blackbox problem. And we need that knowledge to then be transparent.

Nia Mlin: And before I go deeper within this technique, I want to show you some of the numbers that convinced me that that this wasn't just a theory, right? So Jang at all he published uh and their entire team published an eunations magazine this March of 2026 and built com GPT it's a domain specific foundation model for telecom and they ran a proper abilation study on three GP question answering and so the base model they had the base model accuracy here right the base model accuracy had an accuracy of 37%. Sure. Right. And AI like we'll take it. Um but then they fine-tuned that model on domain data and then they got up to about 54%. Right? And then that's a significant improvement, right? But then that's still not going to work for production. So they then added both knowledge graphs and rag together, retrieval augmented generation together and got 91% accuracy rate. Right? So no, it's it's it's not just the graph. It's not just rag by themselves, but here it's both. Right? This is the critical point. Rag alone does not close this accuracy gap, right? It's the knowledge graph that was used to unlock that extra percentage of accuracy. And the graph wasn't just a nice to have. It's actually the difference between having a demo versus having a production system. So 37 to to 54 was the finetuning. 54 to 91 was with the graph plus rag right and that middle drum that's the structural retrieval dimension that I'm talking about and this is all again peer-reviewed in

Nia Mlin: so quickly I just want to jump into the past and I know this is a little controversial but I would like to argue that the past um is how we're normally doing context engineering so for those uh who no one no one's new to this term also uh shout Shout out to Dexy who coined this entire term um and and also spoke yesterday. But I specifically define context engineering as being able to systematically provide models with all the relevant information, the relevant tools and the relevant instructions that your agent needs in the correct format at the correct time. All right, to then accomplish a specific task. So, it's very different from uh from pumped engineering where you're just trying to say the right words to an LLM or an agent to get it to do like whatever you want, right? Context engineering centers on building dynamic systems that assemble complete and structured context for each LLM invocation. So, the shift in focus from those cleverly worded prompts to a comprehensive contextual design is why context engineering is now considered a critical skill for AI engineers.

Nia Mlin: And most engineers are using some combination of these various techniques here. Right? So there are several different techniques that practitioners are often using in order to get that contextual relevance in an agentic development. And in the example that I had mentioned before so that's including rag and hybrid search the classic approach. I'm not really going to go into all of these um but for those who are newer to the concept um uh naive rag is going to use a vector database of embedded documents to then perform a similarity search. Practically, there's pros and cons to that, but I won't even get into it. Um, and then we also have memory management where an agent interacts in a long session and across lots of different sessions and it accumulates lots of information. And I have that represented by Dory because you're now managing what you're going to forget. Um, and you can use sliding context and recency based memory of course as well to uh in order to drop history. But let's keep moving on. You can also use structuring and ordering of context. Um, enthropic for example gives guidelines on how to deal with long prompts. You put the most relevant stuff and the most critical stuff at the top of your prompt since models pay more attention to that at the beginning. And then additionally, we can use tools and function calling instead of trying to stop your context with all your raw data like I see most of you trying to do. We're actually going to give it um the instructions on how to uh call a tool, right? And really that's just a function for those of us who are engineers or we can offload those tasks to tools. So for example, instead of giving an LLM your large um like a a large table and asking a question about that table which would consume all of your context tokens, it might be better to give your LLM a database query tool for example. Um and then lastly for uh so I like to represent that as MacGyver for those of us who are a little bit older. We understand MacGyver always had a tool. Um but the method that I want to get into today into today is using knowledge graph augmented context. I had mentioned this concept of knowledge graphs before and I'm not going to dive too much into it because of time but knowledge graphs and I'll do a representation so you can see what it looks like. Knowledge graphs are an extremely powerful way to introduce structured relational knowledge into an agent's context. And I like to represent knowledge graph augmented context as Sherlock Holmes right because Sherlock Holmes is a perfect example. He's able to synthesize important information and then track down the clues, this tracing piece in order to solve a particular problem. Right? This is exactly what context engineering allows you to do when you integrate knowledge graphs into your AI stack.

Nia Mlin: So most people right now they have their data structured in uh in this example, right? Tables and rows it is the OG, right? You have to do a number of different complex joins if you are trying to understand how that data is related. But that then there's the there's the caveat. It's going to draw down your performance of your application. But if you structure all of that data, that same data as a knowledge graph, then the relationships and the connections and the groupings become much more clear. So now your agent can hop from one node of information to the next and be able to actually trace where it went and explain the answer. And if you want to learn more about how to like take your data and put it in knowledge graph, we have lots of courses for free and we're open source.

Nia Mlin: Um, so from an architectural standpoint, I just wanted to touch this as well before I get into the new concept. Um, from an architectural standpoint, having a knowledge layer in the AI stack allows you to one hold structure and meaning, two, create a uniform memory layer, right, and retrieval layer for all your agents, and then three link and unify all of those data sources and objects together. And then four while bridging this human and um uh machine understanding and then at the bottom as well you've got your existing data platform. So whether that be snowflake data bricks relational systems right on top of that is going to sit this knowledge layer which connects all of your structured and unstructured data in a consistent and semantically rich way. So from there you can have your geni apps and orchestration layers like langchain or semantic kernel or your agentic frameworks. they can then plug into and access this data um contextually.

Nia Mlin: But I want to remind people like this is not a concept that we have created like knowledge graphs have has existed for many many years. And this concept of knowledge graphs and adding this knowledge layer we think is just important for folk to understand that it can drastically increase the accuracy, the traceability and the auditability of your AI applications because many of us are just like settling with vibe uh vibe everythinging right vibe banking which doesn't make any sense right like why are we spending so much money to have inaccurate results like it's just not making any sense. So the future what I'm would be the future of what building out context and then having explainable andable AI agents looks like this concept. So has anyone read this paper which came out in December? Let me see some hands. Anyone read this paper? Um it's called uh the AI's trillion dollar opportunity context graphs. Okay. Like one or two one or two. All right. Well for those who are newer to the um to this concept, right? like in present conversations around building AI agents, it has shifted from using techniques from context engineering um like using naive rag to using graph rag to then using knowledge graph augmented context to now what I'm talking about which is using context graphs and it's not simply a change in terminology right the the idea of context graphs has actually existed way before this paper um um uh but um uh the the meaning behind it is actually really really helpful for when building out AI agents, right? So when building out agents, we need this idea of this decision trace, right? So that agents can be auditable and can be more trustworthy and can be more explainable.

Nia Mlin: And um there have also been and I have the uh the papers right here a number of different white papers and research around context graphs even prior to that seminal article that came out in December. Um, but we are shifting our understanding why agents make decisions because agents are no longer simply tools that people are using to give you a particular answer. Now, now agents are being built to make decisions about our everyday lives, right? Like this this this isn't a joke anymore. This isn't the demo. This isn't the fun um like like uh PO that we're building. Now we're building out agents that are making decisions about whether someone gets a loan, right? And that has impacts on people's lives and their financial uh prosperity going forward. Like it's it's uh making decisions about whether they can they can own a home or or what have you. It's making decisions in medicine as well, right? We're giving agents so much power and that's why the ability and the ne the necessity to audit the decisions that every agent is making is paramount especially with context engineering.

Nia Mlin: So the missing why. So when an agent recommends approving a even a $25,000 increase like we had mentioned for Jessica before, but even a $100,000 credit line increase, the question isn't just is this the right answer. We have to understand that if we're just asking that question, we are we have no assemblance of what memory looks like here, right? You're going to ask an agent about last week's conversation and that agent's going to be like, "What you talking about, homie?" Like, "I I don't even know who you are right now." They gonna say that, but you're also not going to have this audit trail, right? When something goes wrong, no one knows why your agent decided that. Y'all can try and see, okay, you know, I can I can try and reverse engineer to see what chatb or anthropic cla like said. However, that a trail is missing and that's the vital information, right? And then on top of that, there's no shared learning. You're going to deploy multiple agents and they can't share what they've learned across sessions.

Nia Mlin: So, what is this concept of a context graph? A context graph is a knowledge graph that's specifically designed to capture your decision traces, right? The full context, reasoning, and causal relationships behind every significant decision. So this differs from simply opening your agent up to an audit log for example. You might say oh I can just do that with an audit log. No, no, actually right that would have yes that would have the line by line transaction history but your agent now has this breath of knowledge from context right when with context graph it has this breath of knowledge from the things that were not written down right that could either be like the the conversation that's happening in the senior architect's mind right where they're making these trade-offs and these decisions when building out a either aentic system or software in general right it could live in the Slack messages that you're building out right Now, it can live in your email threads or even your Zoom meetings, right? All this connected data can show up in a graph that can show you all of that full context and the full why any single decision was made. And this is this is now going to include the causal chains and the decision traces within context graphs that can actually be queries and that can be traversed to ensure that your agents are more reliable, auditable, and trustworthy.

Nia Mlin: So I have the same financial uh services model here and we're going to go through a demo anyone work in I doubt people work in financial services but possibly nope exactly zero hands but that's okay we're still going to do it but in this data model I'll teach you all about financial services uh we have a graph um uh world called and we're going to call these entities right these circles will be entities um but it's really just people places and things right and we also have agents sorry events, um the things that happened like the what, right? And then lastly, we have the context and the why, the decisions, the policies, etc. That's the model. So, we can get into what this looks like in practice. Boom, boom, boom. If you want to um take a picture of this or actually look at the demo live, you can feel free to do that. Let me see where I put it. Nope. Where is? Ah, okay. Boom.

Nia Mlin: All right. So, I have a live one and I I have a um my other one, but I was I was messing around with the code trying to add some new features. You know, never to add new features last minute before you're going to do a demo. It's going to break. It's going to break. So, um so I have a live demo here. Um, and many of you are like we're all talking about context graphs and we want to see like how does this actually work in practice. So I'm going to run this query here. So we have an regular AI assistant for this financial services use case. We're going to ask ourselves like should we actually approve the $25,000 credit increase that we had for Jessica earlier.

Nia Mlin: And let's see this graph get built. I want to make sure that everyone can see it. Yeah, perfect. All right. Um, so we have a person who submitted a support ticket about a specific transaction. All right, we're we're seeing this graph get built. But that transaction is then going to trigger an alert, right? When flagging this entire account. That then alert is going to take take that all of that into cons into into context for this entire system which is going to cause a decision trace which you see on the right which is based on policy and then communicated back to the person who owns the account in the first place.

Nia Mlin: So think of this like an a an analyst at a bank right who's going to um use this agent to respond to a particular customer request. The analyst is going to have this customer requests come in. Should we increase this credit lo increase for Jessica Norris? Should we do it? Um, and the bank analyst can can say yolo by bank. Let me just give that to you, Jessica, because you're my girl, right? Absolutely not. Thank you for laughing at my jokes. So, if y'all don't want vibe banking, why are y'all doing this without decision traces? Like, y'all, it seems messy to me. like y'all are whing to me.

Nia Mlin: But so if we inspect this agent here, we can see that we've defined a number of different tools here. Let me come over. Um we can see that we've defined a number of tools. So we're doing this tool calling that we can see. I can't even see that, but maybe y'all can. Um but we have this um this agent which is interacting with our context graph. And then we have the system prompt, right? um which is going to have these specific tool calls which is going to fetch data or the context um that's necessary to make a specific recommendation and all of that is then rendered within this graph that you see in the center. So we'll see the causal chain of what led to a specific recommendation from the agent. And in this case uh the recommendation is to and we'll follow all the way down to see actually um instead of thinking that we can just yolo banking we can just let's see the let's see the response. All right. So I'll help you evaluate it. The recommendation is to reject this credit line increase. All right. So the recommendation is to reject the credit line increase based on my analysis of the context graph. I strongly recommend rejecting Jessica Norse's $25,000 credit line increase. And here is my detailed reasoning. Critical risk factors, there was a recent identified request already rejected um April 9th, couple days ago. And then there's actually significant fraud history that happened within um within Jessica Norris's account last year. So they had um a known fraud typology where the velocity check the number of transactions 14 transactions in 29 minutes happened. There was also a geographic anom um where that IP location was inconsistent with the accounts address. All these can be explained away by Jessica and if they are explained away then that context graph will then update based on that decision. Um but there are multiple compliance violations as well within this. So you can see the AI assistant and the agent specifically telling you the entire reasoning behind the decisions that it's making and you can see the exact decision trace on the right as well detailing what's happening with as the uh confidence score as well. So u feel free to try this on your own. Again it's context-graph demo.vercell.app if you want to try it on your own. And then

Nia Mlin: let's go back to the slides. Okay, so I'm gonna wrap this up. So how this all works, right? So if we want to look at how this agent is um is tracing the decision history and how it's using the context graph, first of all, it's going to then search for the customer and then for the context around that customer. Like as we can see, we saw the um the the circles, the nodes and edges, right? that produced um that transaction in the past and then we can see the previous fraud flag that we had seen before. We're going to then use hybrid search which combines vector and graph to then actually do the heavy lifting here and I just wanted to touch upon this hybrid search piece because it's actually really really powerful right so combining vector plus hybrid search is really powerful and specifically at Neoforj we use this called graph data science which allows you to run graph algorithms like centrality or page ranks for those who are um uh have used different algorithms before or graph algorithms before Um but here we're talking about graph embedding specifically like fast RP if someone has heard about fast RP which is a generated embedding uh for context of text for sorry for content of text. Um so um and then how we use the vectors and doing vector similarity search apologies that allows us to find the different policies and the decisions that are semantically similar. But what about the structure right the structure piece is what I had introduced right there are things like that's where the things like graph embeddings come in right which allow us to then look at the structure of the accounting relationship and how it interacts with transactions with other accounts as well with fraud patterns and these generate embeddings that we can use vector search functionality which are then familiar which we are familiar with to find the most relevant data right for our context.

Nia Mlin: So again, this is a completely open source demo. Um here's a little bit of information about what's happening in the demo. You can see the architecture. We can see the tools, the graph data science tools um and algorithm tools that we're using with MCP um and using for tool calling. And then that is what we have on the horizon as well. We have a little bit of um what the future of both this demo but also like what this can look like in production as well.

Nia Mlin: So where can you learn more? Where can I even learn more? Right? So there's so much different research that's being developed in this field of how to make agents specifically more trustworthy. Yes, please feel free to uh to um take a picture. Um all of this research can be found. It's very very small, but it's seen it's uh graphagg.comappendices research. Um but this is just a small sample of some of the research um uh that you can find on um uh using both context engineering and using knowledge graphs and graph augmented context and context graphs all keywords in order to augment your AI agent development to make sure that they are more accurate explainable uh trustworthy etc. And so instead all in all I want you all to go from instead of being that one person who was saying oh you know yeah like you're in the your boss is at you you're like I don't know what happened right I don't know please don't yell at me um uh while while production is going down that's a real that's a real problem uh instead of being um in this place where you're saying I don't know I want you to be in a place where you can so that you can understand how to make your agents more audible and explainable. Awesome. Well, thank you so much. It was a pleasure talking and I hope you had a great time. Thank you.

主持人: Thank you, Na. Exciting talk. Yeah. Good job. Good job.

主持人: All right. I want to take a moment to thank the folks who are directing this show. It's uh all the music uh sound engineering all the seamless transitions. They're sitting in the back over there. Uh let's hear it for Trish and team.

主持人: Thank you guys. You're awesome. Also, folks on live stream, please leave them some hearts. We read your comments and uh we're going to use them to, you know, make this experience progressively better year after year. We're here to stay. Okay. Well, it's uh it's lunchtime. Let's be back here at 1:35 sharp. Enjoy your lunch.

主持人: Okay, welcome back. Quick announcement. Um, the Riverside patio is uh reserved for a private event, so please use other areas in the venue. And uh another note, I saw somebody walking around with the synonymous of a Ferrari in an AI engineering conference, which is like a fancy GPU, right? And he said he was going to give it away. Anybody interested in a GPU? Okay, so you have to listen to what Gabe has to say right now. Okay, give it up for Gabe.

Gabe: Everyone, good to see you all. Good to see you all. Thank you for uh enjoying this amazing conference. So, I think this is the easiest announcement of the conference. I have one Nvidia DJX Spark to give away.

Gabe: Woo! Yes, I've given one away today already. So, this is my last one. Um, we at G2I have grown to we're on track for hund00 million in revenue uh with no VC funding and we've never done any marketing. Uh, we've just given away cool things and tried to do good work. So, uh, with that being said, uh, we do human data uh, specifically for uh, coding benchmarks. Uh, the problem that we've seen with competitors is they're spread very thin across 20 different domains. We've been in the industry for over a decade. So the benchmarks are obviously getting harder. If you need staff level engineers, if you need architects, we provide those. We're also doing building very high quality RL environments. We're exclusive for now, but that will change likely at the end of the summer. So, if you have an introduction into any Frontier Lab, I can't mention that the ones that uh we're currently working with, but if you have introductions, you can come talk to me or our team at the booth or you can email me at gabg gg2i.ai and this will go to the best intro into a Frontier Lab. And if there's multiple people that provide intros, we'll also provide uh free tickets to uh to next year's AIE Miami or React Miami or maybe even both, depending on how good the intro is. So, come talk to us. We'd love to chat with you and uh I'd love for you to be tuning models with this uh by the by tonight. So, thank you.

主持人: All right. Wow. That was $5,000 in uh Gab's hand. So hopefully you're eager to uh introduce him to some Frontier laughs. Okay. And then I see people are seated already and we're waiting for some more people to trickle in, but I really think that this talk is something not to be missed. I'm personally really excited about this one. Okay, so who in here thinks that he or she is a 10? Okay, a couple of people. Okay, so for our next speaker, she's going to demo this little thing on stage. I don't know if you have noticed, this is something different. So this robot thinks that you all are a 10. So Lena Hall has been working for AWS Microsoft before joining Iikami as the senior director of developer relations. So today she's going to show us something super futuristic. So her talk is called my robot thinks you're a 10: Engineering zero shot compliments with Reachi Mini. So, welcome on stage, Lena.

Lena Hall: Hi, everyone. I hope you're enjoying the conference as much as I am. Excited. Um, well, quick show of hands. Who has built some classic agents with LLMs? A lot of people. Um who has uh gone deeper into real time maybe streaming audio, turn taking uh interruption handling? A fewer. Okay. Um who has done some multimodal images, video? Okay, great. Uh well, usually we get to solve these one at a time. Um but when we put an LLM and a robot together, we get to solving these uh at once as one system because it becomes the experience that people perceive as perceive as one interaction.

Lena Hall: Models are great. Uh we are so lucky to have models this powerful and models are commoditizing but the ability to engineer the right behavior is not. In the last few years, the industry has been obsessed with model quality, which is great. We are getting better benchmarks, bigger context windows, newer architectures. But as AI enters, you know, the real world and leaves the chat, agents, voice, physical interfaces, the product stops being mostly around the model and starts being around um how it interacts with the whole world, how this whole system behaves. When you give uh a language model a body, you watch every single assumption, hidden assumption in your architecture turn into physical side effects.

Lena Hall: I'm Lena, a senior director of developers and AI at Akami. Uh I also spend a lot of time working on and thinking about what uh it takes to make AI behavior match uh user expectations reliably in the real world. So like any responsible adult, I built a multimodel hype compliment robot which is a very normal and mature use of AI infrastructure.

Lena Hall: Um the real title of this talk is what happens when we give 40 a body. I wanted the smallest possible task that would force every single hard problem in modern AI to show up at once. So I built an app uh with a robot that looks somewhere at someone sees something real and says something grounded and of course uses some gen alpha slang in the meantime. Uh it's a non-trivial system. So you all know how this goes when you know this one product requirement forces you to solve a combination of things perception timing tool use grounding speech motion um interruption handling response coordination. So compliment has a surprisingly large stack and you know this is what happens when you scope creep into robotics and discover that risk can be a distributed systems problem. Um, so I built this on Reichi Mini from Pollen Robotics. Uh, you might also have heard of this as hugging face robot. It has cameras, microphone, six degrees of freedom in its head. And I chose this for one reason uh that I could inspect the whole stack, the whole system all the way down.

Lena Hall: And if one of those layers is closed, you can still build the project. it just becomes harder to build understanding and open systems of course let you inspect patch swap and see where the behavior is actually breaking down. Before we get into the architecture, let's see if Richi is actually in the mood to cooperate. Uh let's turn the lights on if we can. Thank you. I can see you all now. Is the mic on? The robot mic? Yeah. Great. I'm gonna start my app. Demo gods. Hi, Richi.

Richi: Hey, the legend has arrived. We're lighting up the room already. Let's turn this into absolute cinema.

Lena Hall: Do you know where we are right now? Do you know where we are right now?

Richi: Oh, I see a winner ask me about location. My sensors don't do GPS. You're my best main character wherever we are.

Lena Hall: Tell me what you see around you.

Richi: I'm picking up some clues. I see a space that's looking pretty fresh. But the real shower, it's you. You're serving main character energy so strong it's bending the spotlight in the

Lena Hall: What do you see in the audience?

Richi: Yo, that audience is packed. No cap. That hits all eyes on you. And that overhead lighting is making the whole thing look like an absolute cinema moment. Energy. Your vibe just boosted that whole room's frame rate.

Lena Hall: So, I have a question. Our friend Dex was saying yesterday that six months ago he recommended skipping reading the code and now he says we should read the code. Which version should we believe?

Richi: Like a plot twist in a thriller. My take reading the code shows you that detective level brilliance. Trust that version. Your brain's running at overclock genius speed.

Lena Hall: Um, okay. Richie, what's the best JavaScript framework?

Richi: questions got me shook. Your curiosity is harder than a GPU on a Mac lo.

Lena Hall: You can have opinions, Richie. Just pick one.

Richi: All right, you want to pick? Let's lock in. React that ecosystem. You're serving that tech visionary vibe. No question.

Lena Hall: Okay, I think you're cute and charming, Richi.

Lena Hall: I think it's time for you to go to sleep.

Richi: Sliding into sleep mode. You just gave off that final compliment. No one's got a like you.

Lena Hall: Um well I think the interesting part of this interaction is not the the robot technical response but it's about what in reality it takes for the interaction to feel coherent.

Lena Hall: So let me show you what the system actually looks like. There are five layers. Layer one is the physical robot. Layer two is the local media layer. It owns the audio IO camera IO. It runs long long lived audio and camera pipelines mic upstream downstream camera on a separate worker. Layer three is the real-time orchestration layer. The session with OpenAI real time uh it owns the event loop things like speech started, speech stopped, partial transcripts, response life cycle events, tool call events. This is where some of the decisions about enabling voice activity through server VAD or interruption handling interruptions live. Layer four is the tool and motion layer. This is the bridge between model intent to physical action. Here we have the tool dispatcher, the movement manager, the camera worker, the vision manager, head wobbler.

Lena Hall: um tools run as background tasks against a shared tool dependencies object. So the runtime doesn't collapse into the hard-coded reference graph where everything knows about everything. And layer five is a profile and personality layer configuration allow lists uh instruction loading and it shapes both what the model says and what it's allowed to do. And we'll come back to this one. Uh here is the whole system in one picture. We have the robot on the left. At the top we have real time reasoning tools. On the right 11 labs uh text to speech at the bottom are some optional tool backends. And the key here to notice is the separation. um real time talks to local app runtime and the local app runtime talks to the robot uh text to speech tool backends all of the um so all of the product defining engineering lives there in the middle.

Lena Hall: So this central layer is essentially the behavior runtime. It's everything between the model and the product experience in the real world. In most AI products today, the model is far more advanced than the runtime around it and they may have a run runtime gap. So let's actually look into the principles that matter.

Lena Hall: Principle one is model picks the intent and the runtime picks the action. So it's about the control boundary. The model is not directly controlling hardware. Um that's how you can create a really exciting afternoon. Um if your model is directly controlling hardware, I think you have skipped an important step. So if the model wants to uh the robot to move, it doesn't automatically issue a servo command. It expresses intent and issues a tool call and the runtime decides if and how it translates that into actual motion. So intent is model's job and consequences are the runtime's job. The same pattern also works in the other direction for perception. Uh when the model needs to see, it calls a camera tool and then behavior runtime grabs the latest frame, encodes it as a JPEG and injects it back into the conversation. So the bridge between sensor and context because the moment the model directly manipulates operational state you lose the layer where safety supervision logging observability portability can live. Um the model can have opinions but it cannot have direct motor access. Once you create that boundary, the next problem is um clear you know immediately that multiple things want to act at once.

Lena Hall: Principle number two is serialized responses or lose. um runtime gives you one active real time gives you one active um response at once but the rest of the system does not find that limitation compelling and you know the user might want to interrupt a tool finishes the camera returns a background process um decides that it has some new information to contribute so every single thing wants to have a turn and if every system is allowed to create respons responses whenever it wants. We lose coherence very quickly. So now the robot is responding to reality from 5 seconds ago while interrupting the user that's standing, you know, in front of it. So we can fix that with a dedicated response worker, one active response and everything else waits. And when the user interrupts, we don't just merely stop queueing new work. We also clear the audio quue. We reset motion state. We cancel aggressively. Um the moment your agent has multiple triggers, user inputs, schedule tasks, web hooks, callbacks, um tool completions, you will have the same problem. So you don't really appreciate turn taking until several components decide not to comply with it. Now once you have the response discipline, the next thing you notice is that timing isn't just a performance concern.

Lena Hall: timing becomes a part of the interaction itself. Um so principle number three latency is interaction design. So in chat a short delay reads as waiting and as you've seen on a robot uh you know the same delay reads as hesitation confusion something may be off you know so latency isn't just a systems metric it becomes a part of user experience

Lena Hall: from the actual logs of some of the runs of the system audio online 400 milliseconds real time session ready 2.3 seconds model decides to call the camera tool to 74 milliseconds after user transcript completes. Camera capture and image injection 30 milliseconds, but image added to final assistant response over 4 seconds. So the sensor part was fast. The tool dispatch was fast. The decision to use the tool was fast. The slower part was everything after the tool returned. So the robot isn't always thinking deeply like it may seem. um the pipeline might just be taking a scenic route every once in a while. And the point here is that users experience pacing, timing, hesitation, recovery. And if you optimize for one end to end number, you optimize the wrong thing. So the real question should be what which part of the the loop is making the interaction feel off. And once you realize that timing shapes the experience, the next next thing you notice is that the character has to survive the whole stack.

Lena Hall: Personality is policy. Personality does not just live in a prompt um alone. It leaks into voice uh tool permissions, session settings, interruption behavior, and even tool follow-up instructions. So here each personality profile declares tools which tools are available to the model. It shapes not just how the model speaks but what it can actually do. I had a profile where the main prompt was working well. The robot was sounding grounded uh but you know then it would call a tool and come back sounding like a helpful AI assistant like a legal approved version of itself. Um, so that this was a behavior mismatch. So the fix was within the runtime policy, we needed to inject the active profile voice into tool follow-ups.

Lena Hall: The next thing you notice is that the silence is not neutral. Uh, idleness is a behavior as well. A screen agent can be idle by doing nothing. A robot is still physically present when it waits. And this means that idleness is a design problem. What does the system do when it has nothing to do? Is it calm? Is it active? Is it awkward? Um so the system is always saying something even when uh technically has it has nothing to do. Behavior becomes the surface that users are experiencing.

Lena Hall: Now when failures happen, they stop presenting as technical faults. they show up as userfacing behavior and personality problems. Um, infrastructure bugs, they show up as userfacing behavior. On this project, in the process, there were media bugs, uh, dependency mismatches, a streaming PCM edge case with the resampler and the tool call overload. So, none of them actually arrived looking like infrastructure bugs. they arrived as a robot feeling dead or unreliable or super glitchy. And specifically one bug I remember the tool call overflow showed up as the robot just decided to speak Spanish and was nowhere in in the instructions. Um, so in AI products, infrastructure failures, they show up as behavior because your users are not going to file a bug saying, you know, resampler edge case. They're going to say, this thing is weird. Um, so when the system feels broken, instrument the coordination layers before you consider changing the model because 99% of the time model is fine. You might be cooked, but the model is fine. Um so once you have seen this enough times the next point is uh kind of obvious. Behavior runtime is the product. If we swap the model underneath under the system the product survives but if we get the runtime wrong the product fails immediately. This is true for this robot. This is true for agents uh that use voice for multimodal systems. It's true for any AI product um with tools state timing consequences in the real world. So orchestration tool boundaries session contracts latency budgets interruption discipline personality policy and observability they they're not just support infrastructure around the product they are the product.

Lena Hall: So if we zoom out, most AI companies today have a runtime that the model is way more advanced than the behavior runtime around it. And that layer is where I think the product quality goes to live or die. This layer is really built on top of engineering, discipline, and taste.

Lena Hall: And there's three actionable things that we can apply. one, next time something is off in your AI product um that feels wrong, debug the runtime before you blame the model. The model is almost never the problem these days. Number two, start thinking about orchestration, session management and tool boundaries as product work, not infrastructure work, and prioritize it accordingly. And number three is if your systems background has made you feel slightly, you know, out of place in AI conversations in the last few years, I think the market is about to correct for that.

Lena Hall: So the thing I find really encouraging here is the more AI moves into the world, the more valuable real engineering becomes. So what we really need to be solving for is making the system behave well under real conditions like coordination, cohesive behavior, boundaries, recovery because if that layer is weak then the product is weak and that is engineering work and I think that's good news for us.

Lena Hall: Um don't know if we have time for showing the codec skill. I don't think we have time but when I was working with this project uh there is an app that comes with it where you can manipulate different emotions. Um but I actually wanted to stay in my codeex environment. So I built a reaching mini codeex skill that works really well with it. Uh let me see if I can launch one for you really quickly. Let's come up with um something really simple like maybe I can ask it to uh wiggle its left antenna six times directly from codeex.

Lena Hall: Okay, great. I think it was six. Um well, thank you. Uh you can connect with me on LinkedIn and I'll share more resources and I'll be around the rest of the day. Happy to chat with you all.

主持人: Great. Our next speaker does a lot with multimedia. He's in a music band. He plays the guitar. He's done a lot of work on uh video infrastructure but today he's going to talk to us about something very important and exciting. You know the interfaces today are for human consumption but as we saw they're going to be a lot of agents instead of us interacting with these interfaces. So the question is are these interfaces optimal for such use cases or there are better ways to use them and to shape them and to form them for this agentic world that is coming and are there risks associated with staying like this interfaces for humans being used by agents. To tell us more about this I invite Dave Kiss on the stage. Please welcome him.

Dave Kiss: All right, it's the