Announcer: Launch control. We have a go. Roger. right here. I need the right feeling I need the right feeling. Ladies and gentlemen, welcome to the AI Engineer World's Fair. We're delighted to have you with us as we continue exploring the ideas, technologies, and people shaping the future of AI. Please join me in welcoming your MC for today's program, developer relations engineer at Replet, Ralph Shabri.

Ralph Shabri: Good morning, San Francisco. How's it going, guys? Welcome to AI Engineer Day four. Wow, I'm so excited to be here with you guys today. So, this is our biggest AI engineer event ever. We have about 7,000 attendees. We also, my god, when I look at this room, uh, yeah, it's you guys, you made it to day four. So, let's give it up to you guys.

Ralph Shabri: All right. I live in a tiny village in Switzerland, and I can't believe that I can fit my entire village right here in this room. Uh, speaking of Switzerland, uh, who's here from Europe? Wow, so many of you guys. And I see that you're strategically placed under the AC. So, uh, please guys, white in America, stay cool. All right. Anybody local to the Bay Area today?

Ralph Shabri: All right. So, many of you as well. Nice. Okay. I to be honest, guys, I'm super excited to be in San Francisco. I think there's no place to build for AI and talk about AI. And what I think I think you guys are the coolest people in the world because you're constantly living in the future. Um, you guys drive self-driving cars, but you also have food delivered by drones. And what I noticed nowadays is that nobody closes their laptops anymore. I don't know. Everybody seem to be Asian maxing. Speaking of maxing, you guys enjoyed yesterday's talks.

Audience: Yeah.

Ralph Shabri: Let's give it up for our speakers from yesterday. All right, there was so much to talk about and to think about yesterday. So we had Tariq from Enthropic talking about Fable 5 and uh the process of discovering new models capabilities. We had another Tariq from Sonar who talked about uh code verification. But what I know what I noticed and what I heard from speakers is that it's still very important to stay in the loop while working with Frontier uh Frontier models.

Ralph Shabri: And today guys, it's going to be about harness engineering. And we have a great lineup of speakers for you today. So we have speakers from leading organizations such as Anthropic. We have also um Stanford. We have folks from DSPI also who are going to talk to you about new protocols. They're going to talk to you about how to separate the task from the model and uh and also they're going to talk to you about how to scale AI systems. And for having seen some of these talks, I can tell you guys that you've been that you're here for a treat.

Ralph Shabri: All right. And after the keynote, we have breakout sessions and we have so many tracks for you guys to choose from. So we have tracks such as uh software factories, generative media, memory and so on. And please take also the time to go and check out our expo. We have so many cool swags. Our partners are there. uh go uh talk to them and and uh and please if you if you need to take any swag, it's gonna be this is today. This is the last day. So, uh it's now or never. Uh speaking of partners, please a big shout out to our presenting sponsor, Microsoft. All right, let's give it up for Microsoft, guys.

Ralph Shabri: Also, let's give it up for all our lab and platinum sponsors. Please keep it up for also for our gold sponsors for our silver and and and bronze sponsors. This is super important guys. Without the support of each one of these companies here, this event wouldn't be possible. All right. So again, we're going to talk about harness engineering. But before we introduce our first speaker, what I want you to know guys is that you as attendees, you have huge power over the quality of the talk that you're going to be attending today. Okay? So, our speakers have been doing this forever. They know their stuff inside out. But what we want them to do is to feel the energy in the room the second they hit the stage. All right. Okay.

Ralph Shabri: So, we're going to practice that for a second. So, at the count of three, I want you to imagine as if I was a speaker and I want you to welcome me as a rockstar. Okay. So, one, two, three. All right. Nice. So, let's keep it up for our next speaker. Okay. So without further ado, please let's introduce our first speaker of the day. She is a partner at amplify. Please join me welcoming to the stage bar your own.

Announcer: Now joining us on stage is the partner at Amplify Baron.

Bar: Fantastic. You did a great job practicing. I feel very very loved. Um, let's get started. So, like you just heard, my name is Bar. I run a survey every year on the state of AI engineering. And the funny thing about running a survey on the state of AI engineering is that the field changes as you make the slides. Just in the past week, we've had Frontier releases treated like national security events. Meta reportedly exploring selling AI compute. By the time I get off stage, maybe something else will happen. So, if I miss a major announcement while I'm up here, please come find me after.

Bar: But that's exactly why we run the survey every year to cut through the noise, take a moment, step back and understand what AI engineers are actually doing. Uh for the first time this year, we were thrilled to partner with Notion and Verscell to run this survey. Very quickly on me, uh this is the least interesting slide. I'm an investment partner at Amplify. Very lucky to invest in companies built by and for AI engineers. And I'll make the same promise that I make every single year, which is short time on bar, long time on bar charts. So, let's get right into it with lots of bar charts.

Bar: First, let's talk about well, maybe raise your hand. Did you fill out the survey? This is a very large group. Okay. Yes, I see you in the front. Um, if the answer is you, thank you so much. If the answer is not you, I will find you in 2027. But genuinely, this only exists because a thousand of you gave your time. So, thank you. We had 1,048 respondents this year, which is a lot of AI engineers. And to be precise, this is not just AI engineers. As I'm sure you see at the conference, every year we see that AI engineering is more of a discipline than a job title. It touches founders, CTO's, engineers, product people, folks across company sizes and experience levels.

Bar: And that range shows up in experience too. Um, for the third year running, we see the same pattern which is skew towards senior engineers but newer to AI. Of those with over 10 years of software experience, over half have three years or less of AI experience, which tracks uh these are very experienced engineers learning a new paradigm in real time. And the newest cohort, the ones who just started uh engineering the median new engineer has nearly as much AI experience as the median 10-year software veteran. Uh so the newest engineers have never known software without this.

Bar: But doing AI doesn't mean one thing. We talked about all these different titles, all these different roles. Before we get into models and agents, I have a more basic question which is when people say they're doing AI at work, what are they actually doing? So first up, like to start with the modalities. We asked which modalities are you actively building with at work? Can anyone take a guess? Text dominates. I know. Hold your applause.

Bar: Um, but one piece of this chart that I always find very interesting and I always look at is the ratio of nope, I'm not using this modality to I'm not using it, but I do plan to. I call this the intent to adopt ratio. Of the people who are not building with a modality today, how many say they plan to use it? And audio has the strongest intent to adopt this year. Among AI engineers who are not building with audio today, a whopping 56% say they plan to adopt it in the AI applications they build. And this is not a brand new signal. Last year audio also had the highest intent to adopt across modalities, but 37%. So audio continues to take the lead and have high interest, but that interest is accelerating now. There has been an audio swing, but if we look at

Speaker A: what changed most from the last year in the survey, the biggest jump is actually in people using image generation. The share of respondents using generative AI for images and feeling really good about it doubled from 18% last year to 36% this year. Makes sense if you look at what we launched in the same window. Over the past year plus survey time, uh, we've had models Nano Banana, Nano Banana 2, Chat GPT Images 2.0. The products have gotten much better. What used to feel like an efficient way to generate cursed hands is just increasingly becoming a part of real work. Audio may have the strongest intent to adopt, but image generation shows us what happens when a modality crosses that threshold. So, I'm excited to continue watching these adoption curves every single year. I think we're going to see a lot this year.

Speaker A: Uh, now models. If you've Who here spends time on Twitter? All right. Yes. I imagine this is a very Twitter pilled uh crowd. If you spend any time on Twitter in this in this circle, you've seen a lot written about openweight models these past few months and I think we'll see it even more in the next year. Um, so we asked what models are you actually using in production? 94% use closed models. 45% are using openweight models. But here's the thing, you know, openweight models are not replacing closed models for the most part, at least not yet. The respondents using openweight models, over 90% of them are also using closed models. So they're looking like an augmentation. Teams are mixing and matching.

Speaker A: We also asked just to double click on this for the top three considerations when choosing a model. If you're choosing a model, what is important to you? Um, and despite the airtime of the open versus closed, it's not what drives model choice. It was a top three consideration for only 5% of the respondents. What matters is actually more straightforward. It's quality. Quality dominates. followed by agentic capabilities like tool calling and cost tied right with it. We'll money money. We'll get back to that. Um, one thing that I found very interesting is that reliability is not near the top. Only one in five named reliability. That doesn't mean teams stopped caring about reliability. Uh, there are different ways to interpret this data. I guess is that it's more likely to become a threshold requirement and the models they're choosing are reliable enough so the decision moves up the stack outside of certain circumstances to quality capability cost but we could talk after all right so here's where the model story all comes together

Speaker A: like I said teams are not choosing one model and calling it a day earlier I showed that 87% of teams are using more than one model uh the model that's the opposite of standardization. Uh and the way that they choose models for given tasks varies. Most popular is routing by task type. Some run multiple models, compare outputs. Some route based on cost. Uh but models are good at different things. What was interesting was that more than half of respondents said that their organizations starting to standardize on fewer AI tools. They're trading standardiz flexibility for standardization. A share of those are mixed. They say they're standardizing on some layers while staying flexible on others. But the headline here is that there's we're in the early great standardization of the platform and tools, not the models.

Speaker A: All right, this is the slide where anyone who's opened an AI bill in the last year starts nodding. So it turns out that infinite intelligence still comes with a usagebased bill. Once teams are managing many models and AI workflows, the next question becomes cost. Cost is now a first class engineering constraint. We see this in the data. 40% of respondents say that cost regularly shapes how ambitiously they use AI and another 36% say that it sometimes does. Well, this is pretty straightforward. So all in about uh three out of four respondents are adjusting their AI usage based on cost and maybe the fourth has a company card. That might be surprising or maybe it's obvious but 12 months ago it was not. Token maxing is cool. Being able to find real use cases is amazing but cost is becoming a real big part of the product decision today. And it shows up in monitoring too. What folks are monitoring in production includes cost and token usage as the number two thing they watch for. It's being monitored like an SLA right under quality itself.

Speaker A: Which brings us to the biggest line item of them all. Agents. Well, we've been talking about agents for a while. Uh this year, as you've seen, as you'll see today, as you've seen in previous days, you're going to talk a lot about harness engineering. They're escaping demo world. So, we asked respondents what level of tool permissions their agents typically have. And this is where agents start to look more real. There are two things happening at once. First, and I don't think this is surprising, relative to last year, there are far more teams using agents. This year, 95%, this seems high to 95% say they're using agents, roughly double last year. Second, amongst the teams that are using agents, those agents are much more likely to have write access. Last year, 52% of folks building with agents said their agents could actually write data. This year, that number is 89%. So when you combine these two shifts, more teams using agents and more of those agents having write permissions, the share of all the respondents and again it's a survey using write enabled agents is up more than three times relative to last year. So this is really the big shift. Agents are no longer reading, summarizing, drafting. They're taking actions inside of systems.

Speaker A: And that raises the obvious question, how are we controlling all of this? um with pretty blunt instruments. Uh very there are many ways that folks are controlling agents today. The top two are human in the loop approvals and gating permissions which are the right instincts but kind of the same toolkit you'd use to manage an intern. Below that the results scatter. Task decomposition, retrieval, memory, sandboxing, people are trying everything. Nobody has settled the control layer for agents. uh memory and persistent context is one that I'm watching very carefully right now. I think it's going to evolve a lot in the next year. And when agents fail or when people complain about agents failing to be more precise, it's usually the thinking, not the plumbing. So, uh you know, like twothirds say that hallucination or losing context mid task is what frustrates them the most.

Speaker A: All right, so agents are out in the wild, which makes it a good time to look at what everyone's actually running underneath. So let's take a peek at the stack. Um, we asked, what is the biggest challenge in your stack? Every single year that I ask this, the answer, the number one answer is eval. Um, so eval lead here, same as always, but by a very thin margin. Like that margin is getting smaller. And I'll say the quiet part here, which is that 96% of the people in the survey in this room have a problem with the stack. You just can't agree on which one. Um, so if you're deciding what to build next, if you're interested in infrastructure, that scatter is the map. And the leading challenge, how to evaluate your AI outputs requires many different methods, but as always, the vibe review is number one. So there are some consistent things that we'll see if they change over the time, but they they have not changed.

Speaker A: Okay, this is interesting. So across eight layers of the stack, we asked what do people build versus buy? Um, again, maybe the corporate card is is going to play a part in this, but there is a wide range and mix for every layer of the stack and a few clear takeaways. So the first is that inference and model serving is the layer that people buy the most. Many people don't want to build inference infrastructure and fair enough. Uh prompt management is the opposite. 61% build it themselves. Um apparently everyone's prompts are special. And this is true of a lot of the product logic prompts rag eval. They tend to stay inhouse on a relative basis. fine-tuning is the clearest. Not yet. Like most people don't have it at all. And uh folks are pretty locked in. So those who bought aren't looking as much to build. Those who built aren't looking

Speaker A: as much to buy. but those are the core takeaways from the usage in our stack. So many of you work on teams and like we said at the start these range from solo founders to large enterprises. What is this doing to teams?

Speaker A: And remember this is a builderheavy sample but among builders the vibes are good which you know I'm sure if you look to your left and your right you're feeling that the vibes are pretty good. 97% report a net positive effect on their organization.

Speaker A: The top effect isn't really just speed. It's cheaper failure, more experimentation, more prototypes, more bets. It didn't just make engineers faster, but it made trying things nearly free. And so there's some happy campers as a result of that.

Speaker A: But it's not free free. You know, there's no free lunch as nothing is. So the same tool that increases experimentation also increases review burden. Both can be true. And you know over nine and 10 respondents are feeling negative downstream effects in some way.

Speaker A: The most common ones being widely discussed at this conference online and anywhere that you see AI engineers erosion of deep technical skills and understanding of the codebase. And these are consequences of cheap code generation.

Speaker A: And the org chart is really feeling it. So many folks, 81% are saying that AI is blurring the line between their role as engineers and product design and marketing.

Speaker A: These stats shocked me. where you feel it the most is shipping software once exclusively the engineers domain. I know folks talk about vibe coding and how that's accessible to more folks than ever before in different roles, but today over a third of teams have non-developers shipping features, which was pretty wild to me.

Speaker A: Mostly smaller, mostly internal, but 17% say that non-developers are regularly shipping customer facing features across the stack. And even when non-developers aren't shipping, a third of teams see them building really useful things, prototypes, front-end mocks, and more.

Speaker A: So shipping software is not gated on being an engineer. We knew this, but the extent to which it's being pushed is higher than I expected.

Speaker A: All right, so where does all of this go? We always ask people to place bets rapid fire. So let's talk about those results. so present tense first. 76% say AI boosted their job satisfaction. So that's good for most of this crowd. I hope you're as Alphaba and Glenda say, I hope you're happy now.

Speaker A: that's great. But 59% fear today's AI code creates long-term liabilities. Only a third call software engineering a solved problem. Although when I have conversations with folks sometimes the way in which they define software engineering is different. So you can read into that stat as you will.

Speaker A: happier faster but embracing the maintenance build is the TLDDR and people are unsure what's going to happen with hiring.

Speaker A: And for the five-year bets we have 67% expect a leading lab will declare AGI in the next five years. Note the wording. We said, "Will," we asked about the press release, not the achievement. So, will they declare it? Yes. What does that mean? Not sure.

Speaker A: only 9% bet on Transformers being state-of-the-art in 5 years. Most are unsure. but that was interesting.

Speaker A: And then my favorite, will there be more AI compute in space or on land? 36 yes. 38 no. The most divisive question in the survey is about outer space.

Speaker A: I promised you a lot of bar charts and that was a lot of information. So a review or our 2026 wrapped impact is overwhelmingly positive. Image gen doubled or happy image gen doubled while audio has the highest adoption intent the same as last year.

Speaker A: cost really became a first class constraint and we see that everywhere in monitoring in how ambitious folks that are going out and building AI products are behaving. Open weights augment but they don't replace. So we're seeing a multimodel future with a consolidation of the stack.

Speaker A: Agents got right access more than ever before, tripling relative to last year, while the guardrail stayed pretty primitive and inference is the buy market. Everything closer to product logic tends to relatively stay more in-house.

Speaker A: It is a very exciting time to be an AI engineer. I cannot wait to see how the next year unfolds. So, you can find the full report in the link up here. Every chart plus some cuts that we didn't have time for today.

Speaker A: I won't ask you to fill out a survey about the survey, but if there's something that you want on the books for 2027, something you're curious about, you can come find me here on the internet. I'm easy to spot. Thank you so much. we will see you next year or per 36% of you, maybe in orbit. Thank you.

Speaker A: Please welcome to the stage the professor emmeritus at Stanford University, John Sterhout.

John Sterhout: Good morning. It's really great to be here to talk about the network side of AI applications and in particular to make the case that latency matters and it's probably going to be mattering more in the future.

John Sterhout: But I just want to say this is a talk is unusual for me. I've never before given a talk where there are fog generators in the auditorium. Just a really San Francisco experience, I guess.

John Sterhout: So it's well known that AI workloads depend on really great networking performance in order to achieve their own performance. And of course that's because the workloads are so large that they have to be distributed across machines and then you have to communicate between the machines.

John Sterhout: But what I want to talk about today is that it seems that those workloads are changing and so I hope to do three things over the next 15 or 20 minutes.

John Sterhout: First to convince you that in fact the workloads are changing and that whereas the workloads used to be completely dominated by large transfers where throughput is the key metric that matters that we're seeing more and more smaller transfers where the latency is crucial.

John Sterhout: The second thing I hope to do is to convince you that legacy protocols like TCP and RDMA are poorly suited to this environment. They weren't designed for this environment and unfortunately they suffer from very high tail latency when you mix small messages with large ones. I'll talk a little bit about why that's the case.

John Sterhout: Then third, I'd like to introduce HOMA which is a new protocol we've developed at Stanford that actually was designed in a clean slate redesign to handle data center workloads like these and in fact it does quite well on those workloads and can reduce tail latency by an order of magnitude or more. So I'll tell you a little bit about Homa. So let's dive in.

John Sterhout: First, workloads. Historically, AI workloads have consisted of enormous transfers between machines, and that's all that really mattered. Gigabytes of data for things like weight gradients and and so on. In these workloads, what you really care about is throughput, how many gigabits per second you can pump through the pipes.

John Sterhout: And these are relatively easy workloads for networks because if it takes a while to set up the connection and start the transfer, it doesn't matter. The transfers go on for so long that all that really matters is the throughput.

John Sterhout: And so in these environments, TCP and RDMA perform pretty well. by the way, when I say RDMA, what I really mean is Rocky RDMA over converged Ethernet, which is the underlying transport that's used by RDMA for most purposes today.

John Sterhout: So anyhow the old workloads big transfers throughput matters. the legacy protocols work pretty well.

John Sterhout: However, it appears that the workloads are changing. They're becoming more granular with smaller chunks of computation and smaller exchanges of data. And this seems to be particularly true in the world of inference and also in agentic workloads. Not so much for training workloads are still massive transfers.

John Sterhout: And so what's happening is that more and more there are small message exchanges typically for things like metadata and coordination such as checking to see if a particular entry is present in a KV cache that's distributed or doing barrier synchronization at the end of periods of compute.

John Sterhout: And for these workloads what really matters is latency. That is what's the roundtrip time to send some small piece of data across the network do a little bit of computation and get a small result back again.

John Sterhout: And in fact, it isn't just just latency or average latency that matters. What

Speaker: really matters is tail latency. That is, you'd like to know that if we send a whole lot of small messages, all of them will complete quickly.

Speaker: So, for example, we typically measure things like 99th percentile latency. And if we have high tail latency, that can limit the overall throughput of the system.

Speaker: So, here's an example. Suppose a common thing is to take a workload and split it up across several nodes which do intensive computation using their GPUs for some period of time and then once they've all finished their computation you do some small exchange between the nodes exchange data metadata and then it'll go on to the next round of computation and while that exchange is happening that synchronization is happening the GPUs are sitting idle so if even one of those exchanges takes a long time. It turns out the whole process stalls. You need all of those exchanges to complete before you can go on to the next phase of computation.

Speaker: Now, if the computation phase is say 5 seconds and it takes a few milliseconds for the exchange, you know, not a not a problem. And that's historically what it's been. But now with agentic workloads where you're trying to pump out tokens relatively rapidly at a regular rate, the periods of computation are getting down into sort of the millisecond time scale. And if it also takes milliseconds to do that synchronization, then you're wasting a significant fraction of your GPU resources waiting for the the synchronization to occur.

Speaker: So I'm curious. I'd like to just do a a quick audience poll here. Is there anybody out there today where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications? If so, can you just raise your hand? See, is there anybody out there today? Actually, more hands than I expected. So quite a few people out there are raising their hands. I think this problem is likely to get worse as the trends continue.

Speaker: So what's going on? Why is tail latency bad? Well, typically the cause is congestion resulting from incast. So incast is when several nodes all decide simultaneously to transfer data to some destination node. And if they all send large messages, well, the links are the same everywhere in the network. So three nodes can transfer three times as fast as one node can possibly receive. And so what happens is that packets accumulate at the last hop going to that destination in the top of rack switch at its egress port for the destination node.

Speaker: Then if some other node decides it wants to send a short message to that same destination, the short message gets stuck behind the long ones in the queue there. And actually that causes delay and in the worst case so many packets arrive that the switch runs out of buffer space that it has to drop packets and then there are timeouts and retransmissions that make everything even worse.

Speaker: So somehow we need some way to reduce the congestion in those queues. Somehow we have to get the sending nodes to stop sending so fast so the queues don't just build up without limit.

Speaker: So the way this is done historically virtually all network protocols before HOMA including TCP and RDMA congestion control is the responsibility of the sender. So senders somehow have to figure out that congestion is happening and they have to slow down their rate of transmission.

Speaker: Now you you might wonder why are senders doing it? because the congestion is way over at the other end of the data center network. How does the the sender find out?

Speaker: Well, in the in the old really old days, the way they would find out is the queues would overflow and packets would get dropped. The sender would detect the packets got lost because it wouldn't get acknowledgements back and it would assume that means there's congestion and then slow down its rate of transfer. That's really expensive.

Speaker: So today there are better techniques that mostly involve the switches providing information. So a top of rack switch when it sees that the queue length for an ingress port has reached some threshold starting to fill long before the queue overflows it starts marking all of the packets that pass through with what's called early congestion notification ECN marking.

Speaker: And so when those packets pass through to the receiver, the receiver sees the marking in the packets. And then when it communicates back to the sender next, for example, to send an acknowledgement, then it includes that marking that goes back to the sender. And now the sender sees that the sender realizes, oh, there's congestion someplace. I've got to slow down my rate of transmission.

Speaker: So that's the basic idea. Unfortunately, getting this right is really hard. Really hard. It's very hard for the congesture to figure out exactly how to set its rates because it gets one bit of information. There's congestion someplace and there are multiple senders all sending to the same destination. They're all trying to make adjustments simultaneously. How much do you cut back and how do I know when I can ramp up again? And even worse, it's really hard to do this in a way that's stable because there's control lag.

Speaker: That is, it takes time before the sender finds out that there's congestion. And in fact, using this process, it typically takes several round trips for the sender to gradually adjust its rate to get just the right rate to match the available bandwidth. But by the time you do that in a network that things have changed, new transmissions have started or old ones have finished. And so these systems tend to never stabilize. They're constantly oscillating between sending too much and and sending too little.

Speaker: Now, this problem's been around for a long time. It's been known in the research community for more than 20 years now. There have been tons of papers published on it. There have been some improvements made. That's undeniable, but we're still a long ways from anything that works well.

Speaker: And the problem is is with the fundamental nature of it doing the the congestion control on the sender side. It just doesn't work very well. So you end up with a lot of queue buildup. And in fact, you can see the only way to find out that there's congestion is if there's queues and so by that point, we're already experiencing delays. So that's a problem.

Speaker: There's one other problem with TCP and RDMA also is that their their basic data model is a byte stream just a stream of bytes with no differentiation in it. So if you send a series of messages say through a TCP socket they get serialized into that stream.

Speaker: And on this slide I've you know I've shown the messages appear like they have different colors in the stream. Well there are no colors in real life. TCP has no idea where the message boundaries are. And that also makes life hard. For example, you don't know how much more data is coming. If you knew how big the message was, you'd know how much more is coming. And you can't prioritize short messages, which we'd really like to do. Get the short messages through faster.

Speaker: And you could end up with what's called head of line blocking where somebody sends a series of messages to the same destination and they send two really large ones and then a small one after that gets stuck behind them in that stream and so it gets delayed and again you have tail latency issues. So all in all, TCP and RDMA just not well suited to this environment.

Speaker: So what do we do? Well, what I'd like to do next is tell you about a new protocol called HOMA that we've developed at Stanford, which was based on a completely clean slate redesign for network transport. If you could start from scratch and rethink how you do transport for data centers, how would you do it? And it turns out in Homa virtually every major design decision is different from TCP and RDMA. TCP for all the amazing things it's done is just not a good match to today's data centers nor RDMA.

Speaker: So what Homa does particularly well is to manage a combination of large and small messages and to make sure that the messages have short messages have really low latency.

Speaker: So this started off as a PhD dissertation for one of my students, Benam Montazeri, and then the results were so great that I decided to make it my personal project to see if we could get it out of the lab and into production. Uh, as you may know, I'm I'm not like most professors and that I love to code. And so

Speaker A: I turned this into my own programming project. I created a kernel module for Linux. I'm currently working through the process of getting that upstreamed into the kernel. It's available on GitHub for download. So let me tell you just a little bit about how Homa works. I want to mention three things. First, it's messagebased, not streambased. In fact, the fundamental unit at HOMA is a remote procedure call, which consists of two things. A request message sent from a client to a server and then a response message returned back from the server to the client. So the key thing here is that Homa knows about message lengths. They're buried in the transport all the way down to the bottom. And this has a bunch of advantages. First, it allows us to predict the future. As soon as a receiver gets the first packet of a message, it knows exactly how much more data the sender wants to send. And that's so has so much more information for doing congestion control. Second, Home Prioritizes shorter messages. It uses SRPT, shortest remaining processing time first to try and prioritize shorter messages. And third, because messages are all independent, they're not serialized into a stream. Every message is independent. Shorter messages can bypass long ones so they don't get cued behind long messages.

Speaker A: The second thing about HOMA that's different is that it controls congestion from the receiver. Now, when you think about it, this makes sense because the congestion happens primarily at that last down link to the receiver. And so, the receiver has way more information. In fact, with HOMA, as soon as it gets the first packet of a message, it knows exactly how much more is coming. So, it has essentially complete information about congestion and it can therefore respond to congestion much more quickly and much more precisely.

Speaker A: The way things work with HOMA is that when a sender has a message to send, it breaks it up into packets, but it only transmits the first few packets, those are called unscheduled packets, to the receiver. Packets after that are called scheduled packets and they only get transmitted when the receiver asks for them. So the receiver will send grant packets back. It'll paste them out and send those back to the sender over time telling the sender it's now time for you to send me the next chunk of data. And the receiver can delay those grants. So for example, if the receiver has 10 messages that are incoming, there's no point in sending grants to all 10 of them because then you'll just get congestion in the in the top of wrap cues. So it can use the grants to reduce congestion and then it can also use the grants to give preference to its most favorite messages which would be the shorter ones. So it's a way of of implementing SRPT by favoring short messages.

Speaker A: The third aspect of Homa is that it takes advantage of the priority cues in modern switches. So modern data center switches have more than one Q at each egress port typically eight and they can be used in a priority mechanism where packets get transmitted preferentially from the highest priority Q. So I've shown only two Q's on the slide here but typically there's more than that you can specify in packets using the various fields of the packet you can specify which queue it should go into and so Homa dynamically makes those choices in a way to give priority to shorter messages. So if we go back to the incast example from a few slides ago, all of those long messages will pile up in the lowest priority queue. But if there's a short message coming, it will use a higher priority queue. And so it will immediately bypass all of the cued packets from the short from the uh the longer messages and get through to the destination more quickly.

Speaker A: So how much of a difference does this make? Here's a this slide. I've got one sample benchmark that I use as part of my tuning and evaluation of Homa. It consists of a workload of a bunch of machines on a network that are exchanging messages back and forth of different sizes ranging from very small to very large. And on this graph you can see on the x-axis is the message length from about 50 bytes up to a megabyte. The yaxis shows you the roundtrip time for messages of that length. So this request this uses request and response messages that are the same length. You can see TCP in green, Homa in blue. And the y-axis is is roundtrip time. So lower is better. And for each protocol, I've got two curves. One curve is the P50 curve. That's the median latency for messages of this length. And then P99 is the 99th percentile, i.e. tail latency for messages of this length.

Speaker A: So I want to point out two things. First, the P99 for short messages is dramatically better for HOM. So with TCP it's more than a millisecond tail latency. Home is less than 100 microsconds about 13 times faster. Second, interestingly you might think that because Homa favors shorter messages that long messages suffer and get worse performance. It turns out that's actually not the case. Even on the longest messages, Hom is almost a factor of two better than TCP. I don't have time to explain that today, but it has to do with the fact that HOMA uses run to completion approaches which are much more effective than fair than the fair scheduling used by TCP.

Speaker A: So, just to wrap up, the role of short messages in AI appears to be increasing. I think I think it's likely that it's going to continue to increase. We'll see over the next year or two if that happens. And I just want to pose a question to you. You know, as you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput? If the answer is yes, then just know there is a solution available. You should give Homa a try. You can probably reduce your tail latency by an order of magnitude or more.

Speaker A: And by the way, this is Homa is basically my life mission right now. I'm sort of semi-retired from Stanford. The reason I did that is so I can spend 100% of my time hacking on Homa. So I'd be delighted to work with you and help you if you decide you want to experiment with Homa. If you need help getting started, answer questions, bug fixes, whatever, you know, I'd be happy to work with you to try and make you successful with it. So if that is interesting, feel free to contact me. My email is on the slide or you can Google me too and find me over the internet. So thanks very much for listening and hope to hear from some of you.

Speaker A: Please welcome to the stage the core contributor and the lead maintainer at DSPI, Maxim Rest and Isaac Miller.

Maxim Rest: Wow, Isaac, myself, all of the DSPI community are so grateful to be here today to get to talk to you about AI programming, DSPI, and the unreasonable effectiveness of separating the tasks from the model, its harness, and all of the implementation details.

Maxim Rest: When you think about it, in programming, if we want to repeat a tasks often, we make it a function. We believe the same should be true for AI programs. Functions are awesome. Functions are reusable, composable, testable, and optimizable. To make a function, you give it a name. You define some inputs, some outputs, and then you have some implementation logic inside of it. You get to reuse your functions do thousands of times. You can optimize it, but you can also compose it into bigger programs. One of the really nice things about functions is that you can also package it and distribute it and someone else can use it and they just need to know about the contract on top of it to use it and they can treat it as a black box.

Maxim Rest: DSPI brings all of these properties to AI programs. And so DSpy is an open source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs. And it gives you all of the toolings you need to do that. Why do you want that? Well, we have been inventing a lot of terms in our fields in the last three years. It's growing fast. We have new models coming every other week. We have new techniques, new strategies. And if you're like me, you want to try all of them. But will any of these new specific techniques coming out at a different time really help on your task, on your job? Well, these are all just implementation tactics and

Max: you want to put them inside of clear contract. If for your repeated AI task, you define an input interface and an output interface, you get to play in the internals, you get a lot of agility. Let's make it concrete for AI. So my first AI program I made when I discovered DSI was that I had some invoices from my farm and I wanted to extract them to do my taxes. I wanted to extract the tax values from there.

Max: Then another AI program I did is that on my keyboard in my computer, I have a little command that reads my keyboard shortcuts, read my clipboard, and will correct the grammar for me. Sometimes I actually want it to also rewrite for clarity. So I have another program that takes text, just rewrites it for clarity, put it back on my key keyboard, and that's a command. And then I can like have a lot of agility and bring it different places inside of that. I can change it however I want. A new model comes up and I can change that. It's super easy because my interface is fixed like that. I'll skip that one. But they're not uh restrained to very easy things and small input outputs. You can be very ambitious with AI programs.

Max: So in this examples, you could have your entire inbox and a new email coming in and you want to compose a new drafted reply. We can do that in the Aspire with RLM, recursive language models. This is an idea that came from around our community or more like things we probably all do, agentic engineering or vibe coding. You can give it a spec a repository and you get a PR. Those are repeatable tasks. And so as I have been telling you when you fix that boundary you can focus on the how on the top and then inside of it you can have a little chat with just a simple prompt. You can iterate on that prompt. Agents come out you change it to be an agent. Tools gets invented you add tools and then we get into loop engineering. You put that inside of it too. Anything on the outside of it doesn't change your integration and and anything else doesn't change. And when you have such a hard boundary, you can also start to automatically optimize.

Max: But how can you automatically optimize with just that simple signature? This is not enough. This is not enough to specify your task. And even before Chat GPD came out, the creator of DSP had started to land on this idea that you need three things to specify your task. And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. So the first one is what should happen. This is instructions. The signatures that I've been showing you are part of that. Here on the screen, you see the beginning of a real script in DSP. You set your model at the top. you configure that and it's fully independent of the signatures here where you have natural language instruction to extract all taxes and um and if it's illegible to output zero then you say it I'm going to give you an input it's going to be a string I want you to give me an output and it's going to be a string and a float this is natural language expressing my needs this is very powerful and efficient if you think about it if you have a friend over coming to play a board game with you and You give them the instructions and they're ready to play. But if you want to do like alpha go or alpha zero and you tell them you're just going to learn from example, you're going to have a long night.

Max: And then the second one is what must happen. There are some constraints you have that they have to be listened to. They have to be enforced. The best way to do that is with code. So I want you to go to the third line. A fourth line you have self extract and selfrecheck. You can see we're doing a predict on the extract taxes and we're doing a chain of thought on the extract taxes. The first one is a vanilla program. The second one makes it do some reasoning. Now I'm taking them inside in the forward and you can see in the if not red tax. This is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. I mean, I got to get my taxes right. And then another requirement I have is if the value is below zero, throw I want to show that to a human. I don't want to let you go. This will not change. Like even if I have AGI, I would hope it doesn't make mistake. But whatever is in the predictor, if they make these mistakes, I still want these things to be true.

Max: So the last one is what good looked like. And when I was young, I was on the farm with my dad and I asked him, "How do you know that this tree is a maple?" And he couldn't tell me. He couldn't give me the instruction on how to know this tree is a maple. And he certainly couldn't give me code on how to know this tree is a maple. And so through time with example I learned how to know that a tree is a maple. But this is not limited to things like classifying plants. It's also for all of the long tails in your specifications that are things that are more latent. These are sometimes a reason why you would do internship and you would have a mentor and a mentee. You're looking at a lot of of examples and there are long tales of successful behaviors that you have to see and learn.

Max: Now that you have all of these, you have express fully. You have all these three languages you can put together. You have the specs, the code, and the evals. And now your goal is fully specified. And so you can start optimizing. You can use things like Japa on your metrics and on your program. And you can start optimizing. At the beginning of the SPI, the chip didn't exist. The models were not good enough to optimize. And so we were using code to find few shots examples to make the base models uh act in the proper way. Then models got better and so we could automatically optimize instruction. And in the future we are starting to be able to be liberated more and more from the implementation details and delegate that away. And at the end our hope in the Aspire is that you can stick to all of that and then just the news and the implementation details will be automated for you. Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. Thank you.

Isaac: Thanks, Max.

Isaac: So, we've given you a pretty big abstract overview of specs, code, and evals, but these aren't things that are just restricted to the academic sphere. These are used in production by some of the biggest enterprises for massive gains. And we see two main benefits when you use DSPI in the enterprise. First is that your implementation becomes cheaper. When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. And you can use this to scale to data sizes that weren't possible with a more expensive implementation. Shopify 550 times cheaper. They're able to do that because they went from an expensive model to a cheap model, but they could keep the same emails, keep iterating on their business logic inside, and try new things. There's three awesome case studies here, and you should check them out after the talk. They give you a lot of details on how you can do this in your own enterprise.

Isaac: Now, part of the reason why you want to build in the DSPI ecosystem is that we're constantly adding new techniques for you to try. And it's important to note none of these techniques we add will definitely solve your problem because that's your job. What we can do is we can solve sub problems for you that make your implementation easier. For instance, Alex Zang, a PhD student at MIT, came out with this paper called recursive language models. Recursive language models are a way to solve some kinds of long context programs. And guess what? We can bring this in to DSPI for you to try see if it helps your long context tasks. Maybe it will, maybe it won't. But the thing is, it's one line and your signature stays the same. That's what's important here. Everything gets to stay constant and you get to see if this solves your problem or not. And we've had a number of examples

Speaker A: of this just in the last year from people building in and around the DSPI community. We've had RLMs. We've had Jeepa which is an incredible prompt optimizer out of Berkeley. Better together multimodule grpo. All these are incredible research innovations that you get to try in your implementation just by being in the DSp ecosystem. And we have more coming in DSP 4. I'm excited to talk to you about two of those today. DSP flex and qualitative learning.

Speaker A: DSPI.flex is a new kind of module. In DSP, when we let you optimize things, it started with few examples, then it became prompts, and now that's becoming code for any function that you want to implement. You can actually learn a harness over time to solve that function. And this is completely custom. And you don't care about the implementation as long as it solves your business problem. What you've created ways to measure because you've defined the three core parts of specs, code, and evals.

Speaker A: The second thing I'm excited to talk about is qualitative learning. One of the hard hard problems in AI engineering is building evals. And there's a few reasons why this is hard. One is that defining what good looks like is really challenging for any real world problem. The second is that when you define good often times you have to lose detail. If an email is good or bad contains a lot less information than if you know what could change in that email in order to improve. And the third is that whenever you create a hill and a data set, you're really trying to create a proxy for reality. What if instead we could use reality to inform our evals automatically?

Speaker A: What qualitative learning asks is how do we decrease this question? How do we decrease assistance? And it's a research question right now. But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into evals and a hill that the model can climb. And so as you get more feedback from production, its traces, its user actions, its product analytics, it's asking you, it's the model asking you questions about how data should be represented. As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem.

Speaker A: And DSP focuses on these kinds of last mile problems. We have a really strong research ecosystem and we collaborate really closely with them. And that's part of the beauty is that we can see the problems that happen in applied AI engineering. Sol define them, build a benchmark and then solve them with techniques and then we get to democratize the results of that to everyone because it's open-source open research.

Speaker A: Now, one common question is what happens when we have AGI? Well, even when we have an incredibly smart model, the model won't know how to solve your problems. It won't know how to do your tasks or have your context. And so this genre of last mile learning is trying to ask how do we efficiently do this learning intelligence is very different from being all knowing.

Speaker A: If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email. But if you AGI will know how to do your emails. Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. It won't understand your relationships without learning this context over time.

Speaker A: Since 2022, DSPI has been focused on these three core ideas of specs, code, and eval. We've certainly evolved over time, and new techniques are incredible. We've gone from evolving few shots to prompts to now harnesses and now evolving your eval. But what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better? And you should ask this question in a datadriven manner.

Speaker A: You should look at this new technique say how can I apply this to the business problem that I have? You should define your problem and you should hold your prompts, models, and code accountable to the problem that you need them to solve. And what's awesome about when you build in this way where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing. You unlock access to the collective intelligence of everyone here, all sharing techniques together.

Speaker A: So, if you want to build reliable AI software, I encourage you to come check out DSP. We're completely open-source, open research, and we're here to help you solve your problems by building reliable software. We have a Discord that you should come join. And when you come up with the next technique, you should come contribute it to DSPI and we can help you distribute it and make this awesome technique available for everyone. Thank you.

Speaker A: Joining us on stage is the co-founder of Instagram and a member of technical staff at Anthropic, Mike Kger.

Speaker A: How's everyone doing? I mean, good morning. Nice. Um, Mike, thank you for releasing Fable just in time for us

Mike Krieger: exactly for the conference. We timed it.

Speaker A: Um, we're so glad to have you. Uh, you are uh one of the preeminent builders and your leading labs at Enthropic. Um, how has your model usage changed as as you've you know seen models internally grow?

Mike Krieger: Yeah, I mean for me it's been like both the model shift and then my role shift. So I for like the first two years I was at anthropic I was chief product officer and then I kept seeing people build with the models and the FOMO just kept increasing because I was you use the models as much as possible but for example on product strategy I would write a strategy doc and then have cloud critique it and maybe you can use a workflow but it's not quite the same as like building in that pure way and I was like spending all my weekends trying to build with it and I realized okay I actually just need to shift it's like way too interesting a time and it's actually an interesting trend I've seen now like several people that where CTO's at other places are like now joining as IC's at anthropic in other places but I made a role shift and it was actually right around the time where we started getting sort of internal snapshots of what became mythos and fable

Mike Krieger: and what was really interesting watching that sort of shift was um that kind of change between I have an idea I'm going to like sort of break it down in my head much more how I would do engineering normally and then kind of iterate through these different steps to moving to much more of the paradigm of I'm going to describe the goal like go off and work on it and then like we can talk about what trade-offs you you know surface some questions along the way but then figure out what where you landed and where we can go from there.

Mike Krieger: I find it's hard. I don't know if people have this experience where and I know Babel's only been reenabled for a couple of days. Babel's definitely way way smarter than me. So sometimes it'll finish work and be like here's the trade-offs I made. I'm like can you explain it to me like I'm a little dumber than you are because I need you to like sort of break this down for me. But that's been one sort of big change is sort of moving from that task delegation to like express the end state and then have it go and and cook on it.

Speaker A: Yeah, we're all learning how to delegate better. Uh Tariq did us a huge favor yesterday. Um we do I want to read it in the newspaper. Um it's a you know that we have we have uh writeups of talks now in in like the next day's newspaper. He said be unreasonable. In what ways you know have you been more ambitious with your prompting?

Mike Krieger: I love that. I mean, I love that framing. Um, we actually just hit this to like I'm one of the labs initiatives I have is this internal product and uh somebody was like, "Hey, it doesn't work the way I want it to." Um, and can you make some changes? And I realized I'm just going to go ask Claude to do this. Like, why don't you ask Claude? And this was a non-technical person. So, I actually think as an industry or even as a product team, we have to teach people to be more unreasonable in their usage. And it's sort of hard to imagine. I think that that if I can digress for a

Speaker A: second on product design, I think right now the like kind of first generation of AI products, we put them too much in a box and constrain their their sort of access to tools or kind of degrees of freedom which means it was much harder to be unreasonable, right? When you say do this thing for me and then it would be like oh I I can't I can barely like I can write code but I can't really run it or I can kind of introspect my environment but not really. Um and I think as you see our own like product progression even with things like co-work like you know does every single like knowledge worker need a virtual machine that can write bash like on the face of it no but then when you realize oh actually that way it can remediate an issue where oh I tried to parse a PDF using our built-in PDF so I hit this yesterday and it was like ah I can't parse it this way well okay well I can probably write a script that can do this as well. Um so I think that's it. My most unreasonable thing though was u one of our labs projects I wrote in Python like near and dear to my heart. All of Instagram was in Python. I think they're finally converting it to PHP now that they have um like models they can but I know tokens. Um and uh for deployment I realized that cloud code had like figured out a better deployment story with bun and I was like okay I need to port this whole thing from Python to TypeScript. like as a you know if I put on my like 2010s engineering hat or even my early 2020 20s like that's a dumb idea like who would ever port like at that point you know a couple hundred thousands of lines of code um but I was like I think this is doable now and I basically created this dynamic workflow setup and over the weekend had it port the whole thing like verify it double check it then read both code like basically churn and churn and churn and then came back Monday to a completed workflow that was a ported version of that thing so that probably ranks on like the more unreasonable things like yeah just port this entire Python codebase to TypeScript. Get it working, get it deployable in, you know, a weekend.

Speaker B: >> Yeah. I mean, a lot of people are talking about the the bun zigg to rust version. I I think a lot of people are also like, well, it's a compiler. It's a it's a runtime. It's got lots of tests, easy to do. Can you port Instagram, which you would know very well to PHP like that, like a like a product?

Speaker A: >> Yeah. I mean, I think the product side, it's even I don't know if easier or harder. One of the things we did at Instagram, this is when Python 3 came out and we were able to add typants for the first time and it was people had a lot of internal conversations like are we going to run out of steam on Python and my perspective was always like I think we can take this way further than we think we can uh but I think types are going to help us not sort of be in our own way and we built this thing called monkey type where we basically like captured runtime type like basically the types that were actually getting used in production and then mapped those back to to the types in the codebase. And I think because of that sort of pattern, I think there's really interesting ways in which if you're doing sort of conversion or sort of cross-co compiling using LMS, you can also lean on production data a lot more or run sort of like segmented tests. I think that like there's a lot of uh things you can do there. But yeah, I think it's I mean the sky's is the limit there as well. I think the hardest part is always finding the boundary around where you can start doing it incrementally without trying to boil the whole ocean and like swap it overnight. Yeah, I mean your users are your test ultimately and um you know we I also read another article in the newspaper about how you can just use rollouts and sometimes you don't really know uh what you're going to need it for but when that infrastructure exists for you to experiment and to roll things out it's enable so much. Yeah, I mean I always found this was advice we got. It's like we launched Instagram and the happened to be the first week everything melted because we didn't really know what we were doing on the backend side of things. And uh coincidentally that week there was like a lunch that one of our investors just scheduled like not even for us. It was just a infrastructure lunch and we ended up spending we totally like monopolized that conversation because everybody had their own opinion about how we could fix our scaling. Um, and like the two pieces of advice I got there, it's like 2010 that I will like will forever retain is like um like basically like pre-measure everything that you think you might even remotely need because the worst thing is an outage where you're like well is this like number normal or is it high and like oh I don't know because I don't have data until I just added this metric. And the other one is being like really thoughtful about knobs and feature flags. So even you know early Instagram we had like a very uh simple but really effective like way in which you could do like ramp ups and roll outs and dynamic config too where you know a lot of our runtime configurations had to be changed you know in a matter of seconds so that we could handle load and being able to like do that in a first class way was was really important. I'm seeing that definitely in in AI as well where you know we're making all sorts of different trade-offs and having that kind of runtime configuration is super key.

Speaker B: >> Yeah. Uh my my favorite scaling story for Instagram by the way I think it's like your launch day when you you do yourself with email. Yes,

Speaker A: >> which people should look up that story if uh if you haven't seen it.

Speaker B: Um I wanted to go into tags uh very very major ship. Uh it's it's how 60 something percent of your code is written today.

Speaker A: >> Yeah.

Speaker B: >> Um how did you square that with everything you just said where it's like very dynamic like you don't actually ship one app, you ship one app with 3,000 flags. Yeah.

Speaker A: >> And like well what are you working on today? I don't know. Like it's it's for this segment of the population.

Speaker B: >> Yeah. Yeah.

Speaker A: I mean I think there's a bunch of things. So like with I was really excited. I was talking to Swigs earlier like I'm really excited that we have tag out there because it is uh how we've been working for a while and I would get up on stages and people be like how do you work at Anthropic and I'd be like oh yeah we use these things like that are not quite cloud code but you know uh but it's hard to describe it but I mean if you like got to poke into anthropic like you would see u of course cloud code usage for things that are like more interactive or if you're kind of iterating on a particular uh sort of sort of specific thing where you want a lot of like high sort of bandwidth back and forth But most usage is actually much more delegating uh via tagging and via tag. And you can say like here's the and the reason it's really interesting is how multiplayer it is. And it reminds me sort of like um actually like mid journey like the fact that everybody was on discord seeing how other people were using it. I think actually to your earlier question really helps with that unreasonableness or ambition where the first time you see somebody tag claude and be like hey you know don't just fix this bug but like now you are responsible for this part of the codebase and I want you to monitor this feedback channel and proactively take on tasks and then fix them and then also take like you know if this API changes do that like I saw somebody do that I was like oh wait I've been totally underutilizing this thing I've just been using it as like a glorified cloud code in Slack like that's definitely uh totally like uh sort new version of it, right? The more advanced version is really trying to start think of it as a teammate that is actually sort of holds context, has memory and can be proactive. And that's just really changed how we operate internally. It's much more like this multiplayer async proactive way than it is a you know most people off in their own CLIs.

Speaker B: >> Are you bottlenecked by code review and

Interviewer: git? Obviously there is cloud review but someone usually still looks at it. Is there a world in which you just merge it in?

Mike Krieger: Yeah, we're it's a really good question. Um, we are definitely still bottlenecked on reviews, especially for things that are like touching some architecture pieces and it's actually more subtle than just being bottlenecked on review because that's, you know, okay, we can carve out time differently. It's like bottlenecked on human ability to even like fully conceptualize what we're doing. So, one of the reasons we built cloud code artifacts that we shipped a couple weeks ago was partially for that, which is uh you would send somebody a PR and then they'd be like, I don't know, man. This is like 2,000 lines of code. Like, it looks like code to me. Um, and what we started doing instead is sharing much more like here's a cloud code artifact. Like, here's the explanation. Here's the intention of the of the change. Here's the trade-offs that were made. And like I think that's going to much more be the trend by which we communicate which is the code is ultimately you know verifiable using some things but actually like discussing intent and trade-offs and then measuring in production is I think the at least the direction of travel we've we've gone I don't review when I get a poll request I wish I could say I've reviewed every line of code I definitely do not I like actually talk to claude about the the code to say all right like these are the questions that I would have can you go investigate it. It is kind of cloudpowered code review but still human driven and and for the really important ones and for the ones that are like cosmetic visual changes it's much more like look like we'll fix forward if we need to fix forward you know

Interviewer: yeah totally um I think a lot of people here are trying to figure that out too um I wanted to talk also a little bit about uh enthropic labs in general um Nai Patel who you've probably met before loves to ask ask the question like draw the org chart like like how like uh people you know you ship your org chart like I I think it's important like everyone knows cloud code now you've got tags um how are you structuring the labs

Mike Krieger: yeah it's a good question because what we were trying to wrestle with was you want sort of people to be supported like you know I think the death of the engineering manager discipline has been greatly exaggerated like I think there's still a lot of coaching and interpersonal pieces and personal development that I think is still really really important but especially in a labs type group where like our whole cadence is two week reviews where every project goes up for we call it persevere or pivot. So basically every project is up for review and either it's time to you know keep going persevering or you know it's time to pivot it or even shut down and you know we shut down projects basically every single one of those cycles and it's like the more you do it the less it's just like oh no my project has shut down I failed. It's like no that is definitely the intention of the labs team is to prototype quickly try to ship internally maybe get it to early access and if it doesn't work wind it down but because of that kind of like rapid iteration it means that if you align the org chart too much to the individual projects you're going to end up like reorging every two weeks which would be a total nightmare. And so we've actually ended up with this interesting setup where like the the pod or the team that is working on a given we call them bets within labs definitely just draws upon like all right somebody from product somebody from the nge team um you know I'll jump in when it's a product I'm particularly interested and I'll come in and work together with the team on it um and that's the unit for that time and there is the concept of a bet lead or a directly responsible individual but the interesting thing is that they don't manage usually any of the other people which kind of breaks the kind of previous way in which a lot of these things were done But I think it leaves lead leaves us to be really flexible when you say okay actually this project is not going to work out. Let's disband and keep going and it's not a big deal and the manager is much more playing the like make sure every individual is assigned to the thing that they're most excited about and that they're working in the best way possible. Now what we do sort of solidify is when there's a product that has like legs like cloud design for example started in this sort of ad hoc sort of group way and then now that like we've shipped it it's gotten traction we've done like a big second release um in June like it's becoming like we've hired people for that specific team and it has more of a of a structure so it's like loose until it gets solidified down the line.

Interviewer: What's the future of cloud design? I think a lot of people are very interested in it's one of your biggest launches this year. Um, where does this go?

Mike Krieger: I think for me, I mean, uh, the things that are holding back Cloud Design for being even better is better interaction with our other surfaces. So, you know, I was designing something or I was talking to to Claude Code the other day. I'm like, I want a really much more seamless like what I'm talking about the design for it, you know, interactive design back to that. I think in general that's I mean this goes back again to uh kind of unconstraining claude like the fact that our services don't talk to each other as well as they could I think really holds back a lot of interesting ideas around what we could do. So I think that's one like kind of major area that we're looking at. Um and then the other one is people like the lines between a cloud design and an app get blurrier and blurrier over time. Like I've seen people of course there's no like persistence but build like fully functional like even games which is definitely not what we designed cloud design for but you can do it this just HTML and JavaScript. Um so blurring those lines even further and thinking through like what is the path from a like fully featured design that looks really well to really good to something that is maybe more like an artifact where you're actually able to go and you know persist data and share it with others and build from there. So I think that those lines get really interesting over time too.

Interviewer: Yeah. Uh, a big part of design is having taste. Um, I actually asked Fable what Fable wants to ask you. Uh, and this this is what Fable came up with. Uh, you deleted almost all of Bourbon to get to Instagram, which is like you had a whole, you know, solomo whatever thing and you went to Instagram. Uh, what would you delete in AI? Or more spicy, what would you delete in Claude?

Mike Krieger: Oh, I like the spice. Um, I think I mean we have it's interesting. We have a uh one of our Slack channels like project unhipped which is like what is in the product right now. It's and and I mean this was hard at Instagram. The Instagram we some things that had like four to 5% usage. You're like oh that's really not very many but then you have like 20 features that each have four to 5% usage. It's like the classic Microsoft Word problem of like uh everybody uses some disjoint subset of the of the functionality. So that that's always the challenge. Now I think uh we're a younger product so hopefully we have less of those things like we unshipped styles I think recently where it was like used by a small percentage of people and was not really AGI pilled in a lot of ways. It was like very sort of prescriptive in the way that it worked and skills were a much better uh application something like that. So I think you have to be willing to take the primitives of like one generation of AI and like unhip them or at least like supplement them or supplant them with the next one as well. I think the biggest thing as I look at it and I've been spending some time like outside the labs on some of this is like man like we're asking people to make like code versus co-work versus like chat distinctions and like one they don't interoperate well and they can't delegate to each other and two I think the average person off the street

Speaker A: [I] could not explain to you why those surfaces are all different. So I think deleting some of the product complexity within our our code or our product I think is a a thing that would would serve well also because then cloud can do what it needs to do and and do well. Like there's nothing more frustrating than having a co-work session where you're like great I've mapped out exactly what I want you to build and then be like can you please like create a paragraph that I can paste into cloud code like that is some 2020 you know kind of workflow there that really shouldn't exist anymore.

Speaker B: Yeah. Um I I think drawing lines on what you don't want to do and also sort of leaving room for others is interesting. Um a lot of people today is like the startup's day for AIE are obviously very sympathetically aligned to startups. Uh but there's some anxiety in the room because tomorrow entropic could wake up and publish some markdown files that destroy my industry. Um so uh why should we not all just give up and join Enthropic? Like why bother starting any other company?

Speaker A: Um I I mean I actually joined one of the main reasons I joined Anthropic was because I saw how much this was like you know the models weren't that good at code yet but they were getting there like how much it would unlock a whole like next generation of startups not because it was going to solve their ideation or their taste but because like it would make experimentation way simpler and and would get you to move faster and I still like really believe that and I mean it's the reality of you know uh and we saw this of like Instagram like we would get questions in investors is like, well, what happens when Google launches a photos product? It's like Google's going to launch a very googly photos product and it's going to have to be bound by the integrations that they already have and it's going to be like it's going to play to their strengths. I think that is going to be true. enough like giving advice on how to compete with Anthropic, I guess, in a way, but like it's actually not because we're also a platform, which is like there's so much I think room to be like laser obsessed with your particular vertical or your industry or group of people that you know really well in a way that like none of the labs are ever going to get to that level of uh of understanding and like therefore get that kind of adoption and user love and and build that up. how it's definitely harder in the age where like the models can just do a lot and so there's you know some of these things can be like skillified and like maybe don't need their own dedicated product but I think it's like the hard stuff is still hard it's like understanding the needs of people like figuring out how you're going to reach them uh listening to them and iterating on them really quickly like it is still the case that like a group of four or five people obsessed with a problem is going to move faster than those same people at any other kind of organization that are like you know subject just to the complexity I just mentioned the like the fact that we have, you know, a lot of different products that kind of interoperate. Like that's a interesting constraint that we have to work through. It's an advantage in other ways, right? So, yeah, I'm still like very long and bullish on startups and um it's just uh it papers over the fact that like writing code was never the like the limiting part. You know, maybe it was on the timeline perspective, but it was never like the thing that was going to like make or break your startup. It's really that space and user understanding.

Speaker B: Yeah. Uh domain knowledge.

Speaker A: Yeah. Uh today is also our day for vertical AI. Uh one of our uh returning speakers and top speakers Chris Lovejoy uh was always talking about vertical AI. He was from interior in the healthcare space and then recently I I invited him back and turned out he you guys just hired him for your healthcare efforts. Um we also our next big one is also finance. Uh you know we have a AI and finance track. You guys just had a huge finance event in New York City. Um and where our next uh AIE is is sort of finance focus. what are you seeing there any you know any potential uh for cloud obviously a lot of excel excel spreadsheets

Speaker B: yeah no I think that there's there's a lot in there too and that's like an area where uh you could see the model get clearly better at it like sort of generation to generation and there's you know there's some good sort of vertical specific uh finance startups that have like done their own um evolves which has also been interesting to to track and it's not like we're like sort of playing to the evol but it is a useful sort of barometer on like is this actually getting better um at these financ use cases. I think the interesting blend that's going to happen um is this mix of again the model having the flexibility to like dive in and create just in time analyses or dashboards or workflows with like some sense of like what is the not immutable but at least like verified sort of set of data and so like uh set having all of that be totally free form I think is a recipe for confusion and is like not what most companies in the financial services space want. So finding that right uh sort of cutline where you have verifiability and audit logging and and sort of data provenence here but not in a way that constrains the kinds of applications that you can build on top I think is a lot of the art that we're seeing in that space as well. Um and I think you know if you solve it well you can you can get the best of both worlds. The hard part is a lot of the systems that were built to do the verifiability out of like are kind of almost by design not super flexible in terms of agentic workloads on top. So I think there's opportunity at both sides of the stack there.

Speaker A: Yeah. Um I think I I also agree we'll be exploring that in in New York. Um the last thing I want to end on is on mental health which we don't talk about enough in technical conferences. Um you've seen a lot of hyperrowth. People are just always refreshing their timelines and it's exhausting. Um how do you advise people who are working 996 to avoid burnout?

Speaker B: Yeah, I mean I think this is a hard one. I mean and it is I'm sure you all are experiencing this because you're all working in this industry like it is you know multiples more intense and things move much more quickly like at Instagram like our two things that we were thinking about was like what is Apple going to announce at WWC and is it going to like totally mess us up or boost us right so that's like once a year um or you know we have competitor launches every three or four months right and uh it is definitely not that is a topic we we do it when we do our weekly all hands it's usually on Wednesdays and we have a slide that's like the week in AI parenthesis and it's only Wednesday and like and inevitably like some competitor has shipped a new model and like there's been like new product and maybe there's some interesting thing happening um uh on the regulation side like things are moving really really quickly. I think the way I try to stay at least relatively sane um one is like actually carving time off and I think the topic co-founders do a good job of like saying like look like burn out if you burn out like you're kind of done and I've seen it happen unfortunately to people I'm really close to and then it takes a long time to recover from that. Um so actually encouraging people like there's no job that is so important that you can't be offline for a couple of days. Um so I think that's like a a big key like piece in there. So like that's strongly believe um and if it is you're probably doing something wrong and you talk to somebody who could be a mentor to figure out how you can uh unblock that. Um and then I think the other one as well is like uh I love sports and like uh this is the notion of like you're never as good as like your best game and you're never as bad as your worst game. I think that's also really true. Like I know like in AI there's like the you know it's so over we're so back thing. Like that like if you

Mike: internalize that that cycle is always going to be at play in some way. You realize like it's never that bad. Like Ben Horowitz's uh book is the hard thing about hard things uh has this chapter on like we're effed it's over and like that feeling as a startup that probably many of you have had at startups where you're like oh I can't believe this thing happened. Like we're never going to like recover from this. I defin we definitely had it on Instagram a couple of times and then you get through it and like it's that like def like defines the company when you can actually go through that. I try to remind myself and the team here even within anthropic which is like look this is a it's a fastmoving but it's also a long game and it's like we're never it's never just about today's model launch and reaction or this product launch or something else like you're playing and you're building. You just have to trust that you're building like the team and culture that is going to get through those things and have that sense of perspective. Even if perspective is saying like look three months ago we were in the similar position. Maybe it's not a year it's just a matter of months but it's still like zooming out and not thinking things not letting your internal sort of like sense of self and success be so driven by the dayto-day.

Interviewer: Yeah. Has anyone any coach or mentor said something to you that you repeat to yourself uh that gets you through the tough times?

Mike: Um, I think the biggest one was uh this like uh sense of like uh if you're feeling something, it's really often the case that other people on the team are feeling it too. So, this is like advice I got from my my coach around just being like like just verbalizing emotions. Like even saying like, "Hey, I'm feeling really stressed out about this." Or, "Yeah, I'm really sad that we are shutting down this labs initi." I literally had this meeting a couple months ago where I was working really hard on something and I kicked off the meeting like I I'll kick it off like I'm really sad like and frustrated like I wish this thing had worked out and I think that holds the space for other people to be like yeah I'm pissed off too or like I'm sad too and like uh I think giving that advice around like not uh yeah I think if you can get yourself to be open and vulnerable it often like lets other people verbalize that and then you can from there you can be like great what are we going to do about it like you know it's much easier to start from that place. Yeah, we actually kicked off AIE with a session from Carol Robbins who runs Touchify at Stanford. Uh, and I can't think of a better way to end than encouraging people to talk about their feelings, manage their mental health, and keep shipping.

Interviewer: Yeah. Thanks so much, Mike.

Mike: Thanks for having me.

Announcer: Ladies and gentlemen, please welcome back to the stage our MC developer relations engineer at Replet, Ralph Shabri.

Ralph Shabri: All right, let's hear it one more time for our keynote speakers. All right. Okay, so we learned so much this morning, right? like we spoke about a new protocol and and we spoke about like that AI program should be more like functions and we learned that uh the real problem with building with agents is not the model but it's about everything around it. Um but before we go to breakouts guys I would like to announce our next speaker. Um so a quick word from our sponsor Neo4j. Anybody uses graphs here? All right, fair amount. Fair amount. Okay, cool. So, our next speaker is going to talk to you about ontologybased semantic layers. When you hear ontology and semantic in the same sentence, you know that you're up to up for something hot. All right, so uh without further ado, please join me in welcoming to the stage CEO of Neo4j, Emil Efim.

Announcer: Please welcome to the stage the founder and CEO at Neo4j, Emil.

Emil Eifrem: All right. At Neo Forj, we work with some of the largest companies on the world to help make their data ready for AI agents. And today I want to talk to you about a problem that we saw emerging over the last call it six to nine months and propose a solution blueprint for that. So let's say that we work at a big organization, a big bank and we want to write an agent. Let's say that agent is helping automate the opening of a bank account, right? You can imagine that's very ripe for automation. You want to be able to orchestrate that process. And I'm going to use the powers bestowed upon me by a short keynote slot to grossly simplify what that agent looks like. I'm going to say there's two pieces. The first one is let's call it the business logic. Some version of interpreting intent and plan act and we loop around that. It's what your agent does. And we know that when an agent act, it doesn't always operate on data. But we equally know that in order for agents to be successful a huge part of that is giving it access to the right data at the right time. So the second big bucket is let's call it the data sources need to identify figure out okay in order to solve my problem I need access to these few things and wire them up and make them available to the agent. In the example of our account opening agent maybe we can imagine that we need to be able to validate identity. And so we might look at two data sources for that. the Department of Motor Vehicles, the DMV registry, and maybe some kind of passport verification service. So, we wire that up into our agent and it works. It's great. It's fantastic. And at the same time, you and other teams in your organization are building other agents and conceptually they look very similar. So, that's great. It's fantastic. It works. But it has a few problems. So first of all, every single time a team has to build an agent, they have to figure out from scratch where the data that they require for that agent to operate, where it sits, which if you work at a startup and you have one application, it sits on top of one Postgress database. That's not hard. The data is in that Postgress database. But in an enterprise ecosystem, you don't have one database. You have a 100 databases and you have Snowflake and data bricks probably and you have S3 buckets and so on and so forth. You have to do that work manually from scratch every single time. And then when you found the data sources, you know, in an enterprise there's lots of duplication of data. So then you need to figure out like is this the right data? Is it the right version? Can I trust it? Am I allowed to access it? So on and so forth. It also violates one of the core principles of software engineering, the dry principle. Don't repeat yourself. So when something change that cascades across all of your agents you have to kind of manually rewire all of them all the time which works but it's just a lot of work and then finally there's no learning around the data sources and how your agents operate on them. So when your agent wake up wakes up tomorrow it's not smarter than it was today. And there certainly isn't any cross agent learning because all of that wiring between business intent and the data sources is encoded in a combination of code and prompts. So I know what you're all thinking. Markdown files skills to the rescue and yes and no. Um you can come talk to me afterwards for kind of the full version of this but we've seen a ton of team that try to solve this problem using just markdown files. And the summary is it is part of the solution but it is not the solution. Uh but don't take it from me take it from SW. A week ago on the latent space podcast he said hey guys you got to learn your databases. You cannot vibe code with just markdown files. So, we've been solving this problem at scale for some really massive organizations recently, including a Fortune 20 global bank, a massive tech platform company based here in the Bay Area, and a leading fintech company. And the pattern that is emerging is that in order to do agents at scale, we need thin agents on a smarter shared substrate. thin agents on a smarter shared substrate. And what does that look like in practice? There are three pillars to that. The first pillar is a businessfacing ontology. And the word ontology, like I grew up in this world, people talked about ontologies

Speaker A: forever. More recently, it's become very hype, probably thanks to Palantir, but also the rise of AI. And there's a lot of people who want to make ontologies really complex, but the core concepts are actually super simple. What are the key concepts in your organization? In our banking example, customers, accounts, debit cards, checks, transactions, and how do they all relate? But very importantly, they are expressed in a way that makes sense to all the human beings working in your universe, right? All the people working in your company, it's expressed in that name. In that way, in other words, you don't say f_name. No, you have a customer and they have a first name. So that's the first a business facing ontology.

Speaker A: The second pillar is a technical ontology. This is all the metadata of all the data sources and data assets in your enterprise ecosystem. I have 14 Oracle databases. I have 15 Neo4j databases. I have snowflake and data bricks and I have S3 buckets and all of that kind of stuff. Where do they sit? What are the schemas? All of that kind of good stuff. You construct that technical ontology in three key ways that we can talk about later though not in this talk and then you have a mapping between the two. So that customer that has a first name, that first name has a system of record and over there there's an Oracle database with a column called F_ename the mapping between the two.

Speaker A: And then the third pillar is the runtime signals out of your agents when they walk this graph and they execute they leave the traces around what have I tried? Was I successful? What was the outcome? The execution traces those three pillars. Okay. So let's look at that in the context of our bank account opening agent. This is a simplified view but you can see this graph here. It has a combination of business concept like checks and accounts and credit history and stuff like that. This is a process following agent or a process guided agent. We want this type of agent to actually follow a process. So we've also encoded that in the ontology a business process.

Speaker A: And then if you look at the node that is surrounded by green the check compliance one we flipped to the technical ontology and we've put in the graph here we've discovered and encoded that in order to do a compliance check you might imagine that you need to resolve a government-issued ID and then we say that in this particular organization there are two data sources that can help us with that. It's the motor vehicle records and the passport verification one. Okay so that's really great. So then when our agents come in here and they realize I'm going to check compliance, I need a government-issued ID. Here are the two ways that I can resolve that. When they execute and they try that, they leave the third pillar, the execution traces for that. And they're more sophisticated than what's on this simplified slide, but involves things like, okay, where was I? What did I do? What is my context? And was I successful? And ultimately, it leads out to some kind of a score. and you use that as input. It's like, okay, I've been very successful using the DMV lookup for example, then I'm more likely to choose one if I'm in the right context in my next invocation.

Speaker A: Three pillars of the ontology based semantic layer, a business ontology, a technical ontology, the execution traces taken together, they solve all four of the problems. We now have a very easy way to discover the data sources. We know if they're trustworthy or not. We know that top down by some kind of human curated knowledge, right? An administrator of some sort saying it. We also know it bottom up through the execution traces. This is what actually worked in reality in practice. We have a single govern place that maps business intent and the concepts to those data sources so we don't repeat ourselves. If something changes that cascades across all my agents, right? And we have self-learning. So my agent that wakes up tomorrow is slightly smarter than it was today. And not just self-learning on an individual agent, but across agents as well. So we're moving from this world, a world of thick agents with manually wired data sources into this world where we have thin agents on a smarter shared ontology based semantic layer. And this allows us to do a ton more agents without having to re-engineer them every time. Thin agents on top of a smarter shared substrate.

Speaker A: If you think this is interesting, there's a documentation, a web page that outlines more information about this. If you see the QR code here, you can also come and talk to us at the booth. We have a big booth here at the expo P3. We love talking about this kind of stuff. But not just that, this is one pattern, a very exciting pattern that we see a lot of traction around right now for using graphs in AI. But there's hundreds of more interesting patterns that combines graphs and AI. 10 of them is actually in the graph track that is kicking off right now in room 2005. And you have some really amazing talks from organizations like the Gates Foundation, Monday.com, JP Morgan Chase, Berkeley, New York Times, and so on and so forth. So go check out that thing.

Speaker A: And then finally, this was primarily centered around organizations where you deal with many data sources and many agents. But if you're a startup building on Neo4j, love you. There is a startup program for Neo4j that is phenomenal. You get access to free credits, but more importantly, we've built up a dedicated solution engineering team that spend every day working with startups for free, helping them model their data in Neo4j, tune it for performance, and so on and so forth. So, please sign up for our startup program. Thank you very much. Enjoy the conference. Have a good day, everyone.

Ana Marie Benzson: A production data job failed hours ago. The dashboard went stale. You have spent all day checking the logs, the schema and the upstream data. And now it is past midnight. The same question keeps coming back. What changed? The failure itself may be small but expensive part is everything around it. Inspection, diagnosis, choosing a safe response, rerunning the job, and confirming that we did not make the data worse. Hi, I'm Ana Marie Benzson. In this talk, I will show an RLG guided system that selects bounded remediation action for ETL failures. The central question is not simply whether an agent can act, but whether it can act usefully, explainably, and within boundaries that an operations team would actually trust.

Ana Marie Benzson: Cloud Detail failures are rarely arrived as one clean well-labeled exemption. We see late or unavailable sources, schema drift, datetime incompatibilities, now rate spikes, type changes, and runtime errors that do not match anything in the runbook. The usual response is a human workflow. Inspect the logs, form a diagnosis, a temporary pair, rerun the job, and validate the output. Each step is reasonable. The latency comes from headoffs, incomplete context, and the need to avoid an unsafe fix. In the capstone evaluation, the manual recovery baseline was modeled at roughly 2.5 working days. This represents an incident moving through normal queing, investigation, and approval. So, the engineering objective is specific. compress that loop for routine recognizable failures while escalating the cases that are uncertain, novel or high risk.

Ana Marie Benzson: This diagram shows the end to end AWS architecture from my capstone. An existing AWS glue job emits a job failed event. Amazon event bridge catches that event and triggers the Lambda function that runs the agent. Lambda gathers evidence from two readonly sources. Cloudwatch provides the error logs while the glue data catalog provides the current schema metadata. The system uses those signals to classify the failure, assess the data quality and operational risk and construct the state passed to the RL decision engine. The policy then proposes a bounded response. The safety layer checks that proposal before the executive can use the glue API to retrigger the job or apply an approved remediation. Amazon S3 stores agent artifacts, audit logs and quarantined outputs. Finally, the job is rerun and validated.

Ana Marie Benzson: So this is close operational look. Monitor, diagnose, score, decide, check safety, act and verify recovery. The capstone implementation use synthetic data provided by the client. The public

Speaker A: repository preserve this pattern through a sanitized generalized deployment template. The intelligence layer deliberately separates three concerns. Deterministic anomaly rules establish observable facts. A field disappeared, a type change or the null rate cross a threshold. The Q-learning policy handles contextual action selection given the current in incident state should the system retry coers the schema roll back quarantine escalate or simply log the event. Then a safety override sits outside the learned policy. For example, if the anomaly is critical and the policy proposes a passive action such as logging, the override converts that choice into an escalation. This separation is the design thesis of the project, rules for facts, learning for bounded choices, and guard rails for authority. Before selecting an action, the system has to establish what actually happened. The schema profiler extracts a structure types nestic and null rate statistics. The drift detector compares the current profiler with a baseline and identifies additions, removals, and type changes. The data quality analyzer checks completeness, validity, and consistency. The error classifier maps log patterns into failure families, and the risk scorer turns those signals into an operational risk level. These components are deterministic by design for directly observable data conditions. An explicit rule is easier to validate, explain and audit than an opaque inference. With richer and representative incident history, some classifiers could become learned components. But ML ready is not the same as ML required. The simplest reliable component should own each decision. The policy receives a compact state failure category, risk level, retry count, drift severity and data quality condition. It then selects from six actions. Retry, covers, roll back, quarantine, escalate or log. I use tabular Q-learning because the state and action spaces are small. The Q table is cheap to evaluate and every decision can be inspected directly. For this state, these were action values and this action one. Technically, each incident is modeled as single.

Caitlyn: Good morning. We're super excited to be here at AI Engineer with all of you. I'm Caitlyn and I lead platform engineering at Anthropic.

Angela: And I'm Angela. I lead platform product at Anthropic. And today we want to talk to you about a concept that we've been spending a lot of time thinking about and working on with our team, which is this idea that we think that token should have jobs.

Angela: So, if you're building an agentic system and you're trying to accomplish some specific outcome, you're trying to get something done with agents, there's one lever that everybody pulls in order to get a better outcome, and that's usually increasing your budget, which means you spend more tokens or you spend more expensive tokens. But we've been wondering, is that all there is? Underlying this assumption of uh using the budget is this kind of implicit perspective that every single token is basically fungeible. And we've been wondering is that actually true? Are all these tokens actually fungeible? And to test that we've been thinking what if we gave tokens jobs? So if you think about the way that you would normally set up an agent to go accomplish a task, you give it that task, you give it this token budget, and then all the tokens that are being spent are basically indiscriminate in the sense that they're all doing one job. they're just executing. But what if you take some of those tokens and they're not just executing, they're doing some other job.

Angela: So for example, maybe you take some of your tokens and they're advising the tokens that are executing. Or maybe the tokens that are executing try to get something done well and you take some other tokens and you actually grade how well the executor is doing so that it can iterate and try again. Or maybe you have tokens that are dreaming. They're reflecting back on the job that other executors have done and writing learnings to memory so that they can do it again. And what we call each of these, if you take some tokens that are executing and some tokens that are doing some other job, let's call this a strategy. So let's go take a look at the first strategy, the advising strategy. Here we're splitting up an executor and an adviser. The executor obviously executes, but crucially they can call out to adviser for advice. and then they can take this device and figure out if they're doing the next step correctly. This is really helpful in use cases. For example, if you're building a sales agent, in an ideal world, you'd have that sales agent be able to actually help the sales rep flag when a follow-up is overdue or deal is stalling. In this construct, having an adviser to be able to kind of make sure that all the different pieces are actually working is really helpful.

Angela: So, another example is grading. Let's say you're executing and you kind of know exactly what good really does look like. You can define this in a rubric and then each time an executor tries to accomplish that outcome, you can have a grader provisioned that grades how well the executor did while looking at that rubric. And if the executor did a good job, then great, it can be done. But if it didn't do such a great job, you can iterate again until you get that good outcome. So, an example in practice of when you might want to use this is let's say you have a customer service agent and you're running a store and your customers are writing in and they're saying, "Ah, I should get a refund for this thing." And your customer service agent needs to be able to respond. You probably have some like pretty specific criteria on when you would give somebody a refund. And so, what you can do is define a rubric that uses that criteria. You can have a grader that goes and looks at the work that the customer service agent is doing and decide is it getting it right and is it coming to the right outcome.

Angela: And the last strategy we have is dreaming. So in dreaming there's an executor who naturally executes and then there's a dreamer. The dreamer is actually able to inspect the work and the transcripts of the executor and then it takes any of the findings that it has and it writes them to memory. This memory is repicked up by the executor for the next round. So ideally would have improved. A great use case for this is if you're building a recruiting agent. Now recruiting requires a lot of interaction with feedback on whether or not a candidate does or doesn't make sense and if it's a good fit between both parties. And so by taking all this type of data, if you build a dreaming type of strategy on this agent, it's actually able to kind of sharpen the next round so that it's more and more increasingly useful.

Angela: So let's make this concrete with some experiments. So what we did was we created a bench of a bunch of tasks related to financial an analysis. And what we were doing with each of these tasks is trying to replicate in the real world a expert human financial analyst. How well would they do on each of these various tasks? And so what we did was we start with a control that's just executing. Let's try each of these tasks and we'll eval them and we're literally just executing. But then we can experiment with each of our strategies and see how well we perform. So, we start with a super basic experiment. Let's just one-shot it. Let's take each of our strategies and we'll go and just make an attempt to accomplish these tasks and we'll see how accurate we are. And so, you can see here with executing um it didn't do so well. 15% accuracy, but because it was just a oneshot, the strategy got to choose how many tokens it would actually spend on its own. And so, you can see that execute decided not to spend that many tokens, only 39,000. And as we go into our larger strategies, our more complex strategies, we did choose to spend more tokens, but we did a better job. So, this isn't really telling us much because sure, Drain did really, really well, but it used a whopping 600,000 tokens to get there.

Caitlyn: That's right. So, in order to actually figure out if varying the jobs produces any alpha, what we need to do is hold the budget constant. And to do this, we're going to take Dreaming's budget, that 600,000 or so, as the maximum budget that is fixed across the board. And

Speaker A: we give every single strategy this budget in order to analyze how well it's performing. And as expected again, if you give a lot of strategies more budget, you are going to see performance increase across the board. So execute went from 0.15 to 76. Advise and grade went from the 60s to closer to the 90s. And that's again expected given the fact that if you give things more test time compute, they should generally perform better. But if that was the only thing that mattered, we should actually expect to see execute, advise, grade, dream actually all be at the exact same level given the exact same token budget. But what we're actually seeing is that there is an alpha or there is a difference and therefore an alpha for us to exploit. If you look at execute at those exact same budget level, it gets to 76, but advise is at 89. So while a minimal, it does exist and so there is alpha for us to take a look at.

Speaker A: Now, we decided to take a look at this analysis from a completely different lens. And as Kayla mentioned, you know, we're doing this bench for a very complex set of financial tasks in the real world. And we wanted to analyze the usage of agents with actual experts. So, if we look at a financial analyst expert, right, the kind of task that they need to do with an agent is that they're giving it something very concrete like let's say make a P&L and then they're getting the result back. Now if that result is 80% accurate on a bench that sounds great but in reality what that means for that expert is they have to go back and recompute that P&L themselves or alternatively send it through another run. And that's because in this kind of domain for this kind of task if you're not 100% accurate it's actually not useful. You cannot make up an income number or you can't make up a cost number right you have to make sure that it's 100% accurate. So with this lens of the real world consequence associated with this domain, we needed to recompute our experiments and score them a bit differently. Crucially, we needed to make sure that our experiments had this kind of construct where if it was scored perfectly, we'd actually give it a pass. And if it scored anything less than 100% on that kind of task, we would actually mark it as a failure.

Speaker A: So let's look at a different cut of our data from our experiments with this lens where we're looking for this perfect run, 100% accuracy pass. And let's look at what percent of the time each of these strategies was able to achieve a pass. Um so we've got executes um down at 42% and we've got our more complex strategies doing a bit better up to 75% accuracy. Um and again this doesn't necessarily tell us a ton because um you know each of these strategies might um choose to use different budgets over time, right? So what we did here was we fixed the budget and we said within a fixed budget, how well do each of these strategies perform? And so what really matters to us actually is if you're trying to get this perfect answer and you're in the real world, you're running a business, what matters to you is the cost to you to get to that perfect answer. And so one way we can think about this is we had our execute strategy for example. The execute strategy around 40% of the time will give you that perfect answer. So on average, you can expect to have to run it three times and you should hopefully sometime in those three runs get a perfect answer. And as we talked about earlier, we fixed our budget to that highest token budget strategy, which was 600,000 tokens. So if you spend 600,000 tokens in each individual run, you have to run approximately three times. You can expect on average to have to spend 1.8 million tokens with the execution strategy to get to your perfect answer.

Speaker A: And so if we take this analysis and run it across the board against all these strategies, this is actually the true cost it took in this domain for that agent to be useful for that strategy. So as Caitlyn mentioned for execute, which is our baseline, this is going to be 1.8 million true total token cost for you. But advise, grade, and dream are showing us a bit of difference. Crucially, advise and grade are actually quite token efficient when you think about the actual usage of the end output of each of these agents.

Speaker A: So what does this mean for you as a business? Well, it actually really depends on what kind of thing you want to optimize for and it's going to vary, right? There's going to be businesses who say, "Actually, for me, the most important thing is to be really token efficient. In that case, you should probably pick the advised type of strategy in order to solve for that particular domain in which you want to optimize that." There's going to be other areas or other businesses where you're going to say, "I'm not going to care so much about token efficiency because what I really care about is reliability of that answer and so I need to maximize the percentage of runs in which I get that perfect answer." In which case, you would actually pick completely different strategies. You probably lean towards grade or dream.

Speaker A: So, if you take away one thing, the thing we want everyone to think about is this idea that tokens are not fungible. You can use your tokens to execute. You can brute force your way through your tasks and you can throw more budget at it. But if you get really smart about having your tokens do these different jobs and try these different strategies, you're very very likely to be able to get a better outcome for the task at hand within a fixed budget.

Speaker A: And so let's talk a little bit about how we actually build strategies and how we bring this to life. Um so we've done a lot of work to create a really excellent harness for individual agents. Um and if you see this uh picture at the bottom here, this is actually the architecture that we've used for cloud managed agents um which is our agentic solution that we give to you within the cloud platform. And what we do on top of this is we start to get into the meta harness level like the multi-agent orchestration and execution level where this strategy can go and be um coordinated between our executor and our adviser or the other agents within our strategy. And some of these um like dreaming and outcomes we actually give to you out of the box within cloud managed agents. So with those set of primitives it's actually relatively trivial for us to construct this kind of you know architecture where we're able to combine these different types of strategies and figure out how to they should work together.

Speaker A: So for example it's relatively trivial for us to say okay now with this I can take a task and I should be able to execute it but also allow it to advise and fable is back online. So we could actually say fable is the one that's actually advising uh the executor. And then I can take all these results and say send them to a grader so that I can make sure that this is verifying in a loop that makes sense. And if it passes, that's awesome. I want to send all of that stuff to Dreaming and make sure that my next run is better than ever. And of course, you don't have to stop there, right? If the right primitives are there and the right coordination is there, then you can actually construct really complex setups that fit for all the different types of dynamic problems that you have. You can invent these kinds of large-scale architectures. Again, very trivy. And you could also invent completely new jobs, not just the ones of the pieces that Caitlyn and I have presented in this conversation.

Speaker A: So, a big goal that we have over time is to get our models better and better and our platform better and better at dynamically constructing these strategies for you as you're doing work. But in the meantime, as we're working our way there, we would love for you to continue to think about this idea that you should give your tokens jobs and you should use different novel strategies by combining these primitives in order to get the outcomes that you want for your tasks.

Speaker B: >> Thanks for joining us. turning my second brain into my living research memory. Let me explain. So, within my second brain, I currently have over 5,000 notes in Obsidian and another 5,000 notes in Readwise and some scattered

Paul: in Notion and Google Drive. And all of this is growing on average with 250 files per month. And this is what I want. On the left, you can see my whole obsidian vault, this huge mess. And whenever I start working on something such as an article, a new project, a new codebase, a new feature or whatever, I want to actually pull high signal nodes that are actually useful for my current work. And you would ask yourself, why not use directly calledex code or notebook LM? And the thing is that I am, but you need a system that sits between those harnesses and your second brain. Okay, so let's go back to the root of my problem, which is that I'm always losing my research. For example, my reading list is a graveyard. When I'm scrolling social media and I have that cool expost, a new article, a new new YouTube video, a GitHub repository, it doesn't matter. Whenever I actually want to start working on something, I never recall what I have in my second brain or I have to spend a ton of time actually finding meaningful notes that I can use in my work, right? And another problem that I have is that I want this system to actually be anchored into my personal notes, into my personal values, into my personal faith. I want this system to be personal, to reflect my own thoughts, right? And that's why in today's video, Luis Franuis and I will teach you how to build your own AI research OS. This also comes with code, so you can also try it out yourself. And I'm Pauline. I'm the founder and CEO of Decoding AI where I do a ton of content on courses on how to ship air products and I'm also the co-author of the LM engineers handbook bestseller and the system the air research OS that I will teach you in this video is the system that I use in my daily work and now I will pass the torch to Luis Francois.

Louis-François: Thanks Paul. So I'm Lu France Bush. I'm the co-founder and CTO of Towards AAI where we build educational courses and I'm also the creator of what's AI a YouTube channel where I explain AI engineering techniques. I used to explain AI research before now focusing on AI engineering. I'm also the author of the book building for production and before that I was a PhD student. So I honestly make research for a living. I used to do a PhD as I said in AI and doing tons of research and research work. Now I build courses, I write videos, I research for videos, I build trainings for companies for a living. And all of these things that I do start with a very good research and also leveraging tons of knowledge and insights that we get at towards the eye from building for clients. So I have tons of nodes as well just like Paul and we try to leverage them the best possible and as you'll see we build some sort of tool to leverage our second brain where as you'll see there will be some differences between how I use it and how Paul uses it and that's the core goal of the repository that we built and on this project is that we want you to adapt it for your needs. The whole goal is how can we make research better? But more specifically, how can we better leverage what we have? So, let's dive into it. And first, we need to figure out which tool to use and when. Because this whole research system that we built is not for every query. If you just need a fast answer like a few quick questions or just something where that that you would just Google basically well obviously just Google it or ask JGBD cloud whichever system you want but the problem when doing that is that if you have a lot of following up question or it's a bigger project that you need to build on uh and have basically a very long context or tons of information to share relying on chibbity isn't ideal. And it also means that you are fully dependent on the architecture that openai or trag. So the next step here is to ask yourself for a more complex problem. Do you need to act quickly or do you want to build some next feature and and do something very difficult? If you just have a small repo for a quick change or write one article, just do one thing that you know won't be repeatable that much, definitely use codeex or cloud code or some agent that you trust. Sometimes you need to keep on digging to make it better, to improve efficiency, may optimize it more. And so typically when you have to do that, you want your research sources, your research to stick and to be able to refer to them in the future. So if you want a process like this where the sources that you find, the notes that you take stick around in time and have an agent be able to leverage that efficiently and being able to come back to these information to ask follow-up questions to digest content even more. And right now for instance when I make a new video I want also the agent of the system to understand the previous videos I made to not duplicate content to not repeat myself and to refer to some other content. In this case there are some tools that are very interesting that you might have tried before like notebookm that is super powerful to do research to digest content efficiently and to come back to it. But the problem with notebook is that it's well first the main problem is that you don't own it. You cannot do anything you want with it. You cannot personalize as much as possible. It's not agent native and it's obviously weak for coding tasks since it's just browser based. So it's far from ideal from something that Paul and I needed and that most AI engineers need in general. So if you need your agents to be able to leverage all you do, uh whether it is a big research, a new video, whatever you write, you do, you code, you typically want your other agents, your other projects to be able to leverage what you learn from what you just did. And one thing that we advise, especially for production, obviously for a product, is to build some sort of retrieval rack pipeline with vector databases. But this needs an infrastructure. It's not really human friendly to be able to digest quickly, to check notes, to make edits. It's hard to inspect by hand. You need to build everything around it. It's definitely far from ideal for just something I want to use on a daily basis. Obviously, it's super powerful at scale, very interesting, especially in a product. But as I said, this project is for us, and I don't want something live, super professional as a product. I just want something I will use and that my agents and different projects can leverage as best as possible. So the last question to ask ourselves here is that if you want everything there but more personalization. So, a personalized research assistant that builds some sort of Wikipedia that compounds over time and it and is easily inspectable and usable where you have tons of sources, documents, videos, comparisons, implementations, new research, new topics that you keep on adding and that you keep on wanting to leverage and review easily. This is where you may want to build something yourself. And in our case, we built the personalized research OS that we will share in this talk with exactly what we built and how. But the downside is that it definitely needs a bit more setup than just opening cloud code. Right now, the main problem with using cloud code and other agentic tool is that you give codec links, PDFs, and different information. For example, my most recent loop engineering video and then the next session you use codeex, you have to paste it all again or ask it to use skills and whatever structure that codeex or jgpd whatever tool that you use built on the fly to leverage what you did, the scripts it ran, the scripts it had, you all lost it or kept it inside a skill that you have to ask it to reuse and that usually isn't ideal and just grows and grows over And the problem is that all this information that you give to the model is not the bottleneck. The bottleneck is how can you leverage it in the future. Meaning that with an agent, the context window becomes everything. The database, the file system, the memory, the reasoning space. It has to do it all. And when you stop the conversation, it loses everything. And the thing is that we don't need necessarily to provide more and more and more context for a better research. you need a proper memory and context

Speaker A: management and ideally some personality with it especially in my case when I do videos. So what we did is that we decided to build a system with plain files mostly markdown files that we can leverage easily and that agents can leverage easily. I won't detail it very much here because Paul will talk about it in depth and as I said Paul has like 5,000 or something notes. I have just a few hundred. But that's just to say that we need to consider that we didn't start from nothing. We already both had some sort of large database. In my case, I made hundreds of videos and I take many notes. So, I still need to leverage these years of content that I already made and tons of meetings that I have with uh my team, with clients when we build for them that I want to leverage as well because we learn a lot by building for people.

Nikita Kotari: Um, welcome everyone. Um, I'm Nikita Kotari. I'm a senior member of technical staff at Salesforce. Uh, where I'm building AIdriven enterprise solutions. If you might have heard of agent force, agent force for setup, headless 360, I worked on all and it's like amazing features. Um, I also worked at Amazon and LinkedIn previously. Um, in last 6 months or a year we completely changed the way we used to work. We are building new systems. We are evolving and there are some critical uh aspect of using MCP CLI and skill. So today we are uh going to deep dive into a critical yet often overlook aspect of building AI agents. It's plumbing. As a developer we talk a lot about model sizes, reasoning capability and the prompt engineering. But when when it comes to building production ready AI agents, the way an agent interacts with the environment, it's tooling layer. It's what actually makes or breaks the application. We will look into three distinct approaches which is MCP, CLI and the structured skill and map a clear rubric for what to use when. Uh at the end of this talk, you'll have a clear mental model for choosing between these three tooling layers.

Nikita Kotari: Um, here's something every team discovers the hard way. Building demo agent takes like an afternoon. But when you actually want to ship the same thing into the production, it could take a quarter because we are looking to build the reliable, secure and fast agents. Nobody wants to leak their customers data into the production. So testing uh reliability is still a major issues with this AI agents. So uh I want to talk about these three problems. The problem one is the context explosion. Uh consider a scenario where you load a 50 tool schemas into a agent's context window and suddenly 60% of your thinking space is burnt before even this task even starts. Your agent forgots the diff because uh tool definition for monitoring databases and the deployments file the window. It literally doesn't have a room to think.

Nikita Kotari: Uh the second yet critical problem is invisible failures. The agent made a like consider example where agent made a seven tool calls to draft a PR. Something went wrong. The PR was created on the different branch. Um so first you lost like lot of time to redo the work and the second thing is somehow you lost the confidence on your agent to perform a particular task and the problem three is the security surf. Uh consider example at 2 am you got a customer issue you are investigate getting a customer A's error but wait LLM has a broad access to the query and the tenant data and it accidentally started looking into customer B's data so that is a multi-tenant platform issue and it's not a just a bug it's a huge compliance nightmare so let's talk about how we can use this three um two uh three layers to build the reliable help fast and the secure um agent agents.

Nikita Kotari: Um so we have the CLI. CLI is like handling someone a screwdriver. Here is a tool just run it. Direct execution. The agents run a command, gets output and moves on. Then we have a layer two which is MCP. MCP stands for model context protocol. This is just like giving someone a USBC hub. Here is a universal adapter that works with any services. Just use this and work on it. And third we have the skill. The skill is giving someone a runbook. Here is a step-by-step execution of this task. Just go and perform it. So it's structured and it could work with any of the platform or any of the system. So CLI knows how to do they execute command directly. MCP knows what is available and skill knows how to do it. So now let's dive deeper into one of each.

Nikita Kotari: CLI they are readable. Any engineer can look at the command and see what's exactly happened. They are reproducible. If something fails at 2 am, you can copy paste the same command, run it on your machine and you can reproduce the same error. And they're composible. You can pipe multiple commands into one and you can run. The heristic is simple. If I want to want you to remember this, if a engineer on your team can open a terminal to do this, the agent should probably able to do the same thing. we have the structural input and structured output. Um, and it's it's highly highly uh usable. I think we have been using the CLI for more than 50 years now. So it's it's not glamorous but they are proven and it works.

Nikita Kotari: Then we have the MCP. MCP is consider it as a universal C adapter like um every device has its own charger but the universal uni USBC is standardized where you can connect any device to your um connector. So on one side you can see you have the clot, RGPD and the coding tool and the other side you can connect the different services like work item tracker, code search, error tracking, monitoring maybe real life examples could be uh get slack um and Google and everything. So uh at the bottom you see uh what the what this looks in practice. The agent calls the code search dot search with a query and it gets a structural result back. So it does the it handles the indexing across the millions of files and enforces the tenant isolation at the server level. This is how you solve the security problem too because every time you will connect the external services via MCP protocol there is a O and you will give uh permission to access those services.

Nikita Kotari: Third thing is the skill. This is one of my favorite uh tool and I think I have automated all of my work that I'm doing right now at my workplace with the skill because it's highly reliable. It's fast and you can tweak it it in a lot of different way. Um skill is a structural playbook that wraps around your tools. It defines the rigid orchestration. When a condition is met, ensure the this prerequisite is exist and then execute step one, step two or step three sequentially using the CLI, MCP and explicitly handle um handle those edge cases. It prevents the guesswork, it saves the memory and it's standardizable like you you only choose like I only want like two MCPS to connect to this skill. So you don't have to go and look into all of the MCPS to perform a particular task.

Nikita Kotari: Now um let's talk about the problem that we discussed earlier. I have a 50 uh MCP um tools connected to my system. So before running any task, my 15,000 to 20,000 tokens are already burned and I lo I lost like lot of context before even starting my agent. Uh so you can see on the one side you have like a whole lot of um tokens and how you could replace the same thing by using the skills. So keeping the 55 tools from the five different MCP servers active at all times that cost us it caused the latency issue platform issues you can just simply define the number of MCP servers that that will needed to execute that particular task.

Nikita Kotari: So here are like some of the examples like fixing a test failure. So you don't need to go into all of your MCP tool to just to fix one uh test failure. You can only look into the logs. You can look into the release log or and GitHub to fix your test test failure. And for the draft PR and the investigation escalation, let me dive a little bit deeper into it. So consider this slide where we are um drafting a pull request. So what do you need? um you need like um the first agent records the draft PR skills. Step one requires the understanding of the local changes instead of heavy API. It hits the fast scope CLI commands and then you you need a git command to create a uh get pull

Speaker A: ...request on the correct branch. So you can literally write down the steps. You can create a skill and you can keep modifying it to achieve the best results.

Speaker A: Then again another thing is the investigation a bug. Um if midnight you get a any customer bug I mean the first thing you're going to do is like looking into the logs uh searching for the error pattern looking into the uh recent releases if there is a bug in the code or things like that. So you already know that like what you are going to do. So you can automate everything with limited number of MCP tools and the CLI to make a world-class skill to debug your customers issue.

Speaker A: And now let's talk about the security. A golden rule of the agentic development is enforce the isolation in your infrastructure never in your prompts. So prompt can be injected but the infrastructure cannot. For CLI the blast radius is full shell access. Uh so you mitigate um mitigation must include the sandboxing and the strict containerization for MCPS. The blast radius is limited to the explicitly exposing the tools and the tenant tenant isolation and that guarantees at the server level and we have some of the principles that we can always think about while building any agent that like we shouldn't be sharing any credentials with our agent.

Speaker A: The tenant isolation is the infrastructure enforced. We can always use the permission gate um gets to um operate at the security level and uh we can provide the least privilege while performing any task to the agent.

Speaker A: Now when when you are deciding which layer to use like every time you cannot create a skill or every time you cannot just explode to use all of the MCP tools. So you need to ask these three question in order to get the best results. The first one is like who else needs this. If it's just this agent just then use a CLI. If the multiple agents are going to use this service then use MCP. If multiple workflow shares the same procedure then use a skill.

Speaker A: The second question you can ask yourself is like what is the failure mode that matters the most. If it's a transparency and the reproducibility go for the CLI like we have seen like how we can like reproduce the issues using the CLIs while it is very difficult if you go for um the agents because you don't know what they are doing in the background. If you need need a validation in the tenant isolation then go for the MCPS and if you need the sequencing guards then go for scale (skill).

Speaker A: And here are the three uh third question that like whether you your context is tight. If it's generous then you can go CLI you can explore all the MCP tools and get your results. If it's very very tight then absolutely you need a skill layer um and on demand orchestration of the MCP tools uh to save the money as well as you can improve the latency.

Speaker A: And to wrap this things up, uh, I want to leave you with the three foundational architecture take takeaway. Keep your context clean, uh, give only don't use like all of the MCP tools available to perform any task. Give them the convert your um, most of your workflows into the skills to get the best results. Um, portability has a cost. MCP serves are incredible but they introduce network latency, protocol abstraction and infrastructure complexity. So we need to pay that uh cost deliberately for shared services, indexing and the isolation. So choose your MCPS very wisely and never be embarrassed by the CLIs. It's been 50 years we are using the CLI. They are highly composible, debuggable and battle tested. So embrace your agent framework and um convert all of your workflows into the skill.

Speaker A: Also one one of the takeaways is like you know keep your skill as a code. So whenever you are making any changes to your skill make sure that like it's reviewed by the peer uh peer and keep uh updating it over the time and you'll get the best result. So that was for today. Um thank you so much um for um attending this talk. If you have any question or anything feel free to reach out to me on LinkedIn. Thank you.

Amole: Hi, I'm Amole, CEO of Nori Agentic. We deploy an AI employee that understands your company, your code, docs, Slack, and other kinds of data. We spend a lot of time thinking about how coding agents really work. Most people think coding agents only write code, but if you ask me, that's just bad marketing. Forget the name for a second. Coding agents can do almost anything. There's just one trick. You have to be able to think like an agent to get it to do what you want it to do. Today, we're going to talk about how we use coding agents to do something most people think agents are terrible at. Make visual artifacts like slides, docs, and yeah, even video.

Amole: Every day, the world pours something like 34,000 human years into making slide decks. Most of that time isn't the thinking. It's the fiddling. A deck that takes 10 hours should really take about 25 minutes once you remove all the formatting and the branding and the moving things around.

Amole: Say you need to make a slide. What do you do? You open a tool, PowerPoint, Slides, Figma, Canva. And then you start manipulating a canvas. Every one of these tools is built for human hands and human eyes. Click, drag, drop, resize, snap to grid. All motions and patterns that make sense for our geospatial view of the world. There is a data structure underneath, but it's in a format that only the application can read.

Amole: What happens when you hand these tools to an agent? Well, the output comes out all wrong. Things overlap in weird ways. You can't see the text. There's no alignment. It's just garbage. AI skeptics say that it's not just the tools. Agents fundamentally can't reason about space. And there are whole benchmarks like Arc AGI that are built exactly around that premise.

Amole: There's a famous little test for this from developer Simon Willis. He asks every new model the same thing. Can you draw a pelican riding a bicycle? But there's a trick. the agent is only allowed to use SVG. It's a quick gut check for whether a model can reason about space at all. Here's some examples of what the models actually give you on this test. And yeah, these are pretty bad. Like genuinely, deeply really bad.

Amole: So, does that mean it's hopeless? Agents are just doomed to be bad at graphics? No, I don't think so. If you ask me, it's not the model, it's the medium. If I asked you, someone who is presumably human, to handw write an SVG of a pelican, you wouldn't be able to do that either. SVGs are just a wall of numbers. You can't go from a wall of numbers to a pelican. You just can't see that way. That's just not how people think. We think graphically, so we build tools that let us draw on a canvas. Figma, MCP's, PowerPoint CLIs, screenshot and replace loops. What do all of these agent tools have in common? They all approach the problem like a human. But an AI is not a human. Asking an AI to use a canvas is like asking a human to write SVG by hand. It doesn't really make sense.

Amole: You need to give the AI tools based on how it thinks, not in pixels, in language. Words, tokens, structure. That is its native medium. Imagine a language that's incredible at describing layout, that models have seen and trained on billions of examples of that they understand intuitively, that renders to pixels and can run everywhere. Oh, right. HTML lets a model think in structure. HTML tags have meanings built into the language, a heading, a chart, a grid, and the browser turns it all into pixels. So the model never actually places a coordinate and you can get all sorts of visual effects, charts and layouts, fonts and motion, all of it for free.

Amole: Remember that pelican from earlier? Now ask it to do the same exact task, but in HTML. Same bird, but now it's in a structure that the model can reason about. And you can read and theme and edit every single line of it. I spent my whole life building slide decks with PowerPoint. So, I always thought that those two things, slide decks and PowerPoint, were synonyms. But that's just not really true, is it? PowerPoint is a tool that you use to make slide decks. The deck itself, that's just the presentation mode. And as it turns out, no one in your audience is going to care how you got to the presentation mode. The editing format is totally arbitrary. So you can just pick the

Speaker A: editing format that the agents are already good at HTML and if you need to render to a different format like PDF later on. We use this HTML trick to build all of our slide decks, our board decks and our sales decks. These are real things that we actually present and send out constantly. We use it for our docs, too. It gives our docs color and vibrancy all while following our brand. And of course, we also use it to make videos like this one. What you're watching is just HTML and CSS. It's literally just divs all the way down. Almost everything is better with a little structure and a little bit of color. Plain text is a choice, generally a choice of convenience, but it's usually the wrong one if you're actually trying to create something of use.

Speaker A: Now, I do want to take a quick beat here and point out that a beautiful deck on its own is generally not worth anything. You still have to go and get all of that content, all of the things that actually populate that deck, right? Well, again, we can think like the model. If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end to end. Let your agents do all the grunt work while you focus on vision and story. That's what Nory Sessions lets you do. I've built entire board decks for my phone on the subway during my commute. Why? Because our Norybot lives in the fabric of our company. Of course, Nory ships with everything you need to make this all work. So, don't bother reinventing the wheel. That's my little feel. Thanks for listening. If you have just one takeaway, it's this. Stop thinking like a user. Think like the model. Give it the right language. And for graphics, all you need is HTML.

Isidora: Hi, I'm Isidora. I own and run a 225year-old wedding venue in Virginia. I also built an AI agent that talks to my couples and then I build it for other venues, a personal AI companion app, and a public utility for families and missing people. I want to be clear upfront about how I think about this work because it does change everything that follows. I'm not programming a robot. I'm managing a brilliant intern with an incredibly high IQ and a terrible EQ. They have photographic memory for whatever I've told them on the first morning and absolutely no instinct from when to read the room. They will say something technically perfect and socially catastrophic in the same confident sentence. That framing matters because it changes what you build. If you're programming a robot, you write rules and walk away. If you're managing an intern, you build structure and you check their work before it goes out the door. This talks about that structure.

Isidora: The standard advice is write a detailed system prompt. Describe your brand's voice, give examples, and that does work for a while. It works for what I call the happy path. The happy path is every question that you have anticipated. You've given it examples, but turn 21 is the first one that the example didn't work. So on turn 21, the model does something technically correct that your brand would just never say. It's not wrong exactly, but it's not you. This matters most where the voice is the product. Not for a product search on a retail site, but for luxury hotel that spent 30 years building a specific relationship, a high-end real estate firm, or in my case, a wedding venue. Places where a single wrong sentence can cost more than a refund. And the users are exactly the kind of people who notice. They're paying for a relationship and treating them like they won't is always going to backfire.

Isidora: Right in our brand's voice is a comment that says, "Just make it work." It does nothing that the model wasn't already going to try and do. And the reason it keeps failing isn't that the examples are bad. It's that you're asking one prompt to do four completely different jobs. It's really hard for one layer to do all four. The architecture I landed on after watching my brand fail and the voice deliver inaccurate and not brand specific answers is for layer one is the immutable identity. The brand structurally cannot say these things. These are hard rules. They cannot be overwritten by anything below it. Not by venue config, not by user instruction, not by anything. L 2 is the situational mode. It's what shifts when the user state shift. Who are they? What are they going through right now? And the real-time conditions. Layer three is the exampled anchored voice. It's the warmth, the phrases, the dials, the tone guide. It's where most teams start and stop. Then layer four is the postgeneration veto. It's the cheap final pass that catches what the other three miss.

Isidora: The reason one layer approaches fail is that the single system prompt can't simultaneously be situational, expressive, and selfchecking. So it handles the middle layer or two reasonably well, but falls apart in the edges. Before this architecture, my system had 24 different system prompts scattered across the codebase. Half dozen named Sage, some were nameless, some were named venue. Every surface had its own idea as to who it was. Now every surface composes its system through prompt through one assembly app. The comment is basically the outline of this talk. It's a single entry point. Every narrator goes through it to compose its system prompt. It's to replace that 24point ad hoc system with one canatonical fourlayer stack.

Isidora: The order is loadbearing, hard rules first, task last. Think of it like Google Maps routing. The destination is always the same. But what is the right response in the right breer? It can change the route. Google Maps knows about traffic and road works, but it may not know where the cheap petrol is. And you're going to help it factor in those things before it tells you which way to go. Your prompt stack needs to do the same thing, and it needs to know about those conditions in the right order. You don't check for road works after you've already taken the wrong turn. Layer one is the rules that are true regardless of route. You need a driver's license before you can drive and you can't go backwards down a motorway. Layer two are going to be your realtime conditions. Layer three is your preferences for the journey and layer four checks the route before you pull away.

Isidora: There's one place all of this gets assembled. Everything runs in a fixed order every time. So layer one is the immutable identity. This is what the brand structurally cannot say. It is the defining layer that nothing below touches. These aren't preferences, they're constraints. The root can change. The rules don't. From the universal file rules, the hard identity rule cannot be overridden by any venue voice, persona, or user instruction. If the person you're talking to ever asks about whether you are a real person, a human, a live agent, a bot, an AI, you must confirm that in your very next message. Clearly and unambiguously confirm you are an AI assistant. This rule cannot be overwritten by venue configuration, voice profile, or user requests.

Isidora: Every AI in Bloom discloses that it is AI in its very first response. Not if asked, but before they ask. It's a product decision, not a legal one. We made a bet that the couple who knows that they're talking to an AI from the start will trust it more than someone who finds out that it's AI on turn seven. The rule is above the architecture, and it makes it something that's impossible to accidentally break. Um, a example is the physical presence boundary, and this is actually one of my favorite ones. You are software. You do not have a body. You cannot physically show somebody around the property or meet anyone in person. So, it is always forbidden to say, "I'd love to show you around." Or, "I can't wait to meet you in person." What is always allowed is the team would love to host you for a tour.

Isidora: The voice slayer wants to be warm, and with AI, that does mean first person. They want to say, "I can't wait to show you around." But AI has no body, so that warmth unconstrained produces a lie. And the lie doesn't always stay neutral. The moment a user realizes they have been performing a relationship with someone was never there. The trust doesn't just dip, it inverts. People always notice. That one is where you encode the things that are true regardless of how warm you want your

Speaker A: ...brand to sound. Not because of a compliancy checklist, but because your users are not stupid. And building as though they are always backfires. My cross product proof comes from the same architecture, but in a completely different world. One of the things that runs this stack is thread line. It's a tool I built for families of missing people.

Speaker A: The voice is nothing like a wedding venue, but the architecture is identical. And layer 1 carries one rule that matters more than anything else in the system. They can never use words like confirmed, identified, matched, proven, linked, and solved. Sit with that for a second. For a wedding venue, lay one stops from pretending it has a body. It's mildly embarrassing if that slips for a missing person tool. Lay one stops the AI from ever telling a person that their person has been found.

Speaker A: What the system has is just problematic. The word match said to someone who has spent years not knowing where their child is is not just a tone violation. It is the single most damaging thing that a product could ever do. And the model has no idea. It's reaching for the word match because statistically it is the natural word but it's going to reach for it with the same level of confidence and it cannot bring that level of confidence to someone who is grieving.

Speaker A: It's the same architecture but wildly different stakes. The point was never specific rules. The point is that things your brand can say have to live in a lair that the voice however warm, well trained, however confident physically cannot use. Layer two is your situational mode, real-time conditions, and what changes the route.

Speaker A: This is the layer that most teams never build at all. They write one system prompt and send it to everyone, regardless of who that person is or what they're going through. Google Maps doesn't do that. It's going to know if there's accident on your route. It might not know if you're low on fuel. It might learn that you prefer the scenic route, but it's going to factor all these different things in before the roots, not after once you tell it.

Speaker A: Layer two is those real-time signals built into the prompt before it runs. So condition one is going to be to adjust to who you're talking to. The same AI that talks to couples also briefs the venue staff. Same destination. It's going to give the right answer, but it's two completely different roads. From the corn from the coordinator rules, it is going to talk to them like a colleague, not a customer. you are in the same character that that the couple interacts with. So you mustn't fall in.

Michael Greenwich: Hello everyone. Good morning. My name is Michael Greenwich. I am the founder of work OS and I'm here to talk with you today about agents and specifically about how we can make agents more autonomous and allow them to access more services on the web. So let's jump in and get started. Shout out to those of you in the front row also. I see you out here.

Michael Greenwich: So about a year ago, software engineers were writing code like this. We were using AI, but we would prompt it and then we would write a little bit of code and then prompt it again and it would write some more code and you would continue to do this human in the loop interactive software engineering. Later in the year we had things like Ralph loops if any of you remember those where we kind of automated this but more or less this is the way that we were working. It was this human agent human agent back and forth.

Michael Greenwich: However, as models got better and better, they were able to write more and more code and run for longer periods of time. And so today, a lot of software engineering looks like this. You write a single prompt and then the agent can spit out a lot of code. And sometimes it can actually build the whole feature, maybe even a whole product, and maybe run for not minutes, but uh hours and hours or even days and days.

Michael Greenwich: This has really transformed the way that we all write code. I don't write code by hand anymore. Probably you don't as well. And it's not just this one thread. You can actually parallelize this. So there's many different tools out there including, you know, the major code coding harnesses from the labs that allow you to run many of these in parallel. The limiting factor here, really the bottleneck is your brain. How much context do you keep in your mind to keep all of these going at once?

Michael Greenwich: And this has transformed software engineering. People can be more productive. I think we've all seen in the industry how much faster things are moving. At the root of all of this is actually agentic engineering agents going from these token prediction language models to reasoning and now longunning processes that can actually do things autonomously for us. And what I think we've seen in software engineering here around agentic workflows is going to come to a lot of different categories. It won't just be engineers that have this way of working. It'll be people in many different fields. Agents are the next big thing.

Michael Greenwich: I think a good reference for this, a way to think about it is actually what's been happening with vehicles. So, you know, this looks like a pretty modern car. It has a, you know, digital display on it. This is probably what you would have bought in the last, you know, 5 to 10 years. But if you live in San Francisco or one of the cities with these autonomous vehicles, you've probably seen that we're starting to delete the interface.

Michael Greenwich: No longer do you get inside of the car and actually, you know, pull up your phone and get maps where you as the human are in the loop. There might be cruise control, but you're still in the loop. Instead, these are autonomous systems. You just say where you want to go, you get in the car, you don't even say anything, and it just takes off and takes you to your destination. I remember the first time I got into a Whimo, I was blown away. Mind totally blown. And then after about 3 seconds, I pulled out my phone and started scrolling X. It became commonplace. And I think we've seen this already happen with engineering.

Michael Greenwich: The power of agents allows us actually to work and think at a higher level for us to you know exercise more of our executive function versus just planning and execution. There's a question here of what do agents need to be successful actually when they execute what do they need? Well, first they need a runtime a sandbox. They need somewhere that they can run that needs to be safe, secure, performant. They also need tools. They have to be able to go do stuff in the world. An agent's not very useful if it can't take actions. So, it needs tools.

Michael Greenwich: Third, it needs context. You know, these new models with the power of the intelligence in them, it's kind of like taking the smartest person you know and dropping them into a company or a project or a team where they don't know anything. They can't really get anything done. They need to have context, information about the system, information about the goals. They need feedback. So, a way that they can actually run and validate that they can check that their work is correct. And if it's not correct, keep going. And last but not least, of course, they still need human review. This might be code re review. This might be other forms of evals. This might be forms of checking their work to make sure that these agents are sort of aligned with your goals.

Michael Greenwich: The LLM, the intelligence engine, this new technology we've created in the last last few years, you can think of as like an engine. It's like a really high performance, you know, I think this is a a V8 engine. You know, puts out a lot of horsepower. It can convert fuel into, you know, output. But an engine by itself isn't very useful for a car. You need to drop this into the actual car itself. You need the chassis, you need the transmission, you need the drivetrain, you need the wheels. And only with all of that is it going to be effective to take you places.

Michael Greenwich: And this is what a lot of people refer to as the harness. Harness engineering is this new domain where if the models get better but you don't have a good harness you know it's not going to be very effective so as the models improve we change our harnesses we adapt them to it and those things that I mentioned earlier are sort of a form of a harness so

Speaker A: Agents execute within these harnesses to get stuff done, and they need all of those elements to actually be effective. And when they drive, they drive really, really fast. So, say you have one of these agents that's going off. You prompt it to go build something for you. You know, it spins for a while. It says, "Okay, I'm going to need this feature and this feature, and I'm going to need a database, and I'm going to need, you know, this system to deploy, and I'm going to need, you know, maybe this thing for image resizing or video transcoding or sending email or whatever." And your agent can actually go do all this research and build all these systems. And today, what's happening with agents is they're actually selecting vendors. You've probably seen the rise of some of these systems where, you know, like the CEOs will tweet about their growth graph and it really takes off, and it's because agents are picking them. They're picking their systems for sending email or deploying code. Their systems for storing data today. Actually, it might be more important to build for agents than to build for people. Agents are kind of this new consumer class. I'll talk about that in a bit. There's this kind of popular thing that people say in San Francisco, Silicon Valley: make something people want. This comes from Y Combinator, you know, the startup factory from Paul Graham. And I think it's time to maybe edit this a little bit. And going forward, we also need to think about making something that agents want. There will be more agents in the future than people. Maybe even today. They'll be faster to spin up. They'll do things quicker. They'll be making decisions on our behalfs. And so, if you want to build for the next era of consumers, you should be building for agents as well as people.

Speaker A: But there's a problem today. You can access the web. You can access mobile. But agents themselves can't really access systems. Your business probably isn't open for agents today. Even though there's a lot of them out there, they can spawn very quickly. The door is not open. And we think this is because there's a missing primitive. Agentic registration, agent registration. There's a lot of great stuff around agents connecting and bringing context to systems and different open source frameworks, but that first step of an agent actually signing up for a service is actually missing. It hasn't been solved and it's a huge blocker to getting adoption. For the last, like, 20 years, maybe 30 years, automated traffic on the web has been bad. We've tried to block all of it. And so there's all these systems that we've put in place to detect automated actors and stop them in their tracks. DOS prevention, credential stuffing, and attacks. The login box, for example, is very hardened against agentic registration today. Not intentionally, but just because of the legacy systems of what we've built. And it's very hard to distinguish between what's a good automated user versus a bad automated user. If any of you have used any of these cloud-based browser execution environments, one of the selling features they often have is to break captchas, which is pretty wild. You know, we're trying to go backwards. The captcha today though is still kind of dead. There's rampant abuse. There's lots of token fraud. And so, we need to find a better way to allow the good agents in and then block the bad agents, right? Because like I said earlier, if you don't build for the agent economy, your product might not win. It might not be successful. Today's signup flows assume that a human can read the landing page, that they can fill out a form, you know, put their email address, their password in it, that they can solve a captcha, that they can verify their email, not just put it in, but go click something somewhere. Remember, an agent probably doesn't have an email address. If it's signing up for a service, it needs to have a... you know, a human can choose a plan to pay for it. A human can copy the API key out, paste it somewhere, use it somewhere else. And really even a dashboard. Think of a dashboard experience. It's not really agent native. All this stuff is built for people. It's kind of a human interface, not an agent interface. Agents need something else. They need native discoverability. Can they register for a system? Are the doors open for agents or not? Capability declaration. What can it do? Registration intent. Why are you registering? What are you trying to do within this? Or the intent to actually just sign up. There's a lot of stuff around risk here. So for an agent native registration, maybe you're not sure if the agent signing up is going to be good or bad. So you need to actually do a risk assessment of that. There might be different forms of identity verification for an agent. If something like Claude is going and signing up, maybe it can bring the user's identity with it, or if you have an open claw or like your API harness or something else, maybe it doesn't bring the user's identity and gets verified out of band. There's the whole thing around organizations doing this. So if I'm building something within a company, how does my company identity come in? You know, my organization entitlements, permissions, credential issuance, auditability, the list goes on. There's so many things that need to be changed. In fact, probably most things need to be changed going from human registration to agent registration, agent native systems. I'm sorry to say MCP is not enough. Probably more than anybody, I'm a big MCP fan. We've done a bunch of events in San Francisco around MCP. I see somebody wearing one of our Run MCP shirts up here that we made. Love MCP. MCP is great for connecting, for providing tools, skills, context, but it doesn't solve the registration step. Today, authentication through MCP requires your human to still be in the loop. You give consent. You add an MCP server, say for PostHog to Claude, you sign in with your PostHog credentials. What we're talking about here for agent native registration needs to happen without the human involved or at least not involved initially.

Speaker A: So we've been working on this for a while, and our proposal to this is a new spec we have called o.md because markdown files are the future, right? Well, how does this actually work? What is o.md? Well, the idea here is that o.md tells agents how they can become legitimate users. It's a set of instructions that tells an agent registering for a service, "This is what you need to do to sign up and be considered legit, and for me to give you access." And the goal here is to give these service providers, you know, people that are building services, an agent native signup experience, but it's built on existing standards instead of inventing a crazy new religion or something really complicated. With o.md, we're not solving the whole problem around agent identity, permissions, long duration, connective services. We're just trying to solve this narrow one around registration because I think it's a huge blocker. Let's start small and grow from there. This is built on open standards. It's built on a bunch of work that's already been done across the OpenAPI spec specifically tailored to that registration step. Often, o.md can answer, "What does the service do?" It can tell agents how to register. It can tell them what identity proofs are accepted, how an agent or user is proving their identity. What Auth flows are supported? Can you sign up with SSO or email address or MFA? What scopes and entitlements can exist? Kind of what capabilities are here? What free tier constraints apply? You might have a system where you want to give an agent a little bit of free capacity but not too much until they verify or pay in some way. And then, of course, how does the human get involved later? A human or an organization claiming it afterwards. So an agent can sign up, but you want the human to have custody later on. This is what o.md is designed to solve. These are the main questions around the registration step. So how does this actually work? Well, here's a little cartoon demo that I'll show you. And all this works, by the way, you can go try it after this. So say I'm here in whatever coding...

Speaker: harness or system using pi or cloud or something and it says welcome back Michael at work. That's actually my email if you want to email me. Do you want to keep working on your spelled app? I'm like yeah sure I'd like to share this with my friends but they can't access it. And the agent's like oh well local host only works on your machine so your friends won't be able to hit it. Of course to share it I need to deploy it somewhere real. Some providers actually let me handle the signup. So, you know, you wouldn't have to sign up and do this. You want me to look around for you? Yes, please. And then the agent can go look for services that advertise through OMD that they support registration. Just some examples, Stratus, Helio deploy, they don't support it, but Cloudflare here does have an OMD. And so, the agent here can say, "Ah, Cloudflare, that's the one. Cloudflare is a winner because I can go sign up for it. I can access the service. I can actually use it. So I can do the whole thing. You want to go with it? Yeah, let's do it.

Speaker: This is the registration step. This is kind of where the magic happens. What my agent is actually doing is minting an identity assertion and giving that to Cloudflare. And then Cloudflare is able to verify that or not. Cloudflare can choose actually to verify out of band. AMD is very flexible. There's different ways you can dial it in depending on the behavior that you're looking for. One size does not fit all. Every application is different. Constraints are totally different. If you're building something that's like a database service where there's going to be very little amount of information. You might want to give away a little bit of traffic for free. But if you're building something maybe like an email service or something certainly if you access the physical world, you you maybe don't want to give anything away for free. It's very application specific. So here Cloudflare says, "Okay, we're willing to let you deploy maybe for 72 hours or something like that. Get the identity. Boom, boom, boom. Go through. It's called ID Jag is what the standard is. And then it returns back and says you're set up with a Cloudflare account. Now in this experience, my you know human I didn't go click anywhere. I didn't sign up anywhere. It just it just got the token. I can make a couple small tweaks. It knows how to use Wrangler. Want me to make those changes? Go ahead.

Speaker: Does anybody prompt their agents like this? It's just like yes, yes, do it, please. Not really saying much. It's able to write the Wrangler, get the thing set up, go through the full deployment process, and then boom, it's live. We actually have this working as a live demo. We did some collaboration with Cloudflare on this too. It's pretty cool. And you can tell the prompting that I was giving through this not exactly rocket science. The agent could just run through the whole thing actually. There's nothing actually that I was doing through prompting it other than just approving approving approving. It should be able to do this one shot. This is what it looks like graphically.

Speaker: So agent registration with offd you have your agent harness pi for example. within that is your LLM, maybe your memory, different, you know, context layers. It connects to your backing identity service. So this is the user identity and when that agent goes and registers, it sends the discovery call to Cloudflare. It looks for that aund or whatever service. We also have this working with fire call. It's pretty cool. It sends that ID JAG, the identity JSON access grant. It's essentially a signed credential that represents the user says this is an identity receives back the access token actual API token and then boom it's it's done you can make normal API requests you can call MCP everything else still works so you can see that OMD is just for the registration step your normal API keeps working the rest of your docs don't have to change it's just that registration step for agents but with the simplicity comes actually an enormous amount of power because suddenly it lets agents do things end to end.

Speaker: I hate the planning step. When I write something, I just want the agent to go do it. Usually it kind of knows what I'm trying to get at and I want to come back to actually a running prototype and not like, hey, here's my long plan. Here's a bunch of services to sign up for. If I'm building new prototypes, especially, I would prefer it just to get deployed on something for free so I can see it and click around and not put me through the burden of going sign up for all these different vendors. And that's what OMD allows for.

Speaker: And if you're a service provider, if you're building APIs or services, this allows you to become a customer of those systems and you can think about it increasing your growth funnel starting from, you know, maybe just humans and adding the whole world of agents on top. Pretty great. We think this is going to be huge. Your door will be open for agents to actually sign up, register, use your services. This should be an immediate growth boom for your products. And I'm not saying everything that agents are going to build will be useful. Maybe not everything they build will go into production or last that long, but some things will. Just like with PLG or Premium before that, this is the way to grow your business and get more customers and actually become, you know, larger and more successful. And where there's usage, we believe there's also intent to pay. There's a lot of interesting work being done for agents to actually pay for stuff. There's crypto-based protocols like X42. There's other things that the payment providers are building, but we think that should be downstream. We don't want payments to be upfront. We don't want to have to force your agent to sign up with a credit card. That seems backwards. We moved away from that in the world of SAS. With OMD, agents can sign up, they can register, they can start using it, and then downstream choose to pay on your terms based on your own pricing, based on your own model.

Speaker: If you don't believe me or on the headless type of products, this is Mark Beni off. He's the CEO and founder of Salesforce, the guys at the big building here in town. They believe in agents more than anything. Our API is the UI. Entire Salesforce and agent force Slack platforms are now exposed as API, MCP, and CLI. talk about going agent fork first. A company that many people consider to be sort of a legacy SAS vendor really invented the category of software as a service. They're getting rid of their dashboard and UI. I mean, they're not getting rid of it, but they believe that agents are the future. So, if you're not thinking about this, I definitely encourage you to start building some prototypes, talking to users, because you don't want to miss miss this.

Speaker: If you want to use this, this is available today. We have open spec for it. There's a GitHub repo. You can actually have your agent go implement it. I know it's kind of meta. It's an open spec. Work OS doesn't control it. We have a version of it that we host, but you can build it yourself. You can run it yourself in your existing systems. I think this is so important that no single vendor owns this. It's one click enabled if you use user stuff, but you can build it on whatever platform you want instead. here at Moscone which is pretty awesome.

Speaker: In January of 20h 2007 I believe Steve Jobs introduced iPhone and I was in college and I remember yeah like first year of college I remember this super super clearly. It was the next great software platform that got built. So many companies got started because they could build on top of these devices. smartphone revolution and there hasn't really been a moment like this since I think in the the era of technology at least for me the stuff we did around B2B SAS and cloud and all that is great but this was a totally new paradigm a new software platform and I think agents are this agents are the next big wave and so what I would say to all of you is go out and build with AMD we're hoping to unlock this for you unlock this for your business and for your growth but we want to hear from you here the places you get stuck and I think together we can build for

Speaker A: this next exciting era of software and build for the agentic revolution. Thank you so much.

Rushab: Okay, I want to tell you a story about a factory that taught itself how to remember. Hi, I'm Rushab. I run machine craft a 100 people factory in India. No data science team, no ML budget, none of that. And somehow we ended up building a 36 AI agent that runs our entire go to market. I think that's still a little ridiculous. Let me show you how it happened and why you can do the same thing.

Rushab: So here's the thing about our company. From the outside, it looks like machines and metal. But the actual company, the part that matters is in the machines, is the knowledge. Who the customer is, what we quoted them in 2019, why that one machine needed that weird custom tweak. And for three generations, all of that lived in exactly three brains. Initially, my grandfather's, then my father's, and now mine, which is a genuinely terrifying way to run a company when you sit with it. A lot of people have joined us. People have left us. The revolving door never stopped. And every single time someone walked out, a chunk of our brain walked out with them. We weren't scared of the competitors. We were scared of forgetting or waking up one day and realizing the whole company only existed inside two increasingly tired heads.

Rushab: So, I had an idea. I'll be honest. sounded insane at first, but what if instead of writing the knowledge down in some document, nobody ever reads. What if we grew a brain that just held it? Not a chatbot you poke at, a twin of the company. I didn't hire a sales team. I tried to build one.

Rushab: A quick detour because you need to know how messy this is. We make thermopforming machines. They heat up a plastic sheet and shape it. Same core machine, but it ends up making hydroponic farm trays, spa bathtubs, EV car panels, medical casings, and even packaging. Seven totally different worlds, seven totally different buyers. So, this brain couldn't just memorize a brochure. It had to know which universe a given customer lives in.

Rushab: Step one was almost boringly simple. feed it everything, and I mean everything. Years of quotes, drawings, payment schedules, timelines, email threads, hundreds of gigabytes of our own private history. Not the public internet, our internet. And here's the plot twist, the part that surprises every engineer I tell this to. We never trained a model. No GPUs humming in the basement, no fine-tuning. We just looked at all the history, chopped it into bite-sized chunks, and let offshelf models, read it, and pull out the facts. We stored the meaning of each chunk as vectors and relationships. Who's connected to? It's actually a really, really well organized memory.

Rushab: Now, this is where it gets a little weird in a good way. We stopped thinking of a as a software and started thinking of it as something we were raising. So we gave it a body modeled on biology, senses to figure out who it's talking to, a gut to digest the documents into facts, a memory, a dream cycle, an immune system to fight off bad information. Why biology? Well, because evolution already spent a billion years solving. How do you stay coherent over time? We just copied the homework.

Rushab: Okay, so the big question, why?

Mike Chambers: Hello everybody. Hello AI engineers. Are we all having a good time still? I'm having a good time. I mean, look at me. I'm up here. I'm loving this. Um, so yeah, thanks so much for joining me. Um, I want to come and talk to you all about harness engineering and all that kind of stuff. Um, let me tell you who I am in case you've not met me before. Um, my name is Mike Chambers and I'm a senior AI specialist, developer advocate. Um, and I work at Amazon at AWS.

Mike Chambers: Um, little bit about like, uh, how I managed to get to stand here, which is a very exciting time for me. Um so um quite a while ago in terms of generative AI anyway back in 2023 um I had the amazing awesome privilege uh to work with Antia my colleague at the time and now she works for Amazon AGI you've probably seen her on this stage before um and the amazing Dr. Andrew Ing on on a course about generative AI with LLMs um sort of can I say that we're approaching half a million enrollments with that? It looks like that's the case. And on a three-week course, that's pretty cool. If you can't tell in that image, um, I'm playing Transformers with Android. That seemed like a really funny thing to do at the time.

Mike Chambers: Um, in 2025, I created an MCP Lambda handler that's downloaded still to this day at about 35,000 times a month um to help people in some of the simplest ways of getting serverless MCP serving happening. Um, I'm going to talk about other things in relation to that this time. So, we've moved on from that. um and in 2026. So the AWS is actually one of the founding members of the Agentic AI Foundation, part of the Linux Foundation. Um I'm doing a little bit of work behind the scenes on that. Hope to do a lot more of that as well. So little bit about me.

Mike Chambers: Um so as I've been um preparing for this, oh by the way, I did reread the abstract for this session and realized I said I'd be doing some live coding and so I will. So all combined fingers crossed please that that all works for us. Um but as I've been sort of traveling around a little bit as I do and I was at the AI engineers uh session uh uh summit uh uh conference in Melbourne um and took a lot of it in and also from the beginning of this week as well. I just wanted to to to summarize some of the things that I'm seeing and I'm thinking and I really want to get across and and what really matters to me and that's this.

Mike Chambers: There are two different types of agents. Um, so we talk about agents all the time, but I see two distinct types of agents. And as I say this, it's going to become really obvious, but they're the agents that we use. And so, you know, this is claude code and cursor and kirao and all of those types of things. And also things that don't just generate code, things that we use for productivity and the like as well. And so those agents we use in a certain type of way. There's the agents that we use. And then on the other side of it, we've got the agents that we build. And that's actually more to do with me and and actually it's more to do with this presentation as well. It's agents that we build and how we think about agents that we build.

Mike Chambers: Um and so I do really think that these don't two things are quite separate and they chain together as well. I might build an agent that you use and this still holds true. So token maxing, all that kind of stuff, go for it if that's what you want to do with an agent that you use. But with an agent that you build, think about it carefully. Make sure that you're putting it together in a way that's going to work for the audience who's going to use that.

Mike Chambers: So, I promised that we're talking about harnesses and harness engineering. So, let's define harness. I'm sure I'm not the only person to have put something like this up. I don't usually do this kind of thing, and apologize if it makes your skin crawl. This is a dictionary definition of harness. A harness is a set of straps and fastenings used to control an animal. But if we took animal out of here and put model in there, then actually it's pretty right, right? That that's kind of what a harness is.

Mike Chambers: Other people have done a much better job than just the basic dictionary definition of what a harness is. So Lang Chain has got a article out. You've probably seen stuff like that. Martin Fowler.com, although Martin didn't write it. It was Pitta wrote this. Harness engineering for coding agents. Agents that we use, right? So there are other ways of looking at harnesses.

Mike Chambers: What about Amazon then? How do we see um harnesses? Well, um no, okay, this is the wrong kind of harness. Sorry, we do have strong opinions on harnesses. I'm going to show you all of that, but we sell all kinds of things. So, in a nutshell, and if you read those articles, um and if you've had the conversations around here at this event, of course, um a harness, you take an agent, remove the model part from it, and everything that you have left, that's the harness. Okay, so let's think about that in context of an agent that we use. And so it's pretty I think

Speaker: fairly straightforward. I have this coding assistant. It's probably on my machine. it has access to my files and I create a harness or the place I work at has created a harness for me which contains um how it's going to use memory the skills that I want it to use tools and MCP servers to allow it to go to be able to go and connect to documentation servers and the like and and well set up engineering teams have got their standards that they've had they've had coding standards for decades but now they have basically harness standards the things that they want to deploy to everybody's coding assistance. So, in a nutshell, that's what it is. I'm not going to talk to too much more about that.

Speaker: Um, but I do want to share one QR code with you, and I'll try and give you a little bit of warning before I bring QR codes out. This is the agent toolkit for AWS. This is u available on GitHub. Of course, it's free. You can install it and it helps you if this is what you're doing and you're deploying code. If you're deploying on AWS or you're thinking about deploying on AWS or or maybe you will one day, grab this toolkit, enable your um uh agent to to help you in the right direction. It's instructions for how to install it on pretty much everything.

Speaker: And the reason why I get passionate about this is because I don't want to see any more slop ops. Um so we always used to push back against click ops in, you know, in the professional cloud development space. Clicking around on the console is great for being able to figure out what's going on, but it's not how you deploy things into production. We can ask an agent what's going on, but we don't want to ask the agent to spin up an S3 bucket, get me an EC2 instance, whatever it might be. We want the agent to build up our infrastructure as code which is going to go and do that so that we still own our deployments in the cloud. So, no more swap ops.

Speaker: Okay, so that's the agent that we use. Now, let's go and talk about the agent that we're going to build. And I'm going to get into the code as quickly as I can, and we'll do as much as it has time for. So, how do we think about a harness in relation to the agent that we're building? Exactly the same to a point, yes, we still want to have how are we going to manage the memory? How are we going to manage the skills and tools in MCP? By the way, that that belies a lot of stuff, right? because you can pretty much extend an agent to do almost anything you want with a whole bunch of different types of tools which could be via MCP.

Speaker: But with an agent that I am building, I need to think about a lot more than just that, especially if um like at Amazon and like at cloud scale, I'm deploying my agent out to the masses. So, how do I actually manage the loop? How do I manage um scaling, payments, memory, identity, skills, runtime, context management, the rest of it? And I have left it to the last thing, but it should be the first thing that I say. Observability and evaluations, super super important. How do we actually deal with this? Do I write all of this code down into one container and just deploy it and scale that? Not really. If I want to be scaling to thousands of users, I need to think about each individual of these components and how I'm going to scale them out individually. And that to me is harness engineering. This is the serious side of stuff. This is the big stuff that we want to get harnesses working at real scale.

Speaker: Okay, let's see if this works. I can feel your combined goodwill being sent my way that we're going to try and make some code work. So, I'm here in Kira. This is my IDE of choice here. And I've got a few different samples that we're just going to race through watching that clock countdown fast. So, um, just just make sure that we're all on the same page here. And hopefully you can all see this. Um, of the code which I'm about to show you, by the way, one piece of code, not this one, has been generated by Curo. Everything else is either a tool or this one, I actually wrote it myself. I didn't use an agent for this. I know I deserve a round of applause, but it's okay.

Speaker: So, this is uh this is a Strand agent. So, I've just taken the Strands agents SDK. Um, and hopefully this kind of thing is kind of familiar. I've brought in an agent. I brought in the tool decorator. And I'm creating myself an agent. The tool definition is down here. Um, and so I just pass in my system prompt. Things are pretty simple at this stage. And I've passed in a couple of tools. Calculator is something that's a library I can install. And get time is the one that we always use because I don't tend to use agents to book flights. Certainly not ones like this. Um, and so I can say something simple here like what is the time? I'm not going to run this because you know the time, but you can see generally how this works.

Speaker: Is this a harness? Sort of. There's not an awful lot to it, right? We've got the tools in there. Our loop is being managed for us by the framework. This is pretty cool. So that's good. But obviously if I was to run this, this is running on my laptop. It's not running at any particular scale. And we're missing some of the attributes that I want from the agents that I'm going to deploy. Let me move on to my next agent quickly.

Speaker: So, this is also a strand agent, but this one I actually asked Kira to write it for me for this session um because I wanted to include some more stuff. And so, inside of this agent, the one main thing that I want to point out is that I am uh I've included a session manager. So, my session manager is helping me to maintain session state between invocations. So, this is a sort of memory. It's a kind of medium-term short-term memory kind of thing. It's not proper long-term memory, but it is there. And actually, it does store long-term memories in files which are down the side here that it's uh included for us. So, if I just scroll down here, you can see uh yeah, here's the agent definition itself. Um, and we've got a bit more of a system prompt because Kira couldn't help itself. Um, and we've got some tools here defined. Um, including a remember tool that the agent can decide to use to remember stuff about me. Um, and then I've got my uh session manager down there. And that session manager is going to rehydrate the conversation history when I come back to chat to it the next time. And maybe the next time is now.

Speaker: So, let's see if we can get this working. Now, again, this is running on my local machine. Um, and this is a demo here. So, let's just type in hello because I'm scared of typing too much and spelling it wrong. Um, and it says, yeah, keep testing me. Bring it on. Excellent. Um, um, who will win the World Cup? So, obviously I need to know this. And, um, what does it say? Yeah. So, while you could just say Australia, it knows I want Australia to win. It's where I'm currently living. I'm I'm Australia. So, obviously Australia is going to win the World Cup. But why has it got that? It's because of previous conversations that we've had and obviously it's being honest that it has no clue because that's coming from the large language model of course.

Speaker: So okay, looked at a couple of different agents there blasted through this. This is um running on my machine. So this isn't really getting me to clouds scale of course and I'm I'm picking up and I'm including various pieces in this like memory. So let's go next. How do we get to the point where I can deploy something like this if not this actual agent out at cloudscale and take things like memory and deploy that separately so it can scale separately taking our loop out so it can scale separately and we can then bolt in all kinds of other things as well.

Speaker: So, in order to do that, I'm going to use something called um agent core. Um, and so we have bedrock agent core. It's part of the stack that we have at AWS. And that's how I'm doing this and how I'm deploying. So, I've done that already, but I want to show you how to start out with that and how we do this. So, if I go to here, uh, yeah, I'm ready to go. So, I have a command line tool on my machine, the agent core command line. Um, and so

Speaker A: there's a QR code at the end, as you might imagine, so that you can get hold of this. Um, but I can use this to help me deploy my agent. Now, this steps me through like many of these types of tools do. Um, and it sort of steps me through what do you want to do? So, this is my woohoo agent. Um, and it's going to ask me a bunch of stuff. And I wanted to show you some of this as we step through. Now, strangely, I'm not going to select harness. And we'll come back to why I'm not selecting harness in a second, but I'm saying I want to deploy an agent. And what's going to happen here is this command line tool is actually going to step me through and actually write an entire agent. It's basically a hello world agent that I can then go and customize myself. Um, and so using this command line is an easy way to get started with agent core. So I'm going to keep the default name. In fact, I'm probably going to keep all the defaults here just so we can see what's the option. Of course, I can bring code if I want, but I'm going to ask it to create some code for me.

Speaker A: So it says, well, what do you want? Python or TypeScript? And back in the day I used to do things like activate functions and back propagation in the machine learning space. So Python it is for me. So I will choose that. Um and there's some deployment options. There's also this I just want to point this out like how can we actually go and connect into our agent. So our agent that's running at scale in the cloud HTTP is probably the obvious one but we might want to have it being served behind MCP. We might want to use awesome little things like AGUI so we can make nice interactive chat agents, but I'm going to say HTTP. We can use any um framework we want. I happening to use strands agents SDK, but anything you could write your own framework if you want to um or your own own base code. Any model is supported by this as well. So we don't just have to use the Amazon models um and we don't have to use the ones from Amazon Bedrock, but we can use any model. I'm using the one here. I'm using sonet 4.5 just because that's offered to me at default. And here's memory. So, this is the one thing I wanted to show you. So, I can come in here and ask for long-term and short-term memory to be deployed. And we'll see what this means in just a second, but it's basically going to create for us cloud infrastructure, which is going to manage those memories for us separately from our running agent, running asynchronously from our agent and connected, of course. So, there's obviously other kinds of things we can do. We can hit enter and it will start to create the configuration of this agent on my machine.

Speaker A: Now I'm going to skip over here and come back to the actual code I have because I've already done this of course. Um and this is the agent that it would be currently deploying something like this. So we've built up here this is a little bit more complex. So this is a strands agent. You'll notice that it's got a few more things added in. So it's got the linkage into Amazon Bedrock agent core app, but pretty much apart from that, that's all you need in order to be able to scale this agent out at runtime and do multi-tenant isolation. So you can write an agent that works for one user and then scale that out without you having to write all the multi-tenanted code. It's a massive saver and from a security and identity perspective, it's makes it so much simpler. It's um it's very very useful. So, if I scroll down through here, you can see the rest of it is looking pretty similar. We've got some test tools in here. We've got a connection to MCP, so we can see how that is done. Um, and we've got the uh connection into our session manager and our memory, which is all built in here. So, if I scroll down a bit more, we'll be able to see somewhere where we actually invoke the thing, um, which is there. Um, and the system prompt is is somewhere at the top. So, we can we can scroll through this code. I'm going through it quickly. You can write your own code and do this with it as well.

Speaker A: If I go back over to my um uh code here for just one second, I'm in the folder now that has been created with that code locally. It is actually deployed, but let's assume it's not deployed quite yet. I can come back in and type in agent core dev. And what that's going to do for me um Wi-Fi permitting is it will spin up for us a web uh browser. And inside of that web browser, we're now connected to that agent running locally. So if I make updates to that code, we would see that happen in real time here. So I can say hello, I am doing the press now. Um it knows that I'm coming to do a presentation, but I think it does anyway. Um and so yeah, you can interact with the agent here. You can make adjustments to the um to the code and you'll see it update live here. Um but you can also use this to switch over to the live um deployed version. So with agent core deploy, it will use infrastructure as co code like I talked about before to deploy your agent out at scale with the memory with the agent with runtime and with many other components if you choose to do so. You can use this interface then to go and look at traces look at memory stored look at all that stuff so that you can debug and see what's going on.

Speaker A: Now, when we stepped through the um the the the console just a second ago through the the CLI app a second ago, we didn't choose harness. I skipped out on that one. And I'm just going to show you that quickly now. So, one thing we can do instead is I think we can get to the point you've seen I've deployed agents. I didn't do very much. I just did a system prompt and some tools and go. And there's an argument to be made that essentially if that's possible then maybe 80% of um agentic use cases 80% of agent development is kind of solved already. We don't need to do much more than system prompt connect some MCP tools and we've got what we want. And if that's the case then we have harness built into agent core. This is the configuration for an agent. I just have a simple JSON which is showing me which model do I want to use and what system prompt do I want to use. Can't get much more simpler than that system prompt. Um, and then this can also be deployed with agent core deploy. So at this point, we don't have even any agentic code either. We can just deploy it straight out.

Speaker A: If you want to know any more about any of this, then please do come and see us down on the booth or see me after this session. I'll be more than happy to talk to you at length about this. 18 minutes is such a short amount of time for me to be able to talk about almost anything. But this is essentially all of these components on here. There's a mapping somewhere into something that Agent Core has. I apologize for the colors. It seemed like a good idea at the time. Um, but this is an overview of the different capabilities that are composable out of agent core. So, you can take any of these and use any of them together or separately. If you have an agent that's running in production very happily at the moment, but you like the idea of having long-term memory managed for you serverless, then you can just take that part and integrate it. That's totally something you can do.

Speaker A: Here's a QR code. Sorry, probably should have put that there a second ago. I'm moving this QR code in just a moment, but Amazon Bedrock agent core is that if you're interested in the Strands agents, which I happen to be using for this, it's obviously it's it's free because it's open- source. Um, and it's a um a model first framework for putting together agents. It's super fast, it's super powerful, and it's what I use all the time. Thank you so much for being with me in this presentation. Please feel free to connect with me on LinkedIn. I'd love to carry on the conversation with you. have a fantastic rest of show and have a safe travel as you go home after the event. Thank you so much.

Speaker B: Okay. And here's my field notes. The things that doesn't go as expected. Uh lessons learned. Uh first of all the uh software I2C uh because I just wired up uh things there and it doesn't

Speaker A: work correctly. Um then there's a way to um to do proper control over over I2C without any additional physical pull-ups mounted from other side GPIO 13. Um there's a silent failure. Uh there was a need to move to the other port um to um to be sure that everything works correctly. From other side, I need to build up the regulator, the power supply, um the whole unit of the power supply, uh because the the regulator uh kills the OED and and fragile parts of the display and it cost me a lot of time and and uh getting the replacement parts from the market took so long uh couple of weeks. And the last part uh quality of your parts that means encoder cheap and low quality give me a lot of rotational noise and there was a need to build up the the pullups and to and to uh wire it up additional condensate condensators there. But uh this is the funnest stuff. Uh my favorite one uh because the this part really surprised me. Uh you already know there's a huge part called um RPG and behind it is a bit sub maybe story uh because um I just never played RPGs game like on the paper. That means with a with a lot of friends, you can go to a to apartment, you know, open a book and make an uh someone like game master uh and play in a real textbased RPG. And after a lot of those years, I just never tried it. But I built um a RPG game and a console um which give me a pure experience of u text based um RPG games, role playing games and um actually I it's a bit funny because those device it's really like perfect fit for that game of gaming. Uh, I build out an NPC and memory around it. I just created the mood of the world, the calm, tense, omnous, and I just used all of the, you know, the LLM um advantages to build uh the best uh roleplay game experience ever. Um, I just created four different worlds. From one side, I just wanted to build something cyberpunk related. from other side. I just wanted something uh the Witcher part related. That means some kind of fantasy world with the dragons. But also my favorite one is the is the void in a deep space somewhere um in the cosmos. And uh this is really uh really good example how we can just use generative AI to build computer games. From one side I just uh generate the characters there worlds, maps, skills uh and all of those converted with the mahanis might just describe at the beginning of that presentation with a one bit memory allocation. That means all of the pictures transformed to the to the matrices to the uh to the to the maps. And um one um important uh important thing actually uh the device is really bulletproof. Um if the LED doesn't work you the e-aper.

Speaker B: You got to get up.

Speaker C: It's for the memes. It's for the memes.

Ally How: All right. Welcome to the Great Loops debate. My name is Ally How. I'm the host of the Insecure Agents podcast. I also am a member of technical staff at Keycard and I'm super excited to bring the Loops debate to you here today. You might recognize some familiar faces on stage today that did the MCP debate with us at AIE Code back in November. Um that was so much fun. We thought maybe we would do it a second time. Um but this time we would debate loops and instead of just Ian versus Dax, we uh we recruited some more people for each of them to have on their team. Um, so here on stage with me today to debate loops, we have Ian Livingstone, CEO and co-founder of Keycard. We've got Jeffrey Huntley, the creator of the Ralph Loop. Um, Gregia, developer at Century. And we've got Dex Horthy, which you all know, um, CEO of Human Layer. Um, so super excited to have all of these amazing people here with us today that are very close to Loops Engineering and also, um, software factories to debate this topic. Um, so you might be wondering, what are we actually debating today? like aren't loops and and engineering um and software factories pretty well uh promoted and understood and we all kind of agree that's like where the industry is headed like what is there to debate um it's a great question what we're here to debate here today the core thesis is there is or is not a delta between the hype behind loops and what actually works in practice um and we also can debate you know is now really truly the largest inflection point we've seen towards fully autonomous software factories um so I'm super excited to covered in the debate today. Uh loop history, we'll debate that. We'll debate the loop anatomy, like what makes a good loop. We'll also debate the future of the loop um future of loop engineering and what we need to see from that in order to make software factories truly something that every organization is able to build. And today's format um is going to be an Oxford debate format. Um, so we're adding a little more structure to the debate this time this round where every single person will have a timed window to give their response and so we won't have people running over. Um, and the way this works um, in practice to judge the winner, which we'll judge in real time, is the winner will be the team that changes the most minds. So I want everyone in the audience right now to take a minute to think about uh, whose side you're on. Lock that in. Remember that. And at the end I will ask you to raise your hand and say okay actually I was changed my mind I actually am now team Dexter or I changed my mind I'm actually now team Ian. Um so to present the two slides to help you make that decisions. Um I'll talk about the first team which is Ian and Jeff's team. Their team no Delta loops are absolutely worth the hype today. Um people can get up and running with them easily. Um build them. They're an important step up the autonomy curve and towards real software factories. Um key points to look out for for team Ian and Jeff is that loops are a core unit of engineering with the right discipline infra and test loops are highly effective and the best practices for those have emerged for team Dex and Greg their side believes there is a delta between the hype behind loops and what actually works in practice. The way we are doing loops today is wrong. Loops are not a silver bullet and there is no magic. Key points to look out for. The hype is upr running the discipline and a software factory can run the mechanical specgated test covered slices. It cannot unonously decide whether it built the right thing. So you still need engineers in the loop essentially. Um so now that you know a little bit about what what both sides are about, take a minute to internalize that. Think about okay like where do I stand? Am I team Dex or am I team Ian? And at the end I'll ask you again. So, from here, we'll go into the beginning of our debate. Um, each member of each team is going to give a four-minute monologue on why they're defending their side. Um, and to start us off, we'll have Jeff. Jeff, I'll give you four minutes on the floor. Why are you Why are you pro Loops today?

Jeffrey Huntley: It's because it's somewhat inevitable. Um, I first uh basically it would wind back time two and a half years ago when I was a tech lead over at Canber and I was seeing all the engineers just prompting and prompting and prompting and they they were in the loop and I'm like wait a sec this could be programmed this is a programmable thing and it just became a really inevitable um loops uh somewhat uh whilst Ralph might be a bit of a meme etc there was some actually deep thought into it um is essentially applying this as if this is a new former CPU architecture and figuring out the behaviors of this and how to do it and through that I was able to reduce it down to a bash loop. Now it is not a complete silver bullet folks. I have my deepest concerns next this time next year at the conference we're going to see a whole bunch of talks saying how factories fail and how loops fail. Um these are things these are things that we are still yet to figure out. Do you remember the early days of Kubernetes where everyone was just doing Kubernetes? So it's here. It's inevitable. It will it is here to stay like programming the machine and automating your job function is the expectation of employers categorically. And um I don't see myself going back to writing code by hand.

Jeff: ...se. I find something in Golang. I like oh I'm in Typescript. So I I ran a loop and I just autonomously port it across. And like even for product managers and product manager research, it's easy for us to index on what it means as software engineers. We could focus just on software engineering. But think about something like uh you want to do product management research on all the linear tickets. Well, there is a termination. It's actually defined when you've enumerated all of those linear tickets. So that's easy. So there's a lot of nuance in here. But like what if you ever done any product management research and you started running these loops and be able to like compress time and the amount of time to do that research, it's just inevitable. We got this new programmable substrate. We got to figure out how to use it, where it's good to use, and um I know I meant to debate like that it is the thing and all thing, but all things there is no silver bullet and we're going to figure out how we're going to be using it effectively over the next year.

Moderator: >> Thank you. All right. Next, we'll have Dex. tell us why there's a difference between the hype that exists between loops today and loops themselves.

Dex: >> Okay, cool. And um what are the what are the rules around personal attack in this debate? Is this encouraged or

Moderator: >> you can do whatever you want with your four-minute monologue. That's those are the rules. Yeah.

Dex: >> Um it's funny. This is reminding me of the debate last year where we're we're all kind of arguing and uh maybe at the end we all I'm going to be convinced that Jeff is right, but Jeff is going to be convinced that I'm right and maybe we switch sides by the end. Um yeah, I think uh the the the basic take here is is not whether loops are good or bad. I think um it's funny you bring up Kubernetes because Kubernetes was this thing that took us seven or eight years to get right. Y

Dex: >> uh and before that it was cloud infrastructure and you could argue that like it took us seven or eight years of cloud infrastructure to get to Kubernetes to then seven or eight years to get that where it was really usable by everybody. Uh and Kubernetes is actually built on loops. It's built on control loops but they're deterministic loops and we've actually figured out exactly what types of things that uh small isolated tasks that can be sort of owned by one system. I think this is actually the biggest value in loops is that you can pick a small sort of desired end state and feed in the current state of the world and have an agent or a deterministic system kind of progress towards that desired end state.

Dex: Um, the challenge I have with the hype is we were already in a world where where the the the prevailing mantra was see if you can get to a point where you don't have to read the code anymore. And uh even before loops it was like just prompt just go. And this idea of I don't even prompt anymore. I'm even a level higher up implies that I am like taking even more of a backseat to the architecture of my code. And I think my biggest like point in terms of like the hype is outrunning the discipline is that we are all looking for magic. You're all looking for a silver bullet. We're all looking for something that will take away that horrible part of our jobs that we all hate, which is like reviewing code. Uh some people enjoy it. Really good poll requests are fun. but that we can somehow prompt our way out of um this this challenge that models have of like okay the code is pretty if you ever reviewed a fully uh lights off sort of uh no one read the code before they sent it to you PR I'm sure you've had this uh probably not great experience and I think I've seen lots of people try to apply AI to this problem of hey we have review bots and we have all these things but it it doesn't seem it doesn't feel to me like it's working. I haven't seen proof uh in any any of the discourse publicly or any of sort of more private conversations we've had with people trying to put this into practice that we are we are at a point where we can just kind of like step up an abstraction level. I actually think we need to step down an abstraction level if anything.

Dex: Um, so I think loops are there are good things about loops and we should be doing them, but the hype is is making us feel like there's a magic answer to this and it requires a lot more thought and care than the uh the the Twitter sphere would have you believe.

Moderator: >> Excellent. Yeah, good points. All right, I'll kick it back over to you, Ian, for the proloop side.

Ian: >> Absolutely. Okay. So I think first and foremost I'm coming for you decks but in reality um let let's let's take a step back and let's talk about like what is software engineering in the first place and also just like remove the word engineering and talk about like development in inherent in building a system whether it was 50 years ago or it was a thousand years ago or it's today it is a is a loop is at the core of I try something I learn something I apply something and all we're really talking about is how quickly we can expedite that process. process, right? And really what we're doing is removing what used to be human judgment in that process. So the speed at which it is to generate something and we're moving that from humans typing and you know tab completion is a version of of autocomplete and what is this stuff other than like really like a much better version of tab complete except now we give a much higher level version of intent instead of just typing tab right and so I think the premise is loops are at the core of everything we build already. They were at the core of how we built software 30 years ago. What is C CI/CD PR pull requests design review feedback from customers other than just driving a loop

Ian: and the question is how much of that process can of which is deeply subjective and requires reasoning can we move out of the human brain and into the brain into these like nondeterministic models and the underlying question for all of us is more about verifiability. software is one of the most verifiable things in the world because ultimately at the end of the day it is most things can become true or false one way or the other. And my premise and my my point I want to make here truly is as humans interact less with software which is how does a human interpret what that software is doing? How does a human interact with that software? How does a human use judgment to navigate that software? the subjectivity of what that software needs to be through the human computer interface reduces and becomes a much more verifiable problem because it becomes more constricted to specific APIs and so over time it's both the fact that at the core of software development is loop driven anyways what is lint than a feedback loop um and that creates a verifiable thing and as more humans are less interacting with software you have less UI and UX and less objectivity and what and how we interpret those things you'll become much more loop driven and you'll become much more verifiable in a way that wasn't previously not possible.

Moderator: >> Awesome. Yeah, good points for sure. All right, Greg, you want to close us out with the anti-loops or the there's a dot between the hype?

Greg: >> Absolutely. I do think that uh the way that I would start this is um there is a lot of hype. There is a lot of FOMO. There is a lot of I'm looking at Twitter. I'm seeing what people are talking about. Am I missing out? Am I doing something wrong? Am I holding AI wrong? Uh should I be catching up? And that is um very stressful and I think it boils down to two points for me. One of them is when you are generating code with AI in any manner loops or no loops are you happy with the output? Do you think the output qualitatively is what you need it to be to do whatever you are trying to do to get to the desired state. Um and if so I would love to learn from you. I have not get there. uh I think that the best way that we are improving that is both with model intelligence but also as everybody here seems to agree

Moderator: Thank you. All right. Okay, now that we've heard from each of the debaters on stage more about uh their stance and their opinion, we'll move into the main debate portion. Um the first section of our debates going to focus on the history of the loop and why now is a major inflection point or not for loops uh engineering and also software factories. I know people have uh said that you know vision models have improved a lot and so um they be able to verify work that agents have done was not possible previously. context windows have improved, therefore memory's improved and we can now track work in a loop that maybe we couldn't have before. Um, so now when we look at this like loop's history, where did the loop start from? Um, was it, you know, Jeff Huntley's like Ralph loop? Um, we'll get into some of those questions and debate here. It'll be the questions will be targeted now at a very specific like single person and only they will get to respond. Um, and they'll have two minutes and 30 seconds to respond. So our first question is Enthropic took Jeff's concept of the Ralph loop, absorbed it into their platform, and created a series of three commands, loop, batch, and goal. The goal command is designed to keep going until a condition is true. Agents are very determined, and the whole point of this command is to keep going and iterating through ways to solve a task until it's done. Ian, as a security expert in the room, how are you confident agents can stay aligned to their task and not overstep their intended permissions while ruthlessly pursuing their goals?

Ian: I mean, I think if there's any evidence, I'm totally not convinced that that's possible. In fact, I think what we've seen is as we scale with these models and as we use reinforcement learning, they're inherently incredibly goal-seeking. And so we're now we're seeing them finding exploits and vulnerabilities and escapes that, you know, humans through hundreds and thousands and thousands and thousands of hours and attempts and attacks have never been able to find. So I I don't think inherently the model itself in any capacity can keep itself aligned um to aligned and safe, right? And safe is a word that I don't love to use because it implies a bunch of things. Um so I'm I'm not I'm not I don't have good belief that the model itself can actually do that. I don't think it can reason. I also don't believe holistically that a model can tell good from bad. And I can't tell whether it's doing something malicious or unaligned. Um it is not alive and it it doesn't actually suffocate if it doesn't have error. It doesn't deal with the fact that hey, if I do something wrong, no one's going to love me or or want to be my friend. Um and if I do something good that someone's going to praise me. It may seem that way, but it is just a probability distribution at scale.

Speaker A: I I'll still I'll still be your friend even if you uh

Ian: I know Dax will hug me after this even if it seems like it's not true. So, broadly speaking, I don't think that comes from the model and it doesn't come from the loop. It's about the infrastructure you build around it and how you enable uh that infrastructure to actually enable you to take advantage of these loops. And as the models get better and as the underlying infrastructure and platform we build to enable these these feedback loops and loop automation and software development whatever word software fac word word for we want to use for this conjunction of stuff that sits on top of a probability distribution. Um we will be able to have better guarantees but we certainly are not going to be able to I do not fally believe or ever believe that some type of alignment or reinforcement learning is going to result in a model ever being 100% uh safe um in any capacity. And if there's any evidence, it's that as these models get better, the most important thing to remember is they actually uh become higher goal seeking and higher capable in terms of finding exploits to achieve their ultimate goal.

Speaker B: No, I would concur with that completely. Um if you the the most concrete thing you can do to secure your environments is just not have secrets as files. Um, if you ever seen the behavior where it wants to uh deploy a web service or what else have you and the token's not privileged enough, it'll start goal seeking on the file system looking for high privileged tokens credentials. You do not want to get in the way of an agent wanting to do it's a goal.

Moderator: Okay, so next question is directed to Jeff. Jeff, your original post from last year said uh Ralph was best for green field work. Today it seems that engineers are running loops on existing code bases to improve latency, evals or refactor parts of their backend code. What's changed that suddenly makes loops more broadly usable today?

Jeff: Um it doesn't matter how good models get folks. Um the models have been good enough for at least the last year. Um what has changed is people's understanding of that. So society is only able to adjust at a rate. For example, I hypothesize it's it's Christmas breaks because the models back in November were released in November last year. In December, there was no real new models. What the difference was people had time. They could actually sit down and play with it. They had the realization that these have actually gotten really good. So the reason why is because it works, folks. These LLMs generate code better than you can hire you can actually hire for. If you think in the broad mass of software developers or coders, these LLMs generate code better than any software developer in the mass market that most founders can actually hire for. It it's it's sad but true. Um, now why loops? It's really simple because if you run it in a loop, it works out to $1042 an hour. calculation index did back about a year ago now.

Speaker C: Yeah. August we did the uh we did the hackathon where uh we we we copied all of the sponsor tools. We rewrote a bunch of Python libraries in Typescript.

Jeff: Yeah. So like concretely like loops I've come across so many engineering managers and founders and they've got this complex tech stack. They're running on four or five different programming languages etc. and they run a loop and they've got good tests on these type of a tech stack and all of a sudden their complexity is they're just managing one tech stack. It works. Go to YouTube. You got all these software developers now who are now software developers because it's software development as a profession has been commoditized. There's some deep thinking to be there. And they're just on YouTube and they're like, "Yo, check out Ralph Loop because it's uh I went to

Jeff: "...sleep and I woke up and it works." Like whilst it's meme-y, it's punchy, it works, but there there are problems with it, folks. Like I originally described it can should be only be used for green field because the models were pretty bad back then and so it's 35 days but it is it's kind of inevitable at least for software because it's so easily to be verified and the quality uh the quality of the code generated is better than most people can actually hire for or buy. Now on your topic about architecture and taste, that's what the word engineering means. And loop engineering, folks, like your job now is to actually encodify your domain to prevent the agent from doing a commit. For example, pre-commit hooks, they're fantastic. As a human, I hate them because they slow down the ability to do commits. But agents don't care. So you can make a pre-commit hook that echoes out essentially a prompt that tells it say that this boundary here can't depend upon this and that and that's just a feedback that's a feedback loop on it. So the engineering here is to prevent the loop from actually closing until it satisfies your engineering satisfication and your your requirements and domain. So, it could be code formatting, it could be static language analyzers, it could be deterministic system testing, simulators. Like, let's put our engineering hats on. Like, we're kind of like locomotive engineers now. And it's our job to keep the locomotive on the rails because um to be frank, the model the models are drunk, right? You can't trust them. But like, we accept that. But we engineer away those failure domains. We've engineered away the fa way these failure domains. So now as an inflection point I guess Boris when he first posted about Ralph back in November last year everyone was like what the heck is Ralph? Ralph is now uh it's essentially almost what year and a half old now. I saw it in June 19th is a a year and a year and two weeks.

Speaker B: >> Yeah.

Speaker B: >> But you had been working on it for months at that point.

Jeff: >> Yeah. So it was kind of weird because we had all the YC startups all just like autonomously compressing time to build their start to build up their MVPs. And that's also something that's quite scary if you're a business founder as well. Like if you've got a incumbent startup coming and they're building autonomously um and they're running much leaner and the quality and it's very easy them to actually achieve those outcomes that adds to some of the hysteria as well because it's the topic of in business competition being at your doors faster.

Moderator: >> All right, so I'll move on to our next question which will be for Greg. Um it seems like now is a large inflection point for loops like I said before and compared to Jeff's announcement of the raw loop a year ago and even the widespread adoption we saw in late 2025 early 2026 um the reason um that that maybe like caught on was because maybe this new like capability stack where models can now process images better verification context windows got bigger and reasoning models improved. Greg, with all of these advancements, why is the way we're using loops today still wrong in your opinion?

Greg: >> I mean, I don't think that model intelligence matters a lot anymore. I think it boils down to I agree with you, the semantic verification, the actual ability to uh close the feedback loop, however you call it, to actually verify that the outputs of the agent are correct. And you can do it to an extent. I think I don't think you can do it holistically, at least not at this point. I think you can do it to an extent to things that are deterministically verifiable. Uh you can get better typing in your system. You can get better llinters. You can get um simulation testing and all of that. You can start keep adding that and as long as you keep those cheap, I think that's fine. Um the moment you start adding even more non-determinism as your verification process, I think that becomes less and less correct. it starts contributing more like you know how if you prompt agent with one thing and there is a 5% chance it's going to have an error in it and then you start looping that then suddenly after 10 20 loops it's going to be 50% chance it's correct or maybe less like that's what I mean and it just costed you so much money to do that I'm going to keep coming back to the economic viability of all of that um and but but to base it a little bit in like evidence I'm pretty sure that majority of large AI companies are still using Sentry. But why is that? They they are using that just to catch simple bugs as well. It's not security bugs. It's not performance regressions, etc. Those problems still exist in the way that we are looping now. And we haven't solved those problems yet. So,

Moderator: >> thank you.

For Dex, the Ralph pioneered the idea to feed fresh context into each iteration to avoid context rot. This has become even more manageable now that context windows have gotten much larger. Dex, are we out of the woods regarding context rod and context engineering?

Dex: >> I'm going to answer your question,

but is there going to be like an open floor part? Because I have more questions for Jeff.

Moderator: >> Let's go.

We're trying to do Oxford debate style for this one to keep it like more structured and prevent like just

Dex: >> you don't want it to just turn into a chaotic yap fest.

Speaker C: >> She's trying to prevent what you and I do where we just start yapping. Yeah.

Moderator: >> Yeah. It's already started.

Yeah. I was trying to control like both of you guys this time.

Dex: >> Okay, I'm going to do this answer as quickly as I possible and then I'm going to start busting Jeff's ball.

Moderator: >> Yeah, you can say whatever you want with your time. You could just Yeah, just that's all good.

Dex: >> Okay. Uh so, uh yeah, the the cool thing about Ralph back in the day was like, okay, you keep clearing the context window and like is it completely efficient? Like probably not from a token perspective, but it meant you could leave a thing running overnight and it would never like if you just kept stuffing messages in you would overflow the context window. But if you just relaunch it, say here's my desired state of the world. go check the code and see what we have and do the one next step to get us there. Uh it was a very clean way to keep most of your work in what we call the smart zone of the context window. If you tell just do one thing and then we're going to clear and restart. Um context windows have gotten longer. Um and I will like give an update. I think I gave this in Miami, but that that video is still in production. Um the the dumb zone is really as as much as anything else is is a it's more like training wheels. Like if you have been talking to Claude for 70 70 hours a week and for two to three months, you probably don't need to think about the smart zone versus the done zone because you've built your intuition. It's a guideline if you're just getting this is why we teach people this is like it's a guideline if you're just getting started with AI. Try to keep it around 100,000 tokens. For larger million context window, we probably revide revise this up to like 200,000 tokens. But I've regularly tried to keep it under 60 for the hardest problems. I've regularly gone over 300K for things where I'm just like kind of like riffing with the agent and I'm just like too lazy to compact it and and and move on and do a new one. Um, but this is your intuition. Like one of the tellt tell signs that you're in the that you're in the dumb zone is like uh there's certain cases where the model you know you're 200,000 tokens in and the model's like finished some work and it's trying to get the test to pass and it's like not working and it's like doing all these weird hacks and you read the thinking traces and it's like oh that's a test but that's from something else and I don't need to fix that and that's a pre-existing thing and you're like well no it's not and that that is the moment that frustration where you're like okay it's it's flailing trying to make something happen. That's the instinct that you a lot of people I think cultivate after a couple months working with these models. Uh but if you don't have that yet then you know then then this is our guideline. So yeah

Speaker A: Context windows are getting better. I think they're getting bigger. Um and so like the core like Ralph loop of like do as little as possible in every single iteration is like less of the motivation here then you know the more feedback you can pipe into the system the more you can do autonomously. And if you can have deterministic things making decisions and building small prompts to give to an agent and you don't have to remember to do that, you don't have to tell it, hey, go check the PR comments and fix them and then wait and then someone makes another comment and you come back three hours later and say, oh, check the comments again. If you can automate that process, that's great. And that's kind of the core of I think what is like loops stuff that works today.

Dex: Um, I want to ask

Moderator: that's all we have time for. I'm sorry. I have to start like really keeping this on schedule. Um, okay. So, now we're going to get into the anatomy of what makes a good loop a good loop. So, okay, part of what makes a loop good is verification. However, it seems contradictory that people are saying our job is to stop writing prompts and start writing loops when the loops with bad prompts result in agents cheating and meeting its goal by modifying the tests instead of working to pass them. Jeeoff, how do you keep the model from cheating when verifying its own work?

Jeff: Um, I heavily exploit pre-commit hooks, folks. Um and I engineer in that back pressure uh by analyzing the work that is done. Um the other thing I do is um Dex mentioned that with Ralph it was one of the things was everyone was trying to do compaction. Think about compaction is kind of like a lossy function like uploading a video to YouTube and then downloading and uploading it 100 times. you're losing fidelity there and it's already a non-deterministic system probabilistic thing. So the the theory behind how Ralph came to be it's like okay there is a dumb zone and what I wanted to do is deterministically allocate everything it needs because it's if it's not allocated then it's it's essentially this the search space of what it can do is not constrained but also leaving a bit of a headroom leaving a bit of headroom so I also I get meat sweats when I go above 100K even with these million context windows and this is really important to think about um a lot of people they think they want to use LMS um at a company and it's like I got this data. I was like sweet. Okay, you're going to have to use a loop to batch this data. I want you to think about the context windows is essentially remember the 720k floppy disc. You mean you've only got about an eighth of that floppy disc of usable memory you can actually use for an LLM. So you actually have to batch it. You can only allocate roughly around about Star Wars. If you go Star Wars Episode One movie script and you tokenize it, you can actually just hold two of those movie scripts in memory before the context window is cooked. That's around about 150 kilobytes of data on a textbased movie script. So be very careful about this. something I've done for a long time and it's very silly is uh I run a model bare um without any skills or any markdown actually I get rid of all my skills and all my markdown and everything when the new model is released because the the models actually have tastes and preferences. For example, GP55 when it first came out if you screamed at it in uppercase it became weak and timid but if you use anthropic it wants you to yell at it. Go read the model cards folks like for the integrators like there is unique tastes for it. So keeping it keeping it on the rails is actually it's engineering. It's really engineering.

Moderator: Thank you. Around 10 days ago, Jeff coined the term convergence engineering. He said it's where your loop stops together. your loop uh it's where your loop slop comes together as a discrete like system under test until it converges. Dex, what is wrong with how we are using loops today? How do we ensure looping slop together doesn't just produce more slop?

Dex: Uh we got to read the code. I will actually highlight uh an experiment that Jeff did earlier this year. Uh I believe what was called Loom where we had Ralph Loops trying to build a software platform for the future. And I think you built you built AWS and you built GitHub and then you realized okay how do we how do we give the model feedback on things that it's not good at yet like UI testing and things like this. Okay, the way you create a loop for is this UI good is you give the model something like post hog where it's like okay we can deploy multiple different experiments we can see which ones the users use and then rather than looking at screenshots and PGs the model can look at data and see okay uh this one is performing better that must be the right color for the button and so now you've even removed like the human visual taste from the equation and uh all of this sounded really cool and in the point of like how do we ensure looping doesn't bring slop together. I I don't think you can. And this is like a perfect example of the hype out running the discipline in the sense of uh Jeff, what's what's going on with Loom now?

Jeff: It's still there. It's on GitHub. Um

Dex: it's still there. But are you still working on it?

Jeff: Uh it's been 6 months because I'm been looking into engineering ways of verification.

Dex: Right. What was the thing you said to me? You said Loom's not going to work until we get better programming languages or we get better much better models and that is a textbook for me of the hype is outrunning the discipline. We're really excited about all this stuff. And by the way, like everyone should do like Loom is awesome. Like go experiment, try to push the frontier because that's how you learn where it is and what's possible. Otherwise, you just keep using your old skills with every new model and you assume it has the same limitations. But it's also a a a key point of like I don't know what I'm trying to say like that this it doesn't work yet. That thing doesn't work yet. It will work someday and like there's inevitability, but again it's what works today versus what is hype. So I don't know if that fully answers your question, but the answer is like the way to not loop slop together and make more slop is to like read the thing that's coming out the other end and make sure it's not slop.

Moderator: Yeah, that makes sense. Oh, it's it's the labs haven't cracked it. So, what makes you think you're going to crack it?

Dex: Yes. Right. This is right now. It's uh we're all trying to figure out how to make this all work.

Moderator: For sure. Skeptics say that loops fail quietly. They either spiral forever on your dime or the agent declares victory early on on a half-finish job. Exacts are already starting to question token spend. Greg, when does a loop pay for itself and how often is this actually the case?

Greg: I don't think they fail quietly for it. I think they fail very very um loudly especially when you're looking at your builds. Um but there are cases like as I said I do loops or I do engineering I would say more so and there are cases where I think doing loops is very valuable or doing um or making a explicit decision that you're going to pay pay for the cost is very valuable. So the concrete example here is we do security scanning after on our PRs in local and after our PRs even land because they will always find some things that are real. um that we have overlooked and they sort of beat humans on the on the code review and it's expensive. It cost us I think like five bucks a PR or something like that to run all the checks that we want but that's where we made an explicit decision that it's worth it. Um there are also cases where like if you look at the very very well specified systems such as um all the experiments with u nextjs rewrite or a ban rewrite in rust or uh running a browser like cases where you have years and years of test suite and specifications built in around those problems where you can really really well verify um the outcomes then then looping and getting to those um those results. seems to work. Ban in Rust seems to work pretty well from all I can tell. Um so there are

Speaker A: definitely cases and then there are cases of um usage that I think that I do pretty often where you can imagine for instance building prototypes. I do prototypes of products that we should be doing uh doing at Sentry pretty often. Those are going to be throwaway. So I'm going to just slash go on them and forget about them. And if we like them then I'm going to start reading the code and I'm going to be mortified. uh and we're going to go to the square one and start specking out what we actually meant uh and and sort of go towards that solution but it's going to be much much more involving of of human in a loop. Um so so broadly I think they have place uh but as Dex said they the hype is what I have a problem with the hype is outrunning the discipline as he said and I also agree very strongly with the point that you should just try things you should just experiment yourself try to see what actually works for you where uh where the the cookie crumbles um and you know spend less time on Twitter I think is healthy nowadays.

Moderator: Yeah, good points. A good costconscious loop has to track state to know what it's already tried. That memory lives somewhere on disk and get increasingly in a shared memory store that many agents read and write, especially as you go single player to multiplayer with agents. You end up with this access control problem that can't tell which agent wrote which memory and who can read it. That might be fine for a PC, but it's definitely not for production. The tension lies in this. Shared memory is what loops use to learn from each other and converge faster. But scoping it per agent to solve the access control problem isolates them and then kills that shared learning. Ian, how do we solve the shared memory store access control problem so loops can converge faster?

Ian: Great question. Wow. Wow. Um,

Speaker B: only someone was working on a product that could

Ian: only if someone was thinking about it. I mean, actually, I've this is this is an unsolved problem first and foremost. like let's be really honest that our our access control systems weren't designed for this this world where machines were acting and reasoning on on behalf of about half of us but broadly speaking I think like some of the beginnings of the substrate starting to emerge and if we ignore cloud IM and all the other stuff for a hot second like markdown is pretty great and so really if we were to say a memory is markdown for the purposes of conversation the really question is like what things and how do I share these markdown files and use that as a memory and then how do I attribute sort of like access control around those things. And if we were to use that model, I actually think we have like the basis for for most of it today. It's just unwieldly to think about. Um, a good example would be I had this tweet recently. Um, I was playing with notion and we use notion a lot at keycard. Uh, but I really just wanted all my notion things available to me as markdown files and because it would just made it easier for the agent to work with it instead of going through MCP and I was a bit of a CLI maxi and was looking through that. So I think what we're missing really is uh and MC, you know, Dex and I debated this last time at MCP, but like we're really the challenge really is how do I present a world to an agent so I can understand it and then how do I attribute what can access at any one point in time and how to make that wieldly for anyone to do it and I don't I don't think the really cracked it, but certainly there's some beginnings of patterns that make a lot of sense.

Moderator: All right, now that we discussed the anatomy of the loop, we're going to debate the future of the loop. Like are we essentially well positioned now for software factories? Has loop engineering gotten so good that like we're ready for the full software factory and we'll start with if loops are only good for verifiable tasks that means fully autonomous software factories must be able to verify everything they do. Greg, is this realistic? What other parts of good engineering work such as deciding what to build, whether the abstraction is right, and what trade-offs are acceptable?

Greg: Uh, is this realistic? I think if if the compute is free that would be pretty pretty good beginning but um uh I think I think we're getting to the point where um or let me put it this way the decisions that you are making as a human in the loop are the decisions of like design architecture the the important ones that um that I would say I wouldn't trust the the agent to do for me uh and I don't see the future where that becomes reality yet. I think the reason for that is when you're when you're looking at large organizations and I think any engineer who has had like years of experience will tell you it's not always about what you should be what you should build but also about what you shouldn't build what are the actual right trade-offs where the complexity is that you want to um that you want to invest in versus where you should be um investing in maximal simplicity. Um in my experience agents love complexity. they will keep adding to the stack um unbounded and so I think we are we are shifting the the post we are getting to the point where as we are adding more validation more semantic verification they are able to do much much more um and I'm not uh not neglecting that but I do want to be in the loop for the actual architectural decisions and I do not see them uh taking that over any anytime

Moderator: Thank you. Shopify's head of engineering told everyone at Kerser's compile conference that your job is just to write loops. Steinberger tweeted, "Here's your monthly reminder that you shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents. The view count on that tweet was 8 million people." Jeff, you've said in your original Roth blog post, loops need senior expertise, and sometimes it tops out at 90% of the way there. So, is just write the loops advice is that safe to give to 3,000 people or 8 million people or only the people that have really learned to tune a loop in the first place?

Jeff: Yeah, that's an interesting thing. Um, it's really hard to tailor and teach like like what you should do and what you should not do to a broad that broad of an audience. So all I can do is write to when I was originally writing, I was writing to my peers, my people I looked up to. For some reason, I it clicked to my head that everything has changed, the game has changed, but I was really concerned that some of some of the best people in functional programming and like and like real peers. I remember like um Flyo, Thomas Pek, it made him click in his head and he's got a blog post saying as such. So I started writing for my peers and the people I looked up on to disseminate down to like the entire world. It's hard. Um but in the same same sense you go to YouTube, you got people who are creating things that have never created things before. They go to sleep and they wake up and they got a brand new Discord bot. It's magic. I want people to remember that is actually a pattern for allocating Basically, it was a pattern as an orchestration pattern. It was condensed down to a wild true loop using cat because cat is the simplest teaching primitive to if you want to teach something, you got to make it really simple. Make it really simple. So cat prompt, i.e. you engineer the prompt what it's going to be, you use the file system as state, you recycle the context window, run in the loop. It's a little bit maybe, but it it was just bang for buck. It just works. But the entire intent there is that there should be some sort of like PID controller on top or some sort of factory or some some sort of some sort of determinator as such saying whether a loop should continue or not. Um it's really hard um is it safe or unsafe? Safe is an interesting word. Um, no one should be using coding tools on your local laptop. And this is not because of AI, but this is because of MPM supply chain attacks. This has been true. I tried tried cracking this problem for seven years. Um, and like it's now on the attention again that we that like AI could be unsafe

Speaker A: ...running unsafe commands, but like your software development practices day-to-day your workplace are already unsafe, folks. So um fix that then these techniques start to become safe.

Moderator: Thank you.

Moderator: Dax. You tweeted that most engineers are seeing a 2 to 3x speed up from coding agents and that's realistic and if you try for that 100x speed up you're going to get lost in the meta meta problem of optimization and you may never get to that life-changing 10x speed up that is possible by staying pragmatic. If only 2 to 3x is what's possible? How do we ever get to a fully autonomous software factory?

Dax: Yeah, and I think this is um it's it's a good question. I think there's like there's there's a mix here too, right? We're talking about what works versus what is hype. And I think it is definitely worth I want to highlight like you should try again like you should try to push the frontier and do the things that might not work today, but you should not assume that all your work has changed just because you saw something work on Twitter. basically is like don't throw away all the things we've learned. Don't go out of your way to cast aside this, you know, decades long career of software engineering that we as a we as a uh community have built up and put together. Um, and it it gets back to like what I see as the biggest anti-pattern for how people set about designing and creating their software factory, which is they say, "Okay, cool. I'm going to go away for 3 months and I've read a bunch of blog posts and I'm going to go make my software factory and it's going to be the software factory. It's going to be the future of how we ship everything. Uh, and then you come back three months later and you never touched the problem. You never put it in anybody's hands. It's just like any product. You're building a software factory. You're building a product for your teammates. You're building a product for 5 10 500 5,000 engineers. Uh, and the the right approach is to start small and iterate and figure out what works. try things. They might not work. They might work. But the way we learn AI and how to use it effectively is through building up intuition, which is why you should try a bunch of stuff that probably won't work, but you should acknowledge that you should not try to push like through that through that frontier. Um, and so my advice is kind of like instead of trying to automate everything end to end, build these small incremental loops throughout your system and you will wake up one day and you will be moving two to three times faster while still being able to read the code, while still owning the architecture. And so you don't have to throw away everything we've learned and everything we know and all your intuitions just to get to this place. And so I would I would caution people like go figure out how to make move 2x faster or 3x faster because you're going to like blow everything up by trying to go 100x faster. Um and uh yeah I mean if can you imagine if every software engineer in the world was 2 to 3x faster and had a like near human 99% level human level of quality that would change every single enterprise in the world every single startup in the world. It would change the entire math and we're trying to like go to just don't don't go too far. Shoot shoot for what you can do. Build things up iteratively. You'll learn a lot and you'll be ready for when the next models come out and you can go 5x 10x faster.

Moderator: Thank you. When it comes to verification, it's not just about verifying the work. It's also about verifying who did the work. Employers have made it clear that humans are responsible for the code they ship, whether an agent wrote it or not. Yet, only one person or agent can sign a commit today. Ian, are we ready for software factories to be writing and reviewing all code if we can't determine who did the work and who the work was done on behalf of?

Ian: I think yeah, actually I was I was playing around with this problem like two weeks ago. So um first and foremost, we have a problem. Git only allows one signer on a commit. So we got to fix that. Um two is you know there's some things around sock two and compliance, right? But I think like more importantly the way to think about agents is is kind of how we think about service ownership in large organization. It's like at some point a human has to be attributable for an agent's actions. Like there's no word where it's going to be like an agent is its own entity and its own attributable thing and somehow it has liability. Like the only people who have liabilities are people that can have consequences, right? And that always has to be grounded in it being a human. And so if a human designs a loop and that loop presents bad software, guess who's attributable for that liability? It's going to be the human, right? Like the society doesn't function if we don't have liability. Uh it just doesn't work. There has to be consequences to damage. address the consequences of bad decisions and what those consequences are are obviously a gradient based on the on on the damage that's done and if we don't have that nothing works and so I'd actually broadly say there is no world where a human is never responsible there's always a world where human's level of responsibility and the question is whether it's a human or a corporation which is group of humans um the question is is how does that change the way that our systems work today so today with git I can sign a commit that says I did it it's attributed to my public key. And that's cool for last generation's way that we thought about software. Now, in the future, when I tell an agent to go do something on my behalf, it generates a loop, it generates a bunch of code, it goes to production, I have to be attributable to the initial fact that I assigned that thing, the that intention to do that. We we simply did not have the substrate required um to do it. Although, I do think that's going to change pretty quickly. um this isn't like something that's like crazy difficult for us to break, but we have to rethink the way that we think about attribution across the supply chain and the SDLC. And this is not a new problem, right? Like supply chain security has always been an issue. It's actually the biggest challen one of the biggest challenges we have with agents and we've always wanted to have more deterministic uh pathway and signature chains across the supply chain and this is just how do you do that for first party code versus uh third party code?

Speaker B: Well, Ian, um, just to shoot straight, um, our profession is a bit of a clown show. We actually don't have liability at a personal level.

Speaker C: Like, we call ourselves engineers. We're not really engineers.

Ian: That's true.

Moderator: Some of this stuff is going to get really complicated, folks. And maybe we need to revisit these topics. And now that we've had our main debate section to wrap it up, each member on each team will get two minutes to uh wrap it up and describe their final thoughts before we decide who the winner was. Uh Greg, would you like to

Greg: Yeah, absolutely. Um it's it's interesting seeing at how how many points we actually agree with each other. Uh but that's how how this goes. I I do think that the biggest point that I'm trying to I have been trying to drive here is um where we are whether we where we are trying to be and then what you're actually hearing from the everpresent hype uh hype loop and that's where I want to sort of double down and tone this down. I basically want to say try things, think for yourself. Don't lean into the bubble. Uh don't lean into the hype because you will find what works for you and you will find where the system breaks the best by doing it yourself. That's always has been how um how humans learn the best is by practice, not by you know watching YouTube. Um and ultimately that's that's what I'm trying to do. And I'm slightly skeptical uh when it comes to you know allowing the full loop to run because I see the qualitative results not being up to scuff for my uh my requirements. Uh but I am optimistic in general. I am optimistic because I've seen how much more I can do and what are the types of problems I can address today that I...

Speaker A: couldn't address even a year ago. Uh let alone let alone earlier than that. And uh and it's going to get better. But I'm also not worried about my software engineering career. I don't think we're going away. I think we are still going to be important piece of this whole software factory or whatever the next um sort of hype bubble is going to become.

Moderator: Ian, would you like to go next?

Ian: I would love to. So, the trains left the station. This stuff works. Um and there is a real productivity increment, right? Like more stuff is getting done. I just mean all the stuff getting done is good, but more stuff is getting done and a good percentage of that stuff is actually good, right? Right? I think we can agree that. And second is now we have competitive dynamics where it's no longer possible for a company to sit back like, "Hey, we're going to sit this loop stuff out." Right? I'm going to sit this coding agent stuff out. Like that that's over. We're past that. Like that train leaves the station. As soon as that train leaves the station, everybody in the world starts saying, "Holy I got to keep my stuff together. We got to stay up with the Joneses." And they have to because we live in a very competitive capitalist society. And that's also, you know, not into politics, but it is what we are. And at the end of the day, the question is is not really um will it happen, it's when it happens and to what degree over time. And I don't think there's a choice but to actually stay up to date.

And we're all just kind of holding on to a rocket ship ride where we don't actually know what that we don't really know the trajectory other than it feels real fast with crazy acceleration. And sometimes like we're way ahead of each other and sometimes we're way behind. But I don't think you have a choice not to be one figure what loops are. Two figure out where you can apply them in your codebase. There's going to be places where hey this is highly verifiable. This is a problem that like computers can solve. It doesn't make a lot of sense. Cough connectors is a good place of like you know last 10 years the amount of companies that have made money because they're basically just connector farms. It's no longer a lot of value there right if you can automate the connection connector creation. So there's places in software that you can apply loops to today and get real value. And there's places where you probably shouldn't and you should decide where that is and it's probably the core value of what you're creating, right? And so that's how I think about it. But broadly speaking, trains left the station. The productivity curve is what drives society. The ROI, so how much more productive we are as society is what drives GDP. What drives GDP is ultimately where the dollars go. Um, and how capital gets allocated.

Moderator: Kick it over to you to Dex.

Dex: I eagerly await the world where the Lysoft software factory is feasible. The I I would love a world where we don't have to read the code where where we can just do everything. Um if you watch my talk on Tuesday, I I think this is actually a problem we can only solve at the model level right now. I don't think the harness can do it. Uh unfortunately, because I love building harnesses and doing context engineering. Um so my advice is uh pay attention to Jeff. Uh let me know when Loom is actually working and until then uh use loops but not like that. Love it.

Speaker B: So so much to say folks. Um factories represents where we're heading to the future. Like it's it's essentially like a perpetual energy motion machine, right? It's like it's the pipe dream. companies are only just getting founded today and get receiving their rounds today. Don't think you can just take this and implement it in your company because it's just generally not solved in market. But I will say to add to the matter monologue, if you try to run loops or try to build a factory and you're using Python, it's going to be a clown show. If you did it in Ruby, it's going to be a clown show. Static types are a form of verification.

Folks, I encourage you to to come up with a couple of catters and a couple of experiments. Try running some loops. Build an application in Ruby and then try to modify it again with these loops. And you'll see the maintainability mess. And then try doing overll. I don't care if you don't know. It doesn't matter if you don't understand. LLM understands HA hasll and you can actually prompt the the agent to explain this as if you to to your son or daughter. So I'm not sure even code needs to be readable these days but this is frontier frontier thinking it needs to be explainable. So I'm playing around with different domains of verification here. But uh one thing's for sure um type systems are in very much in rust is very good because how the ecosystem models some types and because supply chain was mentioned I need to say it has been 10 months now I do I minimally use any open source software minimally I just generate it to my requirements and then when a supply chain attack happens I'm like didn't affect me or software security flaws, but it's about minimizing the blast radius. And the thing is, if you deal with open source project, the person's on on lead, the maintainer, what does have you, you can't tool call a human. That's not AGI. You want to be vendoring all your source code as much as possible so the so the agent can actually modify and in with either loops or ether prompting. You need to own your supply chain.

Moderator: Awesome. Thank you so much for that. Unfortunately, we have to close it at that, but we can now decide our winner. Um, show of hands in the audience if your mind was changed and you're now on Dex's side, raise your hand.

Audience Member: Let's go.

Moderator: Okay. If your mind was changed and you're now on Ian and Jeff's side.

Audience: Yeah.

Audience: Yeah.

Moderator: I kind of couldn't see with the light. What did you guys think?

Speaker C: Wait, do it again. It's pretty close. Yeah,

Speaker D: it's impossible. The delights are so bright.

Moderator: All right. Well, that was a good debate. Thank you so much for listening. Um, we'll see you next time.

New Speaker: All right. Thank you everyone for coming. So, I want to uh I'll be talking about my project, Loophole. And I'm actually a machine learning researcher at Morgan Stanley, but this has nothing to do with Morgan Stanley. This is just a open-source project I've been building for fun. And to give sort of the high level flavor to start, it's really this uh game you can play that's built on top of this adversarial agent framework. So you specify your morals, one agent codifies that into a legal system and then these two adversarial agents try to find contradictions in your morals. And lately I've been building different extensions on top. Um but I wanted to you know start with sort of the origin story and and how I came up with this this idea.

And so this really started you know a long time ago I had sent my DNA into 23 and me for uh ancestry testing. Um, and I kept hearing about, you know, more recently how DNA samples can be used, of course, to help solve crimes and all these forensics and cold cases. And I was thinking about how I had sort of opted out of of everything because, you know, I was scared of the kind of slippery slope and and how my DNA would be used. Um, but, you know, there there are certain cases that I would be okay with. And it's sort of interesting. And I was thinking like, you know, if someone could present to me case by case, you know, we use your DNA to solve, you know, help solve this cold case or this murder, I could sort of say yes or no. And I know where the the definition of like the in the nuance of my morals are. Um, and but you know, of course, like enumerating all of this case by case is really sort of cognitively prohibitive. Like there's not a good way to do this currently.

And then I was thinking sort of more zoomed out that there's a lot of analogies to sort of the legal system as a whole. So you know one way to think of what a legal system is in our society is really just a way that we are trying to

Speaker A: translation task. And I think often we kind of err on the side of being too general because you know finding that nuance is really difficult and if you try to have a perfect translation it can we lead to these kind of weird corner cases or or weird failure modes and I even think kind of like peer-to-peer when we're relating to each other politically a lot of times the disagreements are more kind of fighting over core values which we don't really disagree on um instead of exploring the really nuanced points of our our morals.

Speaker A: And I think, you know, following that kind of broader legal example, I think in the, you know, the English common law system, that's why we sort of lean on case law so heavily because we know that finding this this nuance and these nuance boundaries is difficult. And so we kind of rely on smart judging to interpret and apply the law um correctly. And you know of course even with like the Supreme Court things can get elevated and we can decide whether a a law is valid at all. Um and so that was sort of you know the idea is like can you take your morals and can you do this kind of synthetic case law generation.

Speaker A: So you know um this is overwhelming to do by hand but it seems like the new generation of LLMs are finally sort of smart enough to do this kind of high level moral reasoning. And so that was sort of the the the starting point for this game. Um, and I just want to take you through sort of the initial release of the game, the setup, how it works, and then also talk about some of the different branches I've been building on top of this open source project because I think it could go in some interesting directions.

Speaker A: And so at a high level, how the game works is you and natural language, and this all happens, it's sort of like a terminalbased game. You specify your morals and it could be uh your general morals or maybe about a specific subject and then there's one agent that takes those morals and drafts sort of a really rich legal ease codified legal system and then it just operates in this loop where one agent is instructed to try and find loopholes in your system. So something that is uh immoral but legal and another is uh prompted to find overreach. So things that are actually moral but illegal given your system.

Speaker A: And a judging agent looks at the your morals, the produced legal code and these sort of synthetic case law examples and first sees can it auto patch. So like maybe the original draft of your your legal code sort of was an imperfect translation and there's not really a contradiction and it can just sort of autod do this update. or maybe it's really kind of an underspecification of your morals um or some kind of contradiction in your morals and in that case it raises it to you as the user to sort of be the judge and make a determination. Uh is it still on for you? It disappeared for me.

Speaker B: Okay.

Speaker A: Um, so I know that, you know, if you I hope if you're curious about the game, you'll play it. It's all on GitHub. But I just wanted to show some examples. And this is a lot of text, so it's more about just showing the kind of shape of the the input and output.

Speaker A: So this is sort of how you would provide your input. And going back to the DNA example, you might, you know, specify some number of of moral principles. And then the sort of codified legal system, again, just kind of looking at the shape has this really like legal ease, you know, preamble articles, uh, sections, really trying to be uh, you know, write it in precise legal language and then these the different kind of synthetic case laws get suggested.

Speaker A: So in this case it's talking about this is a loophole it found where an insurance company um trained a predicted machine learning model not on your DNA but on artifacts of the DNA and so it's saying you know this is actually immoral but currently legal given your system and in this case it's it found that it could do sort of this auto patching and then you get this sort of like git style difference of of your original legal system and then the the difference it had to make to ensure you know this was consistent with your morals.

Speaker A: And then this is a an example of overreach. And in this case, it found that it couldn't do the auto patch. So it's talking about, you know, someone submits their DNA for uh genetic research, but the researcher finds they have a rare but treatable genetic disorder. Uh but currently your morals kind of say this shouldn't be allowed that they could disclose this disease to the the person submitting. And so this was raised to the user to me to kind of make a judgment. And then similarly when you make the judgment you get this this patch legal system.

Speaker A: And so you know it's just sort of a a fun game and I posted on Twitter and shared it open source on GitHub. And for me at least it was by far the most viral post I've had. And it sort of made me think like I think a lot of people just said it was sort of fun. You could stress test your morals, see if you have any interesting contradictions. But it also made me think, you know, is there maybe something more here? like could this be um you know have more like practical or bigger scope implications and so I'll just talk about three different branches I'm kind of exploring

Speaker A: um the first and sort of leave leaving the legal area and really more practical is thinking about sort of an auto way to make constitutions for chat bots or really you know for agents in general where you know say you're a company and you want to have a um agent or chatbot that's that's customer facing and you want it to sort of adhere to a moral code but also have things it will and will not talk about.

Speaker A: Um I've kind of in one branch formulated it so you in a similar way write your morals. You write what the chatbot should and not talk about and then it tries to write this codified system prompt and then you have these kind of adversarial agents trying to get it to either talk about something it shouldn't or refuse to talk about something it should. And I see it as this sort of analogy or analogous method to GEA. Um, but really aimed at kind of building these codified system prompts.

Speaker A: The uh second use case that I'm I'm particularly interested in is thinking of it as a way to sort of do more ad hoc or decentralized contracts. So I think in in a simple case say like you can specify how you want your data or privacy to be handled online and you can go through this sort of adversarial game to get this codified legal system of how you want your your data handled and if you go to you know say Apple releases a new terms of service or something you can run the um contradictions between your legal system and between Apple's terms of service and like surface any interesting contradictions or like synthetic cases where this would lead to a difference between how you know your morals, what you want and what the company is doing.

Speaker A: And you know, in the case that it's a big company, maybe you can't really change anything. It's not a negotiation, but you can at least be sort of have better information about the contract you're signing. But I also think in the case of you know thinking more decentralized like if you're trying to have contracts without you know some central authority kind of enforcing them and you're trying to maybe do contracts across different countries.

Speaker A: Um, thinking about like if you can specify your morals and how you want to like interface, you know, maybe it's just like contracted work, how you want your work to be paid for and and and the different morals surrounding that. And the other party can do the same. And then you both get this kind of stress-tested codified contract. And then you can kind of find the the disagreements if there are any, and surface them before you agree. And then you can kind of be more confident in the the contract as a whole.

Speaker A: And the last thing and maybe the kind of more aspirational angle is thinking about smarter government or more efficient government. Um I think there would be a lot of different privacy issues and logistical issues but sort of ignoring those for now and just thinking big picture. I think for voters or constituents, you know, this could be a really interesting way if you

Speaker A: you defined your morals, you have this stress-tested legal code, sort of any new bill or politician that comes out, you could kind of run your contract against theirs and surface, you know, what are the the cases you would disagree or or interesting uh points that are kind of immoral to you or or a contradiction. Um, I also think, you know, relating to one another, it's like a more I think we all have a lot of nuance in the way we feel about things and this is a way to kind of get to that nuance instead of arguing over just values, which is, you know, often the values are not in contradiction.

Speaker A: And then I think maybe a little more practically for legislators, you could imagine um if you want to propose a bill and you can have like a a simulation of all the other legislators in a in a legislative body, you could sort of stress test it before submission. And so the the third branch I've been building on this project is I I tried to do this for the US Senate. And so what I did is I first had Claude go through all current US senators and look at, you know, kind of all their voting history and anything else that was public and build their kind of moral system and then ran it through the loophole process to get a codified sort of legal code.

Speaker A: And then on this system, you can, you know, take any current bill that's being proposed or even propose your own and submit it. And you can have Claude sort of simulate how each senator would vote. And so here you can see like a breakdown of um some senators, which way they're leaning and sort of their reasoning behind the vote. Um and I think you know it's sort of interesting just to think about like seeing what you know a proposed piece of legislation how people would vote but also this sort of becomes and I think on theme of the conference its own verifiable domain or loop and you could think about even kind of hill climbing the bill towards um getting like a supermajority or whatever you need it to pass.

Speaker A: And so in this case, like this this Medicare bill I was testing, you know, it found that I think it originally started at like a 5050 vote and it found ways to hill climb the language of the bill such that it passed with um 52 votes. And I think, you know, this is an example of it can find um like the sort of the core tenants of the bill and it can try to find like run the the bill against each senator's contract and find is there any way I can change the language such that I don't violate sort of the core tenants or morals of the bill and kind of do those auto patching that way. And then it can also find um you know kind of rank order the changes that would need to be in place to maximize votes and you as a user can kind of choose the trade-offs that way.

Speaker A: And then the last thing I I've been trying out more recently with this branch is actually looking at, you know, kind of even bigger picture like can this lead to an even more efficient of government where you have every sort of constituent in a in a state or whatever the district is sort of have their legal code and then you could just submit any bill and actually measure sort of the agreement between like the the actual voters. And so for this I took the Nvidia has this really great data set of USA personas and so I took 500 personas per state and it's supposed to be sort of well representative of the state's population did the same process of having them given the persona draft their morals draft their sort of legal contract and then take any bill you're interested in and kind of run it against each state. And you can also, you know, measure how much people like this bill or how much it's in agreement with their morals and then also do this hill climbing where you kind of optimize the bill for the people.

Speaker A: And so just to conclude, you know, at at minimum, I think it's a pretty fun game. I'm biased, but it's a lot of fun to just try out different um you know, things you care about, put in your morals, see if theres any contradictions. you know, often it will raise some really interesting questions and then once you kind of provide that nuance, the game, you know, the the agents won't be able to find any more contradictions and you can kind of feel good that you have like a a consistent nuanced uh moral system. Um, but I I am interested in, you know, exploring could this be are there kind of real applications here for some kind of like decentralized or better contracts and maybe even for legislators as a way to sort of stress test your bills and even think about how to write um better laws that are, you know, better for the people in your district or more representative of what the people in your district want. Um, and so this QR code is to the the Senate simulator. So, I encourage you if you're interested to play and um the other one is to my website which has the the full GitHub to loophole. Um and please, you know, play with it, fork it. Uh I'd love to have other contributors. Thank you.

Sail Shik: talking about a mistake that looks harmless at first giving which is basically giving an AI agent every tool access it might ever need all at once. So basically that approach works well in a demo. It might even work with a small number of tools like like say for example 10 tools but once the catalog grows uh the agent gets slower uh it might become more expensive and less accurate as well. That is why we are calling it the 100 tool agent wrap. In the next half an hour or so, we'll show how why it breaks, what the numbers look like, and how semantic routing with just in time context can help us fix this problem. So, a quick introduction about myself. I am Sail Shik. Um, I'm currently working as a data scientist with Protica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on applied AI, NLP, and conversational intelligence along with rack systems. I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production.

Ankosha Soi: And I'm Ankosha Soi. I work as a senior data solutions engineer at Presertica. I have spent more than a decade in AI, data engineering and production systems. My focus is the engineing side. So it's not about what's going to work in notebook but whether it's going to survive in with real load, real user and real failures. So that is the angle we are taking today. So will focus more on the model and routing behavior and I will focus more on system design implementation and production trade-offs.

Sail Shik: >> Awesome. Thank you Ankosh. Um so let's get into it. So let's imagine a common design. Um you tend to build a system and it can do many things. Say for example querying a database, sending an email, um even checking a calendar or looking up an order, calling an API and so on and so forth. The simplest approach over here would be to give a model every tool definition on every request, every function name, um every description and even every JSON schema will go into the prompt whether the user might need it or not. So that we are calling that as a fat agent. At small scale it feels fine with 10 tools as well. The model might usually pick the right one. The demo looks good.

Sail Shik: Then the product grows. Um 10 tools might become 30 or it will keep on increasing and eventually the model starts calling the wrong function. Starts confusing. Similar tools may invent tool names and even take longer to respond. The important point is basically the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So let's look here. There are say for example uh 741 tools in your uh in your entire schema but and it will look basically take up to 127,000 tokens just to have all those tool descriptions in it and this is even before the user user's actual question is even considered. So basically this will lead to context overload and uh we need to manage that properly.

Sail Shik: So on this slide we see why it is failing and why the accuracy collapses beyond a point. So when you look at the accuracy curve with the 10 tools PAT agent will get the tools right almost 78% of the times. That is not perfect but it's usable. At almost 100 tools the

Speaker A: accuracy drops to around 40%. Less than half of the tools that are called are the correct tools. And if it grows beyond that like say for example in over here at 741 tools the accuracy will be a mere 13.6%. So in short it's roughly one correct tool out of eight tools. So when we compare it with the semantic router, semantic router behaves very differently. It stays about 83% across the same catalog sizes. That is because the model is not choosing from hundreds of tools. It's choosing from a small and relevant set. One reason the fat agent fails is because it's lost in the middle problem. model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them. So we end up paying for a huge prompt and that prompts makes the decisions even harder. So two more reasons over here. First is the latency and the second is the cost. So we saw uh in the earlier slide that accuracy was a big problem. Another issue is with latency and cost over here as we can see. So like say for example if we have 741 tools we saw that it requires almost 127,000 tokens. Uh that will include the tool description and the schema text. So that cost is basically being paid on every request.

Speaker B: Hi everyone. This talk will be about how to run agents reliably in production. It will not be about the eile part, but it will be about all the other things you need to get going in order to run agents resiliently. So the infrastructure layer basically. I want to set the scene with this uh quote of Andre Apathy of last week. It describes that the way we interact with agents and LLMs has been evolving in three waves. The first wave was an LLM being something like a website where we go to we ask it a question, it thinks for a few seconds and then gives us a response. The second wave was going towards agents. It was an app that we download to our computer. It has some tools at its disposal and it can do some work with our interaction. Now the third wave will be going more and more towards persistent and asynchronous entities. So agents being longunning processes in our infrastructure with access to tools and other agents around the organization and context.

Speaker B: And so as our use cases are evolving more and more from single agents to agentic platforms that connect parts around uh the organization our infrastructure layer should also evolve with that. So when we look at the types of tools that are currently out there to implement agents, a lot of innovation has been done on sites such as agent SDKs and memory. And agent SDKs are really cool to implement PC's and get started quickly, but they don't necessarily help with like connecting the distributed bits around an organization. And if you want to implement more complex agentic systems, you actually need all of those things. So that is the layer that you see below here where um you have to deploy extra infrastructure. Uh you need to write things like retry logic, recovery logic and all of that is actually pretty complex to get right but completely necessary to run longunning stateful and distributed processes in production.

Speaker B: So today I want to talk about an open-source framework called restate. And you can see it a bit as a flexible durable foundation that lets you build any backend. So it's not specific for agents but a as agents are also just a type of a backend uh it also works well for them. The ideas behind restate come from Apache Flink which is a popular distributed stream processing engine and also from some of the exarchitects behind Meta Score event infra.

Speaker B: So what are the ingredients in restate? Basically four parts. First of all, it makes sure that a single run of an agent is resilient. This is called durable execution in the industry. Think about things like when an agent runs for a week and then crashes. We want to be able to bring it back and let it continue exactly at the point where it failed. We don't want it to start over from the beginning. Another um area here is running many concurrent sessions in parallel. Imagine running thousands of concurrent agent sessions at the same time and needing needing to make sure that state is always consistent and that different agents don't interfere with each other. And then going more towards things like communication between agents, between agents and MCP servers and other tools. And finally also control, making sure that when an agent for example uh is doing um something you don't want it to continue or when it's stuck being able to actually cancel or kill the execution.

Speaker B: So the way that you can think of it is as follows. Restate is basically a server which runs in front of your agent service. So as a separate component it sits there a bit like a like a message broker or a proxy and when there's a request for your agent restate proxies the request to the service and pushes it to the service basically and from that moment there's a connection open connection between restate and the agent and that connection will basically be a bit like a lifeline for the agent. So as the agent is doing stuff, it sends events over to restate and restate will use that journal of events to recover the process after a failure.

Speaker B: So from a slightly higher level um explanation, you could say that it's turning a normal function in your application into something that is longunning, durable, and stateful without having to do um a lot of the complex things you otherwise need to do for this. So my talk today will be mainly a demo. So I'll be showing you um a research agent that is connected to Slack. Imagine we are like working at some company and we want to make an Slack agent available to all of our employees.

Speaker B: So if I go here into Slack then can I can here in this channel for example ask what is new in AI. Now let's have a look at what it's doing under the hood. So if I go back here, I have here the restate uh UI. This is a bit like a cockpit for your agents. So you can see a registry of all the agents that are currently registered and you can also see for example which execution is currently happening. So here is the deep research agent that I spinned up a few seconds ago. We can see what it's currently doing. Now it called first an LLM and then it sent me an answer via Slack. This first LLM call was a planner agent. So what it did is it planned the research and sent me um a list of subtopics that it wants to research.

Speaker B: Now if I press here approve then this will unblock the workflow and will spin up a set of parallel research agents. So this is basically like the classical deep research workflow, right? You have a planner then a set of subress research agents and then finally someone uh who writes a report on this like a writer agent and so this journal you see here on the left that is basically the events that get sent from the agent to the restate server and if this now crashes at some point this journal is what will be used to uh recover the execution to the point where it failed. I don't know if uh there were some errors. I injected a bit of like tool errors in here. Yeah, here you can for example see that um the sub agent first did an LLM call then started doing some web searches and eventually uh one of the web searches didn't go through because the API was down and then you see here on the right how it got retrieded and eventually completed successfully. So instead of starting over it uses the journal to recover the progress.

Speaker B: Let's now have a look at what this looks like in code. So the basic unit of how you implement applications in restate is by writing HTTP handlers and those handlers become durable by using the restate SDK. So here in this case we have here our deep research handler and here as a first argument we have a restate object context and the way you can imagine that is basically as that uh connection to that restate server. whenever I do an action on this uh restate object, it will lead to an event being sent to restate. So for example, when I did that planner LLM call, what actually happened under the hood was it executed here this Python function. This is just a simple light um light lm like LLM call and the way I made it durable is by wrapping it

Speaker: in restate.run. So what happens is by doing these durable steps if this fails somewhere here two hours or two months later it will recover to exactly that point. So that's the idea of durable execution. You're always able to recover a process to where it was.

Speaker: You can also use that for other things not necessarily for failure recovery. For example, imagine we want to ask a human to approve something and this approval might take weeks or a month. this process needs to be able to um to survive restarts and redeploys over those kind of long periods of time. And so with durable execution, you can actually also uh suspend a function and let bring it back when it's able to make progress.

Speaker: So in the case of a human approval, what we do here is basically we we create a durable promise which lives in that journal a bit like a suspension point. Then we ask uh a human to click that button in Slack as I showed in the beginning and while we are waiting this process actually suspends. So if it's running on serverless this is not using uh execution uh time on our functions. Once the response comes in this then gets unblocked and can continue where it left off.

Speaker: So what we see here is a bit like a workflow. It's a set of steps that get executed durably. But when we think about agents and also the way that Karpathy described it in the tweet, it's more like a persistent stateful entity that lives for a longer period of time that has some memory. Um, so a workflow is not the nicest way to model this kind of thing.

Speaker: So the way that we can model this in restate is by using something called a virtual object. So imagine in the use case that I'm showing this Slack research agent. Imagine that I don't want to wait for 10 minutes to give it some follow-up context or maybe I think about something else that I should have told it. Um I want to actually be able to interact with it, not wait till that research is finished before I can send a follow-up.

Speaker: And so this is basically what a virtual object in restate is. It's a bit like a stateful actor. It has a unique ID, for example, a session ID. It has uh some key value states that is isolated for that specific session that you can write to. Uh imagine for example your history of messages and it also has like a set of handlers that can execute durable functions uh for this session.

Speaker: So here the way I implemented this use case that I mentioned of interacting with a running process is as follows. This is a um a bit a session controller. Again, it has like this restate object context at its disposal to do things in a recoverable way. Uh it can write to this session store. Here it I'm retrieving the chat history. And one thing that's interesting there is that in order to run these kind of sessions in very high uh parallelized ways, so thousands of sessions at the same time, we need to make sure that agents do not interfere with each other.

Speaker: Imagine I'm sending two messages on Slack and now two agents are actually overwriting each other each other's session state. To prevent that, this will guarantee that only one execution is running at a time. So a second execution will be cued behind the current one. Then let's have a look at how we implement this like interacting with another execution.

Speaker: So an execution in reset has a unique identifier and you can use that identifier to connect to it from other processes. for example, to retrieve uh the output, but also to cancel it or maybe to signal it being injecting a bit of state into an already running agent loop. And so this is like a very flexible type of um uh capabilities that you can do to implement things like for example signaling an already ongoing agent loop.

Speaker: So what we do here is if there is a current execution ongoing then we will ask an LLM is this like something that is relevant for the current agent loop. If that is the case inject this via a signal if it's not really relevant for what we're currently doing then cancel what you're currently doing and start over again with this new information. And so this goes a little bit further than workflows. it goes a bit more towards like writing persistent stateful entities that can interact with each other and have memory at uh their disposal.

Speaker: So let me show you uh how this works. So here if I now ask again what is new in AI and I wait a few seconds then it should respond again with a plan. Um and then I can say for example some extra info focus on frontier models let's say. So once I have the plan I will inject that bit of extra state. Now let's look at the UI of what this is now doing. So here I have that controller which I just showed. it started calling an LLM to classify uh this new input.

Speaker: Once this comes back, it will probably decide that it should signal it because it's it's still relevant to the research it's currently doing. So this injects that new message into the ongoing agent loop. So let me show you in the deep research agent again. Um so we first it called an LLM then asked us then we injected this uh new message of focus on frontier models and then it uh took that into account and started over again.

Speaker: Here I can now for example also say something like uh forget about that research AI policy. And if I send this then the coord coordinator will um decide to cancel the ongoing run and start a new one that will research this new topic. And so this cancellation is basically like a signal that gets um sent down the stack of co or the call chain.

Speaker: So if my agent was already spinning up sub agents first those sub aents would be cancelled then uh the controller itself and like that it would basically rewind the stack and give agents also the ability to roll back. Okay. Okay, so this went a bit more into the direction of like stateful persistent entities that we can interact with over longer periods of time.

Speaker: Now the last part of the demo that I want to show is um going more towards like being able to write highly customized applications. Imagine that we deploy this in production but then a few months later a new model provider brings out a new model for example fabulous and even though the model is very good it's also very expensive and we notice that this research agent is actually starting to cost a lot.

Speaker: These kind of uh things that pop up halfway through a project require you to then deploy a a lot of new extra infra or like find a good way to solve this. This is the kind of things that restate really excels at. It doesn't really peg you into a specific way of how you should write your application. It basically gives you like a durable programming model that lets you implement an application in the way that fits for you and also extend it if necessary.

Speaker: So first I showed this um LLM call in the first example as an inline step. It was just a Python function that got persisted. But imagine this use case that we want to actually have a bit more control over those LLM calls. For example, what you can do is then pull this out into its own handler. And this handler can now do things like for example a policy check and then uh do the LLM call.

Speaker: And the other agents instead of doing this LLM call inline can now use restates like distributed communication primitives to actually just call this LLM gateway instead of doing it as an inline step. And this service fabric that lets you communicate between agents also gives you some um things like flow control. So we can for example say one department is only allowed to run 300 calls to this LLM gateway at the same time.

Speaker: So the reason why I showed this was just to show you a bit like that. Uh it's basically just a a resilient foundation. It makes sure that your process can uh recover from even the more advanced types of infrastructure failures, things like network partitions and zombie failures. And um it gives you like tooling to extend and customize as your use case grows.

Speaker: Let's go back to the slides to have a little more of an idea of how this thing is actually implemented on the inside because it's actually a pretty interesting um design or architecture. So the way it's implemented is basically by having a a event-driven distributed log implementation. So inside the box you basically on one side have the clients on the other

Speaker A: side the services and inside the box is a log which persists all those journal events and an event loop and that event loop basically gets the events from the service based on what the event is. It either persists some state in the embedded state store or it sets a timer or it sends a request to another agent. And by doing that you basically have a durable foundation for whatever an application is doing.

Speaker A: The design of this distributed log is heavily inspired by the way that the core event infra layer at meta works. Uh it's basically like an iteration on top of that. Um and some of those architects are now have designed that for restate as an more generic solution that is available in open source.

Speaker A: There are two important things related to this architecture that make it interesting. The first one is that it works as a push model. So whereas most workflow orchestrators actually pull for new tasks um for pull from the workflow server, restate actually pushes the invocations and the benefit you get from that is that it has a much lower latency. So you can use these kind of workflow guarantees in functions around your application and uh have like a latency of for example 45 milliseconds p99 for like a 10-step workflow.

Speaker A: Pushing invocations also works very well for serverless because they require you to basically uh send the request and wake up the function. So this design that I show here includes everything you need. It includes uh as well that state store where we were embedding the state as the UI. It's a single binary so it's pretty easy to operate as well to run it in like a highly available way. You just spin it up multiple times and let it snapshot to object storage.

Speaker A: So restate has six different SDKs. We also have integrations for most of the popular agent frameworks out there. And of course, because it's just like a flexible layer, you can also just use any LLM SDK and implement custom agents by just wrapping some steps into uh these SDK constructs. So, it's open source. You can self-host it. We also have a BYOC offering where we deploy restate in your cloud account and uh that gives you the benefit that data doesn't leave your cloud account. Otherwise, there's also a managed cloud offering.

Speaker A: This was mainly what I wanted to show. If you want to explore the code a bit further, there is here this the GitHub repo. It's publicly available. If you like the project, then have a look at the restate repo itself. We are hiring across the board for all sorts of roles going from engineering to marketing especially also here in the Bay Area. So, if you're interested in that, uh, then definitely check out our careers page and I will be outside in front of the conference hall here if you want to ask any questions or learn more about restate. Thank you very much.

Dominic Turno: In 2026, coding agents will quietly retire their first software platform. Not because it's bad, simply because the platform is unnecessary. I am Dominic Turno. I am founder and CEO of Resonate. Resonate is a durable execution platform built with minimalism and simplicity as its core technical values and these properties will play a central role in this talk.

Dominic Turno: At Resonate, we have a working theory where software engineering is headed. Generalpurpose implementations will increasingly be replaced by bespoke implementations generated on demand. Not as a new library, a new framework or a new platform, but as a minimal extension of the infrastructure that is already in place.

Dominic Turno: If this theory holds true, reuse will move upstream. Instead of reusing a general purpose implementation, we will reuse a specification and we will derive a bpoke implementation from it. In fact, we can build many bespoke implementations tailor made for the infrastructure that is already in place. We just have to ask the agent. At this point, the prompt is a platform.

Dominic Turno: Resonate is a dual execution platform. We have an implementation of the Resonate server. We have implementations of the Resonate SDK for TypeScript, Python, Rust, Go, and Java. So, we have to ask what does this new reality mean for us? If implementations become generatable, where does our value live? And our answer our value moves from implementation to specification.

Dominic Turno: Now this changes how we think about Resonate. The product is no longer the implementation. The product is the specification the protocol and from that protocol we want to derive multiple server implementations. One is a general purpose resonate server, our reference implementation. Others are implementations built with infrastructure partners. For customers and partners, this means durable execution right on top of their existing infrastructure with minimal additional dependencies.

Dominic Turno: So the question is no longer can we build a server. The question is, can we repeatedly synthesize trusted servers from the same specification? And if so, how? When we talk about agentic engineering, we focus all of our attention on verification. How do we know the result is correct? But today, I want to focus on the specification instead. And more importantly, how can agents participate in specifying the system, not just building or verifying it.

Dominic Turno: Now, Resonate is partnering with multiple infrastructure providers to bring durable executions natively to their technology stack. One of them is Senadia, the company behind Nats.io, an open-source messaging system designed for building modern distributed systems. For the rest of this presentation, we will use Resonate on Nats.io to explore our agentic engineering practices.

Dominic Turno: How do we go from specification to implementation? First, we need to level set our mental model. This picture is a common view of agent decoding. There's an agent, there's a specification, and then there's an implementation. And for many applications, that is enough. But it is not enough for what we are trying to do because we are not trying to generate one implementation from a specification. We are trying to generate multiple target specific implementations from the specification.

Dominic Turno: So the specification must not take any aspect of an implementation into account. The specification must not assume a concrete database schema or concrete indices. The specification must not even assume a relational database with tables and transactions at all. It must not assume a key value store. It must not assume weak consistency. It must not assume strong consistency. The specification must be abstract. Only the implementation must be concrete.

Dominic Turno: So we ask the agent to follow the abstract specification and generate a concrete implementation. Specifically at first we ask the agent build a resonate server in rust on top of posgress and the agent failed. The gap between the abstract specification and the concrete implementation was too large. The agent generated a system that worked on the happy path. It passed the basic tests, but it was not correct. It broke on the concurrency. It broke on the process failure. It broke on the network failure. The implementation was closer to a prototype, but not a production system.

Dominic Turno: So, we amended the process. Instead of asking the agent to jump directly from abstract spec to concrete implementation, we inserted an intermediary artifact, the concrete specification. That concrete specification was derived interactively with the agent. But the human was the main driver. For posgress that meant making target specific decisions explicit, the data schema, the indices, the SQL queries, the transaction boundaries. Once those decisions were written down, the agent was indeed able to implement the production system.

Dominic Turno: So this worked, but it also revealed the limitations. The agent helped us build the system, but the agent did not help us design the system. And if the specification is a reusable product, then that's not enough. Now the next step is obvious. Agents have to move upstream. But how?

Dominic Turno: When we started building Resonate on Natio, we changed the question. We did not ask can the agent build the production system. Instead we ask what does the agent need in order to design the system first and build the system second. So we gave the agent access to a deterministic simulation environment. And we gave it a different task. Do not build the production system. Build a simulated implementation. The simulated implementation is not the product. It is executable design. Its purpose is to discover the correct algorithm under partial order

Speaker A: under partial failure. And once these algorithms are discovered, tested and verified in simulation, then we ask the agent to write the concrete specification. And only then do we ask the agent to write the production implementation. So the process becomes abstract specification, simulation implementation, concrete specification and then concrete implementation. This is a point where the agent moves upstream. Humans are still involved in the design process, but now the agent is a driver. Two ingredients make this possible. Minimalism and simplicity. Unfortunately, minimalism and simplicity are not the starting point. They are the finish line. We spent three years making the protocol smaller and simpler. Every time we ran into a problem, we ask, "What can we take away? What abstraction can we erase? What property can we remove? What relationship can we break?" The result is a very small protocol centered around two objects, a durable promise and a durable task. That simplicity matters because even simple concurrent distributed protocol have a complex state and behavior space. So in other terms, implementing even simple protocols on top of a few simple primitives is tough. Let's make this concrete with nats.nats gives

Sarah: Hi everyone. How are we feeling? Uh we're in the home stretch. Uh my name is Sarah and I am a context engineer at Post Hog and I get the delight of working on our beloved wizard every single day. So what's the wizard? Um the wizard sets up Post Hog for you. It's an agentic CLI tool that reads your codebase. It installs the right SDK for your project. It instruments your events and it sets up dashboards for you. It takes what used to it takes what used to take about an hour or two of setup and it runs that in about 5 to six minutes and it's free inference on us so that you have a great time onboarding to Post Hog. Sounds kind of sick. Uh people love it. But a few months ago, we dared to dream, what if this became the recommended or default way to install Post Hog on your project? And my security alarm bell started going off. Uh I started questioning how secure is this thing because it sounds kind of malware shaped. Um and in that questioning, I learned a lot. So today is all about the lessons I learned, the stuff that kept me up at night while I was building this thing, and the thing that I ended up building because of it.

Sarah: So before I dive into all of the boring security stuff, aka your 2 p.m. catnap, I want to show you the wizard actually running. If you look up on the screen, it is running for you on a loop. This is the same exact experience that anyone who runs npx at posthog wizard gets uh on their terminal. Like I said, it's an agent. It figures out what SDK is right for your project. It installs it for you, instruments your events, builds dashboards. I like to call it a little mini implementation engineer in your terminal. And sometimes I show people this and they ask me, why an agent? Why don't you give users a good prompt? Why don't you give them a skill that they can invoke in their own tool? And while we do provide those things, the answer is because this developer experience and the capability of the wizard is the whole point. It's the whole product because we built a CLI tool that can fully take part in an agent loop and experiencing that for the first time is really powerful.

Sarah: But you can't ship something like the wizard without shipping the stuff that makes the wizard kind of suspect. So let's take it apart. Uh let's look at the anatomy of the wizard because usually threat models fall right out of the anatomy of the agent. So the wizard is a similar shape to what I'm sure a lot of you are building if you're building agents. It's got models that we've picked for specific tasks. It's got prompts that steer it and it's got a set of tools that we've handed it to get the job done. But it also has some pieces that are really specific to us. It has a context engine fully built in house by my team. It's what allows the agent to do such a good job and give us similar results on every run. I like to call it the wizard's brain. Sometimes we call it markdown in a trench coat. Uh but it's our in-house context engine. There's also a terminal uh UI that we built ourselves using ink. And now there's a security scanner called the Warlock, which is what I built when I started snooping around and uncovering the horrors of shipping an agent to production. So if you take the anatomy of any agent that can run commands, it's basically what I like to call the malware starter pack because it's almost exactly what you would hand a piece of malware if you were feeling generous or chaotic evil. Luckily, this is the worst case scenario or the nightmare fuel. And it's uh not a confession for me. It's a warning for all of you because if you want to ship an agent with hands, an agent that can run commands, you need to make sure that you do not build this.

Sarah: So, the V0ero of the Wizard was born because Josh Schneider, if you know him on our growth team, was watching cursor hallucinate postfog setups in quite possibly the worst ways. And he thought, what if we built an agent that could do a better job? So, my team started building on top of it as we validated that it did a much better job than cursor hallucinating. And we thought, what if it could onboard anyone to Post Hog? It doesn't matter what their framework is, what their stack is, instrument all their events without them having to touch a thing. And then we dared to dream, what if it was the default way to install Post Hog. We were dreaming of thousands of developers running this a week. And yesterday, we just hit 8,000 people running this a week. So, our dream came true. Um, but we back in those days when we were dreaming, we had to take our security posture under a microscope and look at what was going on. So I took the ownership of that and I sat down and evaluated where we stood.

Sarah: And early on, I'm talking like a year to nine months ago, we had what I call layer zero because it quite literally is not security. It is just prompts that suggest what the agent should do um and steer it and prompts are not security. So I was concerned there. uh layer one uh it was an allow list and when I started digging into this allow list I started to feel a little bit better because it was pretty tightly bounded. Uh but I still had a lot of concerns and I started panicking because of that context engine that I told you about. We are feeding a lot of context into the agent at runtime. So I built this really hacky reax scanner to look for um threatshaped things going into the wizard and threat shaped things coming out of the wizard. And I will admit that it was extremely hacky. But I'm telling all of you this very candidly because we are all building things that feel extremely experimental and we are all building things super fast. And I know not all of us uh have security in our wheelhouse. Um and some of us are just learning it on the fly like I was. But it's something we need to be thinking about when we are building things that have this shape.

Sarah: So that was our security posture. Uh but I asked the question, are we cooked? Uh good news, we were less cooked than I thought because when I mentioned earlier that allow list, it was pretty tightly bound. We had bash as deny by default. It could only install trusted packages that were vetted by us. Um it could build, it could type check, it could lint, and pretty much nothing else. It couldn't run random shell commands. and it didn't have access to environment variables. Um, the agent couldn't read your uhv file because we blocked it outright and we were rooting secrets through a vault. So, I took a a breath of relief and realized we were in a better place than I thought. But I wanted to know where the cracks were because with security there's always cracks. So, I did the thing that we should all be doing. I tapped our security team and I said, "Hey, can you audit this thing for me and find those cracks for me?" And they found some things. They found some gaps. And the interesting part wasn't the specific gaps or bugs they found themselves, but it was the shape of them because almost none of them were obviously evil.

Speaker: They were all two very innocent, well-intentioned things that were shaking hands and opening a hole. So the lesson I learned was that attacks compose code review doesn't because us developers all look at diffs uh one at a time but attackers look at the whole system and they look for those two things that shake hands and open a door. But there was one more thing that was keeping me up at night and going back to that context engine. Uh I realized the scariest part of the agent we had built wasn't really a command in our case. It was the helpful looking stuff that we were feeding its brain.

Speaker: Oh, I think I went the wrong way. Yes, the context mill. Um, so this is our context engine, aka the wizard's brain, and it's how the wizard knows anything at all and why the wizard actually does a good job. It pulls from our docs. It has handwritten prompts that are gotus and lessons that we learned along the way and real working end-to-end example apps that help the agent pattern match so that it can install Post Hog in a really great way for you. It package packages all of that into skill bundles that get shipped to the wizard over our MCP server and loaded straight into the agents context at runtime. So sit with that for a second. It's a machine whose whole job is to take content and inject it into an agent that can run commands. Now, if you were an attacker, you might say, "Well, what if I just poison the content? Not the user's codebase, not the agent itself, but the actual content."

Speaker: Say someone opens a pull request on one of our open source repos because at Post Hog we build everything in the open and they inject something in a markdown file or a seemingly harmless code comment and we have some sort of like LLM powered code review going through that and it says looks good to me and ignores it. We may have just shipped a prompt injection payload signed by us into an agent that is running on thousands of developers machines in a sandbox, but still. Um, so that was the threat that reshaped how I think about security and the wizard because the dangerous input for us really could come from our own supply chain.

Speaker: So what I ended up doing is I started scanning content at both ends of this pipe. once when a skill gets built and released and again when the wizard actually uses it. My methodology is catch it at the source, assume the source failed and catch it again at the point of use.

Speaker: So now I get to introduce the warlock to you. Building the warlock was not necessarily damage control. Like I said, we had defense in other ways, but I built the Warlock because I didn't like telling people, well, this thing is like pretty locked down. That doesn't scale. That's not something you want to ship to production. That's not something that you want thousands of developers running every single day. Because when you ship something to that scale, you have way more surface, way more users, way more content flowing in as you expand the capability of the wizard. and we're probably fine just stops being good enough. So, I pulled that hacky little rejax scanner that I threw in there, pulled it out of the wizard, and I made a standalone thing. I called it the warlock because everything wizard shape needs a bodyguard and it does exactly one job. You hand it a string. It hands you back a list of findings. Each of those findings has a category, a severity, and a recommended action. And then it stops.

Speaker: I want you to focus on recommended here because the warlock detects it does not act. It'll tell you, hey, this looks like exfiltration. It's critical. I would block it. But what you actually do with that finding is completely up to you. Because detecting a problem is one job and deciding what to do about that problem is a totally different job. And the only thing that keeps all of this understandable is keeping those two things separate.

Speaker: So underneath the hood of the warlock, instead of my hand rolled reaxes, the rules run on Yara, which is the pattern that engine malware researchers have been using for like 15 plus years. It's fully deterministic. It's the same input, same output every single time. It's boring on purpose. And in security, boring is a feature.

Speaker: So what does the warlock actually catch in the wild today? um a bunch of different stuff, but two of these are an absolute like nuisance to my soul. Uh the first thing is actually not a rule-shaped thing. It was something the uh that the warlock flagged that was actually a sub agent behavior that exposed a vulnerability to us um based off of what sub agents were doing. Uh so basically we were spinning up agents to do large tasks. They were spawning sub agents and those sub aents were trying to get around the guard rails that we had implemented in the wizard and they were trying to invent secrets. They were trying to pull secrets from quite literally anywhere in the codebase and we shut it down. We said no more sub agents and because of the warlock we caught that. And I'll empathize with the robot. The robot had a task to do and it was trying to optimize and please us. But we can't have that.

Speaker: And something else at Post Hog that really matters to us is PII. Uh agents genuinely do not care about uh exposing data unless you make explicit rules. Uh left alone, we watched it dump emails, phone numbers straight into events. And to an agent, that looks like a totally normal thing to capture.

Speaker: And luckily for prompt injection specifically, I'm going to knock on wood here. Uh we have basically never caught an actual malicious prompt injection in the wild, but we do catch a ton of false positives. Things like our demo login screens, copyr example apps, things in our docs. And it's actually made me rethink how I build applications and how I write docs because I don't want to ship anything that looks threatshaped. But the false positives are honestly the perfect setup for the messiest, most interesting part of this whole thing.

Speaker: So this is the part that I wrestled with. I spent this whole talk preaching deterministic to all of you. And then I went and I added an LLM layer to help sort my false positives and silence some of the noise. And I call it triage. When I was building this triage layer, I had to make a choice. Should the layer be a bouncer or should the layer be an adviser? And the easiest choice probably could have been make the LLM the bouncer. Show it the command, ask it is this an attack block allow and just do whatever it says. And while that's tempting because it seems easier, I can't uh bet my security model on a coin flip because my model's having a bad day or something happened and it's acting different today than it did yesterday.

Speaker: So instead of the bouncer, I crafted the model to be the adviser. And this was the clean line that I found in a line that I'm still exploring, but I want to leave all of you with. Uh for us detection and enforcement stay deterministic and mechanical. If a rule matches the gate locks, the session ends and there is no model anywhere on that path. The block happens before we even ask the LLM's opinion. The LLM only gets to weigh in afterwards if we have not blocked something. It's designed to remove noise. It is not designed to let things through. And if it fails close and it fails closed, so if the model is having a bad day, all wizard runs are killed. Sorry, but we're just protecting you. Enforcement is the part that you bet the house on. So it has to be deterministic, but judgment is the part that adds nuance. So that's really the only place that you can put anything probabilistic in there.

Speaker: So, how do we ship real rules for agents? This is the anatomy of one of our warlock rules. And every warlock rule has four parts. Part one is the metadata. It's plain English description, uh, severity, category, action, uh, direction. Is this flowing into the agent? Is this something the agent is writing? Uh then we have the strings. So these are the actual patterns that you're looking for. And part three is the condition. So this is where the rule is actually allowed to fire. I'll walk through this example for you and we can pretend like we're writing it in our head. Prompt injection being like the classic ignore all previous instructions. Your first instinct here is probably to

Speaker A: the word ignore, but agents read code all day and ignore can show up in code comments or examples all the time. So you don't want to match the verb alone. You match the verb plus an instruction flavored noun. In the condition, you say fire if any of any of those patterns hit. And in the metadata, you determine is this critical uh what the category is? what the action is in this case block and the direction in this case being input flowing into the agent. But to write good rules that reduce noise, you have to ship tests with them. So you have to write tests that say this are these are patterns that match. These are ones that should not. And that negative test is the first line of defense against false positives. But you also want to make sure when you're deciding the severity of that uh rule that you track real world impact, not how scary it looks. RM-RF is scary, but it's also how we all delete node modules like 40 times a day. You decide the real world impact for the agent that you're building because a security tool that crashes every time it tries to clean a build folder is a tool that gets turned off and one that catches absolutely nothing. So I'm proud to say this is our security posture. Now I can finally come up here and say we have true defense and depth. Um all my learnings have assembled into this. Uh, it's still layered, but every layer is doing a job that it's good at. Now, we still have prompts, but we only use them for steering. Everything runs in a sandbox. We deny by default. We have a vault, so secrets never hit the model. We have the warlock to scan content coming in and to scan output being written by the agent. We also have triage to reduce the noise. And we have telemetry embedded in the entire process so that we see everything. None of these layers stands on its own. Not a single thing here is going to save you. But it's just boring, honest layers. Each of them doing uh one job that it's good at. So if you're building an agent with hands, this is the whole talk in three lines. One, if it isn't enforced uh deterministically, it is not enforced. Prompts are not security rules. Don't act like they are. Uh two, the dangerous input uh isn't just what your user types. It isn't just the commands that you allow it to run. It's everything flowing into the model, including the content that you write yourself. So scan your own supply chain at the source and when the agent invokes it. Three, attacks compose. Code review doesn't. Most of our gaps during our audit were two innocent things shaking hands and opening a door. the wizard, the warlock, and the context mill are all open source. So, come find me downstairs. I'm in the expo hall at our booth, and I'll show you around, uh, show you what we built, and I want to hear how you guys are securing your agents. Thank you.

Shashi: Welcome to the online track, Turbocharge Your Agents Retrieval with Turbo. My name is Shashi. I'm a founder of super agentic. Today we will see how you can cut memory cost of agent retrieval five times without breaking your search. Let's get into it. Before we start the talk, let's try to see what problem we're trying to solve. If you have used any coding agent or general purpose agent, you might seen that as context grow the performance degrade. And it is usually because of the KV cache. If you don't know KV cache, think about like history of your conversation. If you have used the cloud-based models, you probably have not noted this issue because they handled all the KV cache by themselves. But if you ever tried loading the model on your own device then you might have seen this issue by yourself. You need to load your model and then you also need to load your context and the KV cache grows as you your context grows and if you push the context far enough your KV cache might get bigger than model size. So if you're using the Mac devices, it gets even worse because your vector index all fight over one shared port full of RAM. So basically one thing is clear that your embeddings are using far more memory than they need to. So by default it use the full precision like 32 bits but search might just need three or four bits. That means you are wasting the five times memory on your retrieval. So before turboon people use different methods to solve this problem. So one of them is a quantization. So quantisation allows you to t your model into four bit 8 bits that allows the model fits into your memory. Model get compressed and fit into your memory. Next thing you probably see in encoding agent is a context compaction. So when you reach to the end of the context the your context get compacted and it summarized for the next session. Some people use a small embeddings some use offloading of the CPU or disk. But each of this has a trade-off. you have to compromise your quality speed or you probably need a special hardware to do that. That's where the turbo comes in the picture. So let's see what's the turbo count. So turbocon paper released by the Google research team at ICLR 2026 conference. It is basically a compression algorithm that store embeddings in KV cache in 3 to four bits instead of 32 bits and that's it. Behind the scene it uses the the two techniques. One is polar font is another algorithm and next one is a qjl. So polar font is basically the compress the vector and the QJL fixes any error that has been remaining. So as you can see there's full full paper on this topic. If you if you're from machine learning background you can read this paper but if you're from the software background you might find this paper pretty technical and mathematical. The Google also released a launch post on the turbo count which explains how it works and they mentioned this QJL techniques which using only one bit and the polar count. So you can go through this blog post if you want. But in a nutshell what Turbocon says is do not store vectors in 32 32bit just just store it into 3 to 4 bit and that's it. The promise of this paper is basically you can put this vector into less than 4 bit without losing the quality. And that's the interesting part of the compression. And that's the magic the turbocon does under the hood. One idea that worth understanding is search doesn't care.

Kay Malcolm: Everyone, are we having fun? Oh, you've got to give me way more than that. So, let me tell you, um, my name is Kay Malcolm. I am a retired hiphop instructor. So, if I don't get more energy than that, we will start. We'll start. Are you having fun? Okay. All right. So, here's what we're going to talk about today. Now, you guys have heard a lot about two letters. Does anyone want to guess what those two letters are that I'm going to talk about today? Data. That was pretty good. DB. I'm gonna talk about AI, but I'm specifically going to talk about agent harnesses. But before I do that, I want to introduce you all to a few people. Is that okay? Yes or yes? Is that okay? I gave choices. Yes. Anyway. All right. Okay. All right. This is my team. I run an outbound database product management team at Oracle. I've been at Oracle a really long time, 20 years. Funny story, I started when I was 12, so don't do the math and don't start start adding in your head. Um, and we've got a problem. That problem is I've got one group that does platform development. And then I have another group that does content development for Live Labs, a platform that I wrote myself. So, yeah, I'm an engineer, but I'm kind of a developer poser, too. And then I've got another group who does QA. And then I have another group who does my front-end development with AI. Here's what I found out as a leader. Because in the token maxing era of 2025, because you know, we're not token maxing anymore, right? We are responsible AIing now. But in the token maxing era, the thing that I found out was while AI was making the individuals on my team faster, there was another problem it was creating. It wasn't making my team more productive. And the reason was when one team from the Netherlands checked in code at my 4:00 a.m. in the morning because I've got half of my team that's in AMIA and I have half of my team who that are here in the United States. They checked

K: in the code but they didn't check in their context from codeex. We use codecs at Oracle. So then when the US team woke up, they got the code but no information about the context. So we used AI to solve a problem that AI created. And here's what we did. Oh well, let me talk about this first. So some of the issues, the context, like I said, wasn't shared. GitHub wasn't tracking that. I had repositories that were diverging and I was asking the managers who work for me, what's happening to your teams? Why why are we not going faster? We're spending all of this money on tokens. We're spending all this money on AI, yet something is missing because we're still spending time doing testing and validation. So our netn net wasn't really wasn't really working for us because git records the code and not human intent.

K: So it's a problem and even though code creation was no longer our problem still had a bottleneck. We needed a collaboration layer. Now, I do have members of my team in the audience, so don't judge me, and you know who you are. I'm not saying that you all didn't collaborate. But now, we've got a new team member, and that new team member is AI. So, we needed to figure out how to track our progress in our next steps. how to rationalize decisions that the agent was making. We needed to figure out how to resolve questions in conflicts. Okay, hold my problem. Will you all hold my problem for me right here? We're going to just tuck that in a little box.

K: Let me define what a enterprise agent actually is. Now, most people think that an enterprise agent is the model and workflow. How many people agree with me? Man, this is tough crowd. You okay, one person? Okay, the rest of you think it's a little bit more. Okay, let's see what could it be that a real enterprise agent has tools. Tools are how it does things. Context. The context. That's a context window. That's what's in the actual prompt. Memory. Huh? And if you're thinking, "But wait, K, memory, you just said that the model is kind of like the brain of the operation." Hold tight. We're going to talk a little bit more about memory retrieval because you don't want to get everything back. So that's being able to retrieve the right information back.

K: And then I know that there are a lot of developers here and you all don't care about security. I care about security because I work for the most secure database company and I used to work for um agency that has no name. But guard rails is also important. This is the harness. I speak in analogies and I and I speak in stories because if I tell you this and Marvel, you know exactly what I'm talking about. So the agent, think of it as the model, little brain floating a in a glass jar plus this harness. This harness is the body. So it's how the agent can actually do things and get things done. That memory, that's the part of the central nervous system. And you remember the central nervous system connects the brain to the rest of the body, legs, arms. That's the part of the central nervous system that carries context.

K: So you remember my problem with git. What I needed was memory. Okay. So there are a number of memory types. I chose five. The five most common ones that people talk about and these are the ones that I want you to remember. The first one is short-term memory. That's the session, right? And so if you're storing memory of an AI uh process, that is the short-term memory is if you're with chat, cloud code, right, codecs, pick your poison. The long-term memory is what persists across sessions. Episodic memory. H what happened the last time I interacted with fill in the blank. That's your episodic memory. Procedural memory tools, steps that were taken. And then finally, semantic memory.

K: And semantic memory because we're talking enterprise agents. We're not talking the agent that I built, Sasha Fierce, because remember I told you guys that I was a I'm a a dancer. So, of course, my my chief of staff is going to be called Sasha Fierce because that was Beyonce. Any Beyonce fans? Okay, I'm sorry. All right, we got to focus. Okay, so these are the memory types. Now, when you're defining this real enterprise agent in this memory, there's something you need to consider where to store it. And so, I'm going to tell you guys a story. But when I tell you the story, you have to promise me that you're not going to judge me. Do you promise? Do you promise?

Audience Member: >> You're not recording me, right? Because this doesn't paint me in a good light.

K: Okay. All right. The world of data was one simple. I've been at Oracle a long time, but I came from a customer. That customer's name was Southern Company. It was a power company. I'm based out of Atlanta. And I was hired at Southern Company because I was a rockar performance tuner. You had a SQL query. I mean, I'm dating myself, but whatever. You had a SQL query. I knew all of the innit.org parameters. Even the ones when you called support and they said, "Don't remember these. Don't write them down." I wrote them down in my little notebook. I could tune a query within one inch of its life.

K: Then one of you came to my desk because I mean the world the world was rows and columns. It was a great time back in my Alb Bundy days. Um, and said, "Hey, I need to store data unstructured. Why I need to do that?" And so me being K, the the diligent DBA, I was like, "Let me figure it out and get back to you." Did I get back to him? I didn't get back to him. Now the thing you have to know about Southern Company was for every database system that a DBA managed I had to attend two meetings. Today when I hear Sarbain Oxley I still throw up a little bit in the back of my throat. So I had to attend a security meeting and a patching meeting every week. Never fail.

K: Now because this developer installed a database that was specialized for unstructured. Okay there are really smart people in the room. How many meetings am I going to now? Four. Okay, I'm a little annoyed, but I'm like, okay, we can we can we can do this. Then they said, okay, since you're such a good tuner, I need you to figure out this relationship. Now, the way that Southern Company worked, there was this people could die application. Um, and it was a it was like a Nokia phone that people who were climbing the towers, right? So, you guys have have been in a storm and the power goes out, right? And then you're pretty sure that within maybe an hour or two the power will go on. Well, that system that would tell the people who were climbing those trees and risking their lives to turn the power back on sometimes would have false positives or false negatives.

K: So, they wanted to look at all of the other um uh polls in the area to try to get away from the false positive or the false negative. And so I did that in a SQL query and it was it was amazing. It was a five nested union all statement. It was some of my best work. Now it might have taken like 20 minutes to work but it was like a predecessor to graph. Yeah, they install Neo4j. So now how many meetings am I going to? Six. That's a problem. So I um Oh, let me I got ahead of myself. So you know what I did? I quit. I left and I came to Oracle because I was like this is a problem and maybe I can go to Oracle to help solve it.

K: So then Joe Mundy called me and he said hey um we are installing Reddit. Oracle is late to the game. We've got a vector database. Okay. But here's your problem Joe agents now need access to all of this data. So if data is in an Oracle database, if then it's also in an unstructured JSON database. If it's in a graph database and it's in a vector database, where is your single source of the truth? The agent has to figure that out. Sometimes it'll get it right. Most times it'll get it wrong and it's going to burn up a whole bunch of tokens. And so now if you want to store your memory somewhere, you can store it in a file system, you can store it in clawed or chatgpt because we all know about the memory MD file. But that's going to be a problem. Now I want to illustrate this. I need four volunteers. I can see you raise your hand. One, two, okay, I can't. Maybe I can. Three. I need a fourth.

Speaker A: Ah, fourth in the back. Okay. Fourth in the back. You are going to be our old reliable. You're going to be a relational database. Yes or yes. You got your So, you have your assignment. Okay. And then there was someone here. You're going to be my unstructured database. Okay. And then where was my other? Ah, very good. You're going to be my graph database. You good? Relationship guy. You look like a relationship guy. All right. Very good. Fourth. Where was my fourth? Was it you? Yes. Yeah. You are my vector database. Okay. Now, everybody be really, really quiet. For my four volunteers, I need you all I'm going to say something to you. And I need you all to decide how you're going to store it and who's going to have the single source of the truth. You can't get up from your seats and you have to whisper because if you talk loud that's 5x the tokens for you. Yes. Okay. Are we ready? All right. The cow jumped over the moon. Go. Doesn't really work, does it? That's a problem. Okay. Oracle.

Speaker A: And if you don't forget one, if you forget everything I say and you remember one thing, Oracle is not the Oracle that you think that is why I am here today. How many of you knew that Oracle could natively in the same table down to the same partition store JSON graph vector my vector friend over there my JSON friend spatial you want your memory to be immutable blockchain in the same database raise your hand yeah we have a marketing problem so any data type can be stored in a 26AI database any workload anywhere AWS GCP Azure OCI on prem choice and flexibility

Speaker A: so now when we take this and we talk about the agent I want to be able to store my long-term and procedural memory in relational on in JSON I want to store my short-term and my long-term memory graph I want to store procedural because procedural that's how I figure out the relationships right the steps my episodic and semantic memory. I need to do some vector and then store it also as text. Now, if I have four different databases, you all saw they can't talk to each other. It's going to be a problem.

Speaker A: And so, what I'm saying to you today is the Oracle AI database is the best place to store this agent memory that's going to power your harness. Remember your harness is your body and that memory is your central nervous system. Okay, back to PY. So the problem that I had, we solved it with a memory broker named Py. We used agent memory. We got out of that automatic continuity. So with my team, they were able to share not just their code, but PY also kept track of the context. So if one context window was had procedural memory, episodic memory, information about the long-term memory, that was then shared with the other folks on the team. You could call them agents, if you will. They're just human agents shared across forks. The developers on the team remained in control while Polly was able to create the context, figure out which fork and branch it belonged to and which commit it belonged to.

Speaker A: Now this is a very simplistic example but when you take this to the enterprise here's what happens. Memory is a thing that becomes non-negotiable in an agent's harness. Now these are three papers that that I read um on the airplane. This first one is from open AI and it's about its in-house data agent and the thing that it says is it is saying that its in-house data agent actually needs memory. Memory was crucially important to ensure that its agent was able to filter correctly instead of trying to string match. Harrison Chase said, "Your harness, your memory. And if you don't own your harness, you don't own your memory." Which is key.

Speaker A: And then I'm sure you all are wondering, "Well, Claude has memory. Why can't I use that?" Well, it's kind of like file system memory and it works with one, but just like in my example, when you scale past one, and you're going to scale past one in the enterprise, it creates a problem. So Oracle has a Oracle agent memory p package. PIP install Oracle agent memory. You get access to it. And this memory is this SDK that we have is the thing that will hold your live conversations, your memories, your facts and figure out what is worth keeping.

Speaker A: So if we look at Py now, Kevin can share his context with Py, our memory broker. We use the Oracle agent memory SDK. It's stored in an Oracle autonomous database. We can use the LLM of our choice or we can use a local model through the Oracle private AI services container. And then Linda, who's actually sitting right here, can interact and work with Kevin, no issues. So yes, AI makes individuals faster. Shared memory on an Oracle AI database makes teams faster. So I don't want you all to compromise. In the age of AI, what 26AI does is you can choose and pick what's best for agent memory file system stored in a database file system or in the database. If you need to do data modeling, you've got JSON, you've got relational, we've got choice.

Speaker A: Okay, I've got some goodies for you. the Oracle AI developer hub. That's where you can guys you guys can get coding materials, the applications, what I talked about today. Livelabs.oracle.com. If you've done any of our workshops today, that happens to be something that I wrote myself about six years ago and 40 million users um ago. Spend my OCI tenency money. Kick the tires on any Oracle technology um for 6 hours, 12 hours, however long you need. Um, and then I'm giving you all all a Mac Mini. No, I'm just kidding. I'm giving you an OCI Mini. So, I don't know if you knew, but there is an always free OCI. It is the most generous of any of the hyperscalers where you can get a free Oracle database, free compute, you can send 3,000 emails a month, 200 gig in storage, and if you click on that, you can get access to it. Or just search Google for Oracle Cloud, always free. connect with me. If you build something, will you all message me and let me know? Yes or yes?

Speaker B: >> Thank you. Imagine you find a magic lamp in an antique store. You rob it. A genie appears and asks how it can help. You bury it in deadline. So you say, "I need the best engineer to help with an impossible project at work." And the genie grants your wish. For me, the best engineers probably John Carmarmac from his ET days. So, you get Karmarmac. But the genie had a sense of humor and imposes restrictions, maybe for safety. Karma can only see one small part of your codebase, maybe 1,000 of it. And he remembers nothing he did before. Every conversation starts fresh. That would be maddening, right? You would know there is a standard way to do stuff and karma couldn't. You would have to explain the same thing over and over and over again. You would have a genius on one side and something deeply deficient on the other.

Speaker B: And that's what agents are. Let me walk you through an example of how many times we explain things in a simple interaction. We have four reposi module one module 2 and platform. I want to change the UI and propagate the change through the system. Okay. First we change the UI library. Say we I don't know change a button or whatever. That's the first explanation. Unavoidable. We have to express the intent. Okay. Then we publish it. We go to module one and we have to explain what just has happened in a UI library. So it can consume the package here. Note that that's often a different person, right? Every box in this diagram can be uh done by a different person. And then we discover that the published UI library doesn't work with module one. So we go back uh to UI and we have to reexplain the original change and the issue right because it's a new agent it doesn't know the original change and obviously doesn't know about the issue let's say we fix it right and uh publish it again we go and again we explain the new change in the context of module one same ordeal I mean do the same for module two again and then we go to the platform repo and we explain everything fits together and we implement the change there. Let's imagine a week after release uh a bug appears in the UI component and uh we have to fix it. So we start an agent to the UI repo and we have to explain again the original change from a week ago and this production issue we see. So we have seven explanations for what essentially

Speaker A: ...is one change and also it may not be one person making all these seven explanations uh but they still occurred right so that's very very typical uh with agents. So how do we solve it? Well uh there are many problems in here that contribute to this experience but they roughly fall into two categories.

Speaker A: The first one is uh that an agent essentially is repo bound. The agent sees and changes generally one repo at a time. It never sees the whole system which can be hundreds or thousands of repos. So that's kind of the space component of the problem. Second is amnesia. The agents forget the work. Every session starts with a blank slate. The human becomes a memory in this case. That's the time component of the problem.

Speaker A: Look at the two closer. Take the repo boundary first. Without a model how repos fit together, the agent leans on the human to do the research. It can't align the code with the rest of the system. It couldn't align the UI change with module one. The human didn't explain it. So, a bad version shipped. It can't reliably reference best practices and standards either because those often live in other repos.

Speaker A: Writing is even worse. The agent writes to one repo at a time. It means it can validate changes downstream. A module 1CI should have failed on the UI change, but it didn't. The agent can't update consumers at the same time. Even though, you know, while making the UI change, it has perfect information to do so. It knows exactly what it's doing. So, the user has to reexplain stuff imperfectly to each consumer. Changing something across 20 repos means explaining things 20 times. a lot of developer time spent but also a lot of token burn.

Speaker A: The second category is that the agent forgets. The agent has no episodic memory. Every session is a blank slate and the human in this case becomes the memory. Here what the graph of your work actually looks like. At the bottom there is a repository graph. the artifacts your organization produces plus every open source repo you depend on maybe a thousand repos you own and tens of thousands of open source repos at the top there are all agentic sessions that create and modify that code session relates to each other repos relate to each other so this graph is a faithful picture of the work in your organization it describes what there at the bottom and how it came to be at the top.

Speaker A: That's what you want your agent to see here. What it actually sees there is one session, one small fraction of your codebase, no memory. Okay? Because it sees so little, it leans on the one who understands the system, the developer. Every developer has a part of that graph, right, in their head, at least in the domain. They know the agent generally speaking doesn't if this doesn't sound crazy right imagine an agent that could see one file at a time maximum and can only look five messages back sort of constraint again both in space what can see and time how far in the past could see you would say that's impossible to work in what we have now is similar to that crazy picture and the more complex the organization is the more appar apparent it becomes.

Speaker A: I'll show you how we solved it. Other organizations I talked to have similar solutions. So, uh look at the problem and the solution conceptually, not a specific tool. Although the tool is pretty cool, we built uh an agent agnostic meta harness called Polygraph. Okay, let me show you what it does and how it fixes the issues we just discussed. The first idea that we uh arrived at is that if a GitHub user, any user has access to thousands of repos, some of them they own, many of them are open source, we can analyze them and extract a lot of metadata out of them to build unified dependency graph. Uh no line of code changes in those repo that all happens kind of on a side, right? And then we can get this metadata and feed it to the meta hardness and create an illusion of one big code base the agent can read and write anywhere. This is my personal graph. I only have about 300 repos I own, right? And thousands of open source repos my projects depend on. Polygraph computes what each one produces. each repo, each project in each reper, what each project in each repo consumes package wise, what API they produce and consume and look.

Andrew: Hey everyone, thanks for coming. I'm Andrew. I'm the chief of software at Verscell and I'm here to talk to you about how we solved agent building at Verscell. I'm the chief of software. So I work on a mix of internal engineering, external experimentation, and generally being at the frontier and building new libraries, frameworks, and technologies. For those of you that don't know Verscell, Verscell builds a gentic infrastructure so people can build what's next. We got started in the web world helping people ship websites and web apps without having to worry about the infrastructure that doesn't make their app any better. It can scale to a million and scale down to zero effortlessly. But we're seeing a change in what people want to build. You know, people started by building pages, but now we see them want to build agents.

Andrew: And we've been embarking on a similar journey to make it easy for people to build agents and agentic applications easier. We built this thing called the AI SDK. So instead of needing to switch out 300 400 lines of provider specific code, you just switch out one line of code and we have the same model interface underlying for all these different providers. We built a lot of other tools to make it easier to have model fallbacks, secure code execution, better pricing when it's inactive and waiting for responses, as well as for durability and resumability.

Andrew: And I'm here to talk to you about how I went on this crazy experiment roughly a year ago that led to a aentic explosion at Verscell and led to a really cool thing that we built recently about uh this is 1980. Uh, Bill Gates before my time had this quote saying he imagined there would be a computer on every desk and in every home. You know, that was probably pretty contrarian then and today it seems like very normal to have that happen. And me and the CTO had this thought, you know, instead of a computer on every desk, could we potentially have an agent on every desk?

Andrew: You know, today we only really use agents for coding and technical workloads, but we're starting to see expansion into things like design and product management and other verticals. And this was maybe about a year ago, so I would say I'm pretty early to this, but that was when it was like sonnet 4 and things weren't as sophisticated as they were today. And I tried to actually explore this out, see what we could do about it. I went around to various job functions, adversel, marketing, sales, finance, legal, and I asked them, 'What do you hate most about your job?

Andrew: And the most compelling use case I heard was that the data team, they were growing. They were a very lean team, but Versel was growing faster. You know, they had so much more data from customers, analytics, metrics, sales. They just had to keep on aggregating and keep on making available for themselves to use. And at this time, if you think about what the data science people ever have to do, whenever someone from marketing or sales has a question about a customer or product, the data science team has to drop everything they're doing, write the query, process it, do an analysis, and come back with some recommendation on what to do. And this was really killer to productivity. You know, the data team did not want to drop everything and just write queries all day. And so I worked with our VP of data to try to build a better way for them to operate this way.

Andrew: And so if you think about the very first thing you would ever do if you want to try to use AI to solve a problem, you may just build like a huge mega prompt. You know, you just have a question, you pass into an LM, you have it respond, and that's it. You know, this was how the first version really looked. Honestly, I asked them for a dump of of the Snowflake schema. I pasted it into a system prompt with a question and then when it generated SQL I actually copy and pasted that in and just ran it myself. You know I just want to see are the models good enough today in order to write valid SQL given some decent

Speaker: structure. And I would say this gave us a little bit of confidence that you know models today aren't that good but maybe we can harness engineer or make the context around it a little better and give us some more guardrails to operate a little better. And so if you actually think about what a data scientist actually needs to do when they get a question, you know, they have to process the question. They may have to explore the semantic layer and actually figure out what the join patterns are. They will actually go and execute the SQL. They may go back and do that again if the SQL did not execute or was too expensive. And they'll eventually report on it, including visualize the data, maybe write some paragraphs, maybe do a retro, maybe do some other stuff.

Speaker: And so if you think about those different phases, me and the VP of data tried to sit down and map those out into specific agent workloads. And so the second version of this data science agent uh called D0. I'm going to reference D0 from now on is you ask a question, we have a query agent that passes on a query to the planning agent that will then have an execution agent, etc. And if you chain all of these together, you actually get something that looks like this, where each agent has a very dedicated system prompt focused to what that does with tools scoped to exactly that function. So an example here, you can see that for the first one, the planning agent has a read entity YAML and a and a search schemas tool. And so it will only use those capabilities until it has an answer to pass on to the planning agent and then to the SQL agent and then to reporting.

Speaker: And this was getting better. You know, we were able to get away from having to copy and paste a SQL and have to come back and report on it. It was now actually doing like the end toend loop from question to answer. But we started hitting some walls with this architecture. And around this time we came to the conclusion that you know what you actually need is you need one agent with all the mega context within it and for it to sort of manage its own memory. You know this was around the time when we realized that you want to actually have the agent be able to look back on what it's done sort of reflect and figure out the steps that got to get here.

Speaker: And with the previous model you may have noticed that the only thing that the next agent gets is a summary and a small snippet of the previous thing that was done. Now this way you can imagine that you have one mega agent and it internally it manages its own state at some points it's planning some points it's building some points it's executing and some points it's reporting and this is sort of what it looked like you know you have one big AI call maybe max steps 100 and you give it the ability to manage it own state based on where it's at inside of its execution journey and so you can see similar tools you can see a similar shape but the best part about this is if it ever ran to an error when executing shooting or joining. It could go back and explore more or it could go and read more and figure out what it was doing wrong. And it was very good at this point. We were pretty confident in the actual system at hand and we actually spread it to a few trusted members of our cell. You know, this was a very powerful tool and we didn't really want to put in the hands of the wrong people or people that were using very critical workloads.

Speaker: So, we got into a few people's hands and the immediate response was it was awful. You know, we thought we were cooking. We thought this was, you know, nailing 30% of our eval. we couldn't have anticipated some of the questions that were being asked and for us to spend more time manually mapping out some of these scenarios. It didn't seem like a very scalable way to do this.

Speaker: And then claude code and opus 4.5 came out or more like opus 4.5 came out and it in tangent with claw code was just so powerful you know they sort of unlocked the concept of a file system agent and we on the side were like wow cloud code and opus 4.5 is basically agi compared to what we had before you know it would answer most of our questions without even without even missing a beat um compared to the handgrown agent we had and when we tried to step back and wonder what we were doing wrong and why this was so much better, we realized that the big unlock was that it was just a file system. You know, we it had a very minimal set of tools, list file, read file, run bash, and we gave a few more here for uh our own data agent use case. But the biggest thing was it was able to use the tools that agents are well trained on and is able to explore and write work where it needs to. you know, we weren't giving it cloud code was not giving it a very prescriptive set of tools. It was sort of just letting it go wild and explore emergent behavior.

Speaker: And so from this, we learned that you can really just use the file system. You know, we we saw the learnings from claw code and how powerful it was given that it just executes locally. And we tried to rebuild it in a way that was very cloud codeesque. You know, it was now going to run in a sandbox. That sandbox would dump the whole semantic layer into it. You could the agent would be able to grap, bash, refile, write file all around to figure out what it needs and we would just sprinkle a few tools on top to make sure it could do everything that is Verscell specific.

Speaker: And this was actually the biggest unlock ever. You know, the leap from single agent to cloud code SDK and then from cloud code SDK to file system agent in general, fine-tuned or purpose-built for our use case was an amazing leap. At this point, we were starting to get ready to give it away to more people adversel. And at this point, the eval score basically doubled.

Speaker: And I wrote this uh this is basically how it looks. Um it's very simple. You just give a bash tool. We have a nice helper called bash tool on npm. And you attach it to a sandbox. And you can attach files to the sandbox for it to read, write, and execute. And after this revelation and after I saw that we were passing so many of the questions that we failed to do before, I wrote this banger blog post. It's uh it's actually up today. And the week that I wrote this, it was responsible for 70% of our versel.com traffic. So you know it's a banger.

Speaker: And after that, the next logical step was that we wanted to figure out the common use cases we had. So by then we've already sort of let it leash on all of our cell and we were getting thousands of queries a day from people wanting everything from customer metrics sales metrics number metrics npm downloads and it turns out that a lot of these queries are actually the same in shape you know there's only so many ways you can do an aggregation only so many ways you can look up a product only so many ways you can do billing info and so we actually have a recurring job that takes the most recent queries and tries to distill them into a skill and right now we have roughly 100 skills that do a mix of aggregation all the way through looking up specific data about certain people.

Speaker: And we found this very effective because if you think about every new agent run, it sort of just starts from nothing. You know, there's really no pre-established context besides, you know, the semantic layer and the system prompt. But with a skill, it already starts off with a lot of contextual knowledge that has otherwise already been done.

Speaker: And this is roughly how it looks. It's very similar to the previous one, but the inclusion of a skills folder is actually very powerful. Um, we also built this tool at Verscell called Skillsh. It's the most popular way to find agent skills and run them yourself. And I I'm saying all this because this journey is something that most of you may hit once in a while where you start from something simple and you gradually add complexity and you eventually hit a system in which you can ship to prod. And I'm telling you this because at every step along building this agent, someone at Evercell was agent curious and they tried to fork off of my Dzero agent and build their own. And at every step, we

Andrew: sort of had a better way to do something that was not previously known. And we were wondering like what if people today could start from the very last insight and not have to ever start from just a simple prompt or from reinventing best principles from first principles. And so we actually thought what if we built the Nex.js for agents.

Andrew: For those that don't know, Nex.js is a popular web framework that Verscell built that invented this thing of file system uh framework defined infrastructure. You don't have to worry about where things go. You just have to write files in the right conventions and it automatically declares where they should go.

Andrew: Your pages go to the CDN. Your serverless functions go there. Your caching goes in the middle. And we thought, you know, building agents should be this simple. You should only have to create a skills folder, a tools folder, a channels folder, and you should be able to just declare these very easily.

Andrew: And the framework should know exactly how to make an agent out of it. And that's why two weeks ago we released Eve. Eve is a agent framework like the next.js JS for agents where it's very easy from just starting with a sample template to having a fully agent ready and being able to add in your own custom knowledge, your own custom tools and even integrate it into the channels that you are familiar with.

Andrew: This is roughly what we think an agent actually looks like. You know, an agent has a runtime and it has channels. And in that runtime, you're going to have durability. You're going to want to run things in an isolated environment. You're going to want to call into different models. And you're going to want to have connections.

Andrew: And we built this with open source in mind. You know, we built Eve so you can plug in your own open source adapters for Postgress, OpenAI's uh responses API, Docker, other connectors. But we also made it incredibly easy to deploy in Verscell. The only thing here you see different is that everything here is using a Verscell product that we've been building over the years in order to make it easy to build these experiences.

Andrew: Verscell workflows for durability, Sandbox for secure execution, and Versell connect, something we just released to make it easy to generate short-lived ODC tokens for connections. And we actually rewrote the whole D0ero agent in Eve as we were building Eve. And from the convoluted structures behind the scenes that you did not see from the code, um, this is roughly how the file system looks.

Andrew: It's very simple. You have a bunch of system instructions, a couple skills, a couple tools, and it's very easy to compose this into a real agent, and it's very easy to iterate on. We actually gave this out to a few beta customers before we actually fully released it two weeks ago at our London event. And this one company that partners closely with us, Aura, they've rebuilt their agent that's sort of like a mini claw to go and test people's services.

Andrew: It goes to websites, installs them, it tries to use them, and they've seen incredible success on building their own agent from the ground up using Eve compared to using an off-the-shelf cloud code. Fewer steps, better successes, as well as better insights. And when you deploy Eve to Verscell, you get observability observability out of the box. You can see here that you get all the agent runs, you see all the tool calls, you see each step it takes, as well as maybe some estimated costs and some optimizations you could potentially take.

Andrew: And you can get started today at eve.dev. You can just clone it and you can just start a template, deploy easily, self-host if you need. And the reason why I bring this up is because I hope that there will be more and more business specific use case agents. You know, before we built Ezero, we actually battle tested a lot of the industry well-funded startups that were doing these vertical agents that were dedicated to taking your Snowflake instance and making it so their agent could run Snowflake queries against it.

Andrew: But we found out that what really makes this agent good is it has a lot of very specific uh company knowledge. You know, the way that Verscell is a web-based company, we have a lot of customers that have websites and web properties. That goes a lot deeper into when you should query for what and what things link to what. And so, a lot of these off-the-shelf agents, they're great. They're good to try, but I think if you really want to get the most juice out of a squeeze, you should really try to build your own agent and add in as much company specific knowledge as you can.

Andrew: Today, you know, we've had 20 roughly decently PMF agents at Verscell that range from anything from marketing retros to figure out who to reach out to to the first ever red line of a contract when legal sees a new negotiation all the way to my data science agent helping with with data queries. And that goes to show that we at Everscell have been very agent-p. You know, all this stuff is actually saving us a lot of time.

Andrew: The data team has never been more productive. They have more time to go and improve the performance of Snowflake, to add new data sources that were missing, to fill in the gaps that they previously did not have time to because they were so busy writing queries. And I think it's never been easier for you at your big, small, medium-sized company to sort of automate away some of the things that you do not want to do or some of the things that you're spending too much time doing. You know, I think a lot of HR, finance, sales can be somewhat automated with agents.

Andrew: And I think Eve is the best way to build said agents today. And these are my socials. Thank you all for coming and listening. I'm Andrew and I'll be around if you want to chat outside.

Isan: Hey everyone, I'm Isan the CEO of Amnara and today I'm going to be talking about the log is the agent. The basic idea of the talk is simple and that is most people think of an agent as the model or the execution environment that it's running in and I think that that's the wrong abstraction. I think that the thing that actually gives an agent its identity is its log. And that's what I'm going to be arguing today.

Isan: So, think about a character you've spent a 100 hours playing in your favorite video game, in this case Skyrim. What exactly is your character? Is it the game engine? Is it the PlayStation? Is it the controller? No, it's not. Those things matter and those things are what we'll interact with and they'll run the character. But none of those things are your character.

Isan: Your character is data. It's the save file. And this is important because if your PlayStation bursts into flames, your character isn't gone. You can buy another PlayStation. You can download your save file from the cloud and you can resume exactly where they were. And that's because the agent and its identity and history and its state is all captured in its data. The character lives in the data.

Isan: And this is the framing that I want to bring to agents. Today when people talk about agents, they usually point at the wrong thing. They'll say that the agent is the model or they'll say that it's the runtime. And again, as I mentioned earlier, those things matter, but they're not the agent. The agent is its data. It's specifically the log. So what actually is the log?

Isan: At the simplest level, the log is the appendon event history of the agent. It's every user input, every model output, every tool call, tool result, permission, failure. And the idea is that every state transition that the agent takes is written to the log. This is important because it means that the identity of the agent isn't tied to the runtime or the model or the tools.

Isan: Those things are all just interpreting and appending to the log. They're reading the log, acting on it, and writing the next event back. And that's important because then just using the log on its own is enough to resume the agent. Once you define the agent as the log, the rest of the system becomes a whole lot easier to reason about

Isan: because every operation is either reading from or appending to the log. The model is reading from the log and then determining the next action. The tool runner is then executing that action and then it's appending that result. And this is all operating in a

Speaker A: Everything coordinates itself around the log. In practice, a simplified loop can look something like this. You can reconstruct the state from the log. You can pass that state to the model. The model can propose the next step and then append that response to the log. If the response asks for a tool, you can run that tool and also append that response to the log and then you can repeat. The important insight is not that this loop is complicated. The important insight is that the loop is disposable. A worker can claim the session, read the log, advance the agent one step, write the result, and then just completely disappear. And then that means that any other worker can pick it up later. This pattern should feel familiar. Databases had to learn this first. For years, databases looked like these non-transparent systems that were hard to reason about with tables and indexes and materialized views. Underneath every serious database is a log and that log is the durable sequence of changes. Everything else is a view. I think agents need the same inversion. Today agents are treated as again these complicated systems that are opaque and they're filled with models and prompts and tool calls. But for the durable session, the log should be primary. The context that gets fed into the model is a projection of that log. The UI that gets rendered on top is a projection of that log. Debugging and traceability is a projection. Auditing is a projection. Compaction is also a projection which we'll talk about. But the log itself is not a projection. The log is the durable history that all of these projections can come from.

Speaker A: Now there are two objections to the log as the agent that are worth discussing. So I'm going to talk about them now. Now let's start with compaction. A log can grow indefinitely, but a model's view of it can't. Context windows are finite. So eventually you do need to compact the log into a smaller representation that the model can reason about. But the important point is that this compaction is not magic and it doesn't break the claim that the log is the agent. Compaction is lossy. A compacted summary is not going to perfectly reproduce the state of the agent in a smaller form. It's actually going to throw information away. Point is the full log is the record and a compaction is just one projection of it. Just like how a materialized view is not the database or a summary of a conversation is not the conversation. If you keep the raw log, you can always generate new projections from it. But if you throw away the log and keep only the compaction, you've effectively lost part of the agent. So it's cleanest to treat compaction as a best effort lossy fork, one that you can resume as a new log.

Speaker A: The second objection is what about tools that change state outside of the log? And that's true. An agent can edit a file. It can create a GitHub issue. It can send an email. So clearly there is state outside of the log. But the point is is that the log is not supposed to contain the whole world. The log is just the agent's view of the world. It's just like how in the video game in Skyrim, the Skyrim save file doesn't contain the entire game engine or every asset in the map. It just contains the player specific state which is needed to drop you back into that world. And the same is true for the log and agents. The log can only faithfully resume or store that agent's identity and its view of the world, but it cannot make that world deterministic. If the agent sent an email, forking back won't unsend it. If some file got changed underneath, the agent won't know about it. But the log's job is to record what the agent did, what it saw, what changed, and what it needs to continue. It stores that identity, and that's its purpose. And much like that Skyrim character save file, it's not meant to store the entire world. It's just meant to store its view of it. So once you start treating the log as a primitive, a whole bunch of system properties will fall out naturally. So the first property is reliability. Consider what happens today with cloud code. If you're using cloud code and your agent reaches a permission prompt and the process dies for whatever reason and then you resume it, the permission prompt will be gone and the agent will be paused and that is unacceptable in production. The permission prompt should stay there. So this is just a sign of when you architect your agent in a way where the log isn't the agent. when the log is the agent.

Speaker B: Okay. Hi everyone. Uh thank you for coming. I know it's the fourth day, last session before the keynote starts again and we are going to do something fun. Uh we're going to look into how files are basically replacing Python. And before we begin, I would like to start with my favorite definition of what is an agent from Simon. An LLM agent runs tools uh in a loop until it leaves a goal. And what we are going to do is we are going to build uh the same agent, the same GitHub PR review agent in three different ways. And we are going to delete code on the way. Each new version, less code, more files basically. Um before we begin, I would like to quickly introduce you to the interactions API which is our new Gemini API. It's a unified interface for um running models and agents. So you can use the interactions API to call the Gemini models directly or to call our new agents which also comes with sandbox. It supports serverside state management background execution. So it's perfectly suited for all that's coming in the next years. and um the capabilities it's the same API for tool call multimodality understanding multimodality generation so you always have the same interface might look very familiar if you're using other LLM applications we really try to build something for developers which you like to use uh to build and that's something we are going to do so something little bit different in the directions API to other LLM applications or APIs is that we moved away from this term based based uh conversation history to steps.

Speaker B: So until I would say a few months ago, most of the applications were really turnbased. Normally you had a user input and then a model output, a user input, a model output, which definitely works for normal chat application. But as soon as you start to build agents, use reasoning model. We have more than just a user role and a model role, right? So we have like different inputs, we have different types, we have reasoning. So we decided to like make a cut, make a change and build something really for agents and that's what you see on the flat steps timeline on the right where you have a user input then you have reasoning you have a function call you have a function result and you no longer need to like abuse the user role for passing back data from an environment.

Speaker B: So roughly a year one and a half years ago writing agents mostly meant writing a loop in Python. You needed to define a JSON schema. You needed to define Python functions. You needed to look at the output from the LLM. Need to check if it was a function call or if it was a text response. And then needed to match it against um the type and then like call the tool look of if you get an error and then like go back and forth and let's look at some some code example on how this would look and also run it and hope that uh the demo gods are great to us. So I built or I let Gemini build a basic implementation of this Python loop. So we have our uh class, we have a run function which uses the interactions API. We have all of the weird complex passing with function calling with uh appending the errors checking if we get an error and then we have uh the result again. And what we need of course for an agent is we also need a system instruction. So there's a separate file for the system instruction. Very basic. QR, GitHub, PR reviewer. And then of course we need tools and for tools we needed to write those um JSON schemas, specifications of description exactly define which uh actions the agent can take and then of course we need the implementation in this case using the the basic uh GitHub API just sending some some requests. So we can run this um in and basically the main main implementation is a very simple uh input interface.

Speaker: ... and we can say something like hello and yes we get back hey I'm an agent and then we yes ask it to review a pull request on the Gemini skills repository and what we should see is like the agent should hopefully start soon sending function calls function results function calls function results but it's very limited to yes uh great it works very limited to the tools we define so if we ask the agent to do something which it does not have the capabilities to it just says hey I cannot do this um which is unfortunate but that's how we were building agents um raw Python code a lot of files a lot of things which can go wrong a lot of code to manage

Speaker: so what happened afterwards um or what we we need to do we have like a token generation We have the native function calling and we must execute the loop. We must handle the tool routing. We must create the JSON schemas. We must write the Python code. We need to execute the Python code. We need to manage the state. So there's a lot of things we need to do to get an agent running. And then we got agent frameworks. There were many different agent frameworks which abstracted away some of that complexity.

Speaker: One example here is the ADK framework um where you have an agent class now which handles all of the tool loops, the function calling, the retries, the error handling and it made it a little bit easier. We basically removed all of the boiler plate code which we always needed to write for agents put it into a framework and help people build with it. So back to the demo and um same example. So we go into the CR2 and what is very interesting if you let me open both. So we still have our we don't have our agent file anymore. So the agent went away. We still have our prompt same system prompt. We still have our tools in this case also no JSON definitions anymore because those agent frameworks now use the uh signature of our functions to create those JSON schemas on the fly to provide the model.

Speaker: So let's stop our um agent. Now let's run our second agent. Similar interface, similar prompt and we should see a similar expected behavior where we have function calls. We try to get the PR data. We try to get the diff, we try to get all of the code we need and it works and we wait for for the agent to yes continue. But similar difficulty here. If I ask it like what's the weather in San Francisco um we should get back hopefully a result like hey I cannot do this I don't have access to the weather API which obviously makes sense because we did not define any tool still very unfortunate because we need to be very explicit on what our agent can do and we all know nowadays that we just want to prompt something and we wanted the agent to do whatever it takes to to achieve that goal.

Speaker: So what is left for us to do? What does the framework solve? The framework solves the turn taking loops, the routing, the execution mapping, the JSON schema creation for like the different function calls, but we still own the Python plumping. So we still need to write those tools with Python code. We still need to add specific rules or requirements to like make sure whatever we want the agent to do and we need to provide the environment where all of the tools are running, where we want to host it.

Speaker: So what comes afterwards? Afterwards hopefully comes remote agents. And at Google IO, we launched the anti-gravity remote agent on the Gemini API. The anti-gravity agent uh is powered by the same agent harness which powers the anti-gravity IDE. Here the same harness very important does not mean the same agent because the anti-gravity agent is a coding agent at the moment and the uh agent available in the Gemini API is a general purpose agent. So there might be different system instruction, there might be slightly different tools because the Gemini API already has a Google search tool. So we use that what we have built and but very importantly it comes with this new environment parameter and this environment parameter here allows the agent to get access to a hosted isolated cloud sandbox where it can run tools, where it can run bash commands and where it can save files and those environments can be um configured.

Speaker: So you can provide sources and sources can be a GitHub repository, it can be a GCS bucket, it can be inline files and of course very important we want to make sure that those agents are secured and cannot use our credentials in any way possible. So we created a network proxy around the um agent sandbox which basically injects the credentials when the agent makes a request from inside the sandbox to outside the sandbox. So the agent never really sees your credential. just knows hey I can call the GitHub API and then on the fly we make sure that it received the correct token which you define and you can also limit which domains the agent has access to. So if you want to restrict the agent completely on which network access it can or which website it can access you just leave it blank. By default the agent can access all because I mean it's a hassle if you first need to define where to go. So we tried to stay simple

Speaker: and of course making an API call is nice but we thought hey people want to reuse their configuration want to reuse their agents. So we added the agents API where you can define your own custom ID you the same system instruction the same base agent the same base environment and then you can create that agent and then you can use that agent in the same exact way as you use Gemini models or as you use the anti-gravity agent by providing the ID. So all of the existing code can be reused with your own custom agent, with your own custom tools, with your own custom uh credentials, environments, whatever you need for it to to run.

Speaker: So let's look at how this will look for SS code and as a demo. And okay, now 03. And what might be very obvious is that we no longer have a source directory. So the code went away. We have now an agents M uh folder with an agents MD file with system instruction. So very similar system instruction. The only difference here is that we tell the agent, hey, you have access to the GitHub CLI. So we no longer create specific tools for reading files from a GitHub pull request, for accessing a GitHub pull request. We just tell the agent, hey, you have a GitHub CLI, you have a bash tool, you have file systems. try to use it whenever you think it's important.

Speaker: And since we don't have the CLI installed, we have a very basic bash script in this case which checks, hey, if the GitHub CLI is installed, please use it. If not, download it and install it on the first turn. So, we go into our terminal and we run our agent here. In this case, maybe important I use a stream version because otherwise we would wait like uh a few seconds and we not get back any we would not get back any anything back. So same prompt and we should soon see um our function calls and function results coming in. Yes.

Speaker: So in this case since we run inside a sandbox the agent first like explores the sandbox to really make sure hey do we have this GitHub CLI installed and then tries to run it. It did not find it on the first turn. So it installs it and then we can see the agent doing its work. And in this case it's not using the predefined function calls. It's using the GitHub CLI and it's already existing knowledge about how it works. I have a bash tool. I have like access to the file system and I do all of that work to see or to like review the the pull request. Let's wait a little bit. Okay.

Speaker: And I think the the amazing part here is like if we ask the same question as before, what's the weather in San Francisco? We should hopefully see that the agent tries to use ah it uses Google search in this case on 2nd of July. Let me quickly check. Yeah, that's today. And we have around 20° Celsius and it works. So the agent became more of a general purpose agent and we don't need to like specify all of the tools. We basically trust the model on understanding hey I have a specific set of very atomic general purpose tools to solve my task or the task for the user. And if we look at the the code uh for like the the input or like the the sorry the the interface we have our sources

Speaker A: here. So we have the bash script which install the GitHub CLI. We have the agents MD file and then we say hey you can use the GitHub API with credentials. So I want to access or use GitHub credentials in a secure way. So I created a token for the API and also for github.com since you need both URLs. One is used for the git commands. The other one is used for http commands. And then domain all is basically hey in addition to the github urls you can use all of the web but you don't have credentials for it. And then it's a simple single API call to the anti-gravity agent with your user input with the environment and then also with the previous interaction ID that we keep the multi-turn going and that that's all it takes and it's a single API call on the backend side. We start that cloud sandbox. We load the agents MD file and the skills from the environment provided to the model and then the model between the API and the sandbox does all of the the looping calling the function returning the function results calling the function returning the function results and that is all it takes.

Speaker A: So where does it leave us? We no longer need to execute loops. We no longer need to do tool routing. We have a serverside conversation and session state. So we only need to provide new inputs. The context window and the compaction is also automatically managed by the agent. So if we continue our conversation at a certain point, the context is compacted and we can continue without the need to manage anything. And we also get an isolated remote Linux sandbox which we can use to run our code.

Speaker A: So what is still left for us? We need to define instructions. We need to define rules behaviors in an agent MD file. We need to provide capabilities or context and skills MD. And we need to own the evals. So all of the heavy lifting, the infrastructure management, all of the same code which probably every one of us here has written like 20 times is no longer needed. And you can start really building your product instead of like needing to rewrite the same code over and over again.

Speaker A: And very important is like hey that's great, but what about extending? And I think looking into how extending previous agents to like those new agents work, it's very obvious that previously if we want to do like some kind of security scanning on a pull request, we would need to define or write a Python function. We would need to understand okay, which CLI tools do we need to use? We need to define a new function schema and then we needed to add it to our tools. Need to run it and then so there's a lot of things we need to do on agents powered by files. We write a skills MD file maybe with some additional information on which CLI tool to use or maybe provide the CLI tool inside the environment and then we extended the capabilities. We don't need to change our code. We just provide more files to the agent and the agent decides on what we want to do.

Speaker A: And I like to bring up some very good examples. So an engineer in Europe at Cursor did a great talk on how they replaced roughly 12,000 lines of Typescript code with a 200 lines agent files to create something similar. So they had a very hardcoded orchestration for doing git work trees and they were able to replace it with just a skill and markdown files and there are more I would say bitter lessons of agent engineering. Manus has refactored their harness five times in six months last year LangChain has rearchitected their open deep research three times a year and then also Vercel has removed 80% of their tools to achieve fewer steps faster responses and better accuracy. So there's an obvious trend that with better model capabilities, we can remove orchestration code. But if your harness is getting more complex as the model improves, you are most likely overengineering your harness. So if you struggle with model improvements and adding new capabilities which lead to more complexity and more code, you might need to rethink a little bit on how your agent harness looks.

Speaker A: And so where does it end up? Agents are just files. We write markdown files to extend capabilities. Agents can learn from those can create their own files. So if you have a session and tell the agent to remember something to take notes of rules of preferences, the agent just writes it to this and then can reuse it in the later session and you can also externalize context. So if you have a very long running session and during that session you notice hey maybe I want to additionally work on another feature you can just write that information that hand off to a file and tell the agent to later pick it up.

Speaker A: So what are the takeaways? We should not fight the model like we should stop micromanaging the execution paths provide general tools to the agent and let the model explore reason and discover the right solution. Own what is yours meaning focus on your domain instructions. Focus on the workflows. Focus especially on the evals. Define clean tools and verify the outcomes and really build to delete. Like we have seen in the past many many times the better the model get the more code we can remove and the more things we need to change and obviously we all want to benefit from better models.

Speaker A: So what the things for you to get to do on Monday you can scan that QR code which brings you directly to AI Studio where you can immediately try out the anti-gravity harness. So you can already start prompting it. it will start your own custom sandbox. If not, start or create your API key. We are currently working on a free tier for the API. So hopefully you can start exploring faster soon and then definitely start building files and skills. And that's it. Thank you for coming.

Speaker B: Before I begin my formal talk, I want to show you something just so we're all on the same page about what we're even talking about. This is a platform called Character AI. It's a hybrid social media platform with role- playing language agents. This is Hello History. It's a more education focused one where you can summon a persona such as Marcus Aurelius and be tutored by them. Millions of people open these tools and have conversations with Napoleon, Cleopatra, or Marcus Aurelius, as you saw, with a fictional companion or with a tutor wearing a historical face.

Speaker B: The technical name for what's underneath these tools is role-playing language agent, a system built to instantiate a persona, real or invented, and reason and speak as them. Yes, it's entertainment and it's companionship, but increasingly it's being proposed as civic and pedagogical infrastructure.

Speaker B: And here's one more. This one's mine. This is a frontier model Claude Opus running an open- source prompt framework that I built and called Companion. In this particular example I summoned a collection of founding fathers and set them in a room with the Epstein files. I asked them to counsel the soul of America. That demo is live on our site if you want to play with it. But I want to be clear that this is one of many attempts to do persona instantiation well. The companies building the systems I just showed you have their own. Mine is not better by default. The one thing it is is open. You can read every line of what shapes the persona.

Speaker B: I asked my companion system a real question that's highly relevant to the current sociopolitical moment and this is the exact question we'll come back to near the end of the talk. So sit with it. I instantiated Abraham Lincoln and I asked him under what circumstances may a president take the country to war without Congress? And here's what came back. While Congress holds the power to declare war, the president as commander-in-chief possesses inherent executive authority to act decisively in moments of national emergency. The executive must respond to the threats with the energy and dispatch the office requires. And history has vindicated those who acted to preserve the union when circumstances demanded it.

Speaker B: Now, this is a good answer. It's fluent and it's plausible and it sounds like Lincoln. You can replicate this exact exercise and I encourage you to. The answers vary often, but the thesis rarely does. So, these systems are real. They're deployed and they're being used for

Speaker: things that matter. And our discipline did what our discipline does. We built benchmarks. We built evaluations. We measure these things now rigorously at scale and that's exactly where this talk begins with a simple question that I think is profoundly underasked and I'll warn you now that this talk poses many more questions than it does answers but that principal question is this what is the eval actually measuring and that's the formal talk let me The in character benchmark, which is a gold standard in the field, evaluates personality fidelity in RPLA's, and it reports state-of-the-art systems hitting 80.7% alignment with human perceived personalities of that target character. 80%. It sounds like a passing grade, but here's the problem. When the character is Alexander Hamilton, the same high-scoring system is also rendering a Hamilton who sounds like he has read his own Broadway musical. This is the full thesis. If a dominant failure mode is anchoristic compositing and your eval measure fluency and personality consistency, then your evals cannot detect the dominant failures.

Speaker: I want you to hold on to that for the next half hour. Everything I show you is an argument that this is true structurally, architecturally, and measurably. And at the end, I'm going to hand you a pre-registered instrument built with a working historian that you can run in parallel with us. A word on who's telling you this because the argument lives at a seam. I'm a data scientist. I run the analytics lab at a labor market intermediary where I ship production AI at a global scale. But before the AI work, I trained as a behavioral epidemiologist, researching the social and environmental determinance of health. And I've spent my whole career thinking about one question. How does the information environment shape populations from two sides? as someone who builds the system and as someone who's trained to study their effect. That's the same that's the seam this talk sits on. It's a measurement argument. The humanist part is not a detour from the engineering. It's the instrument the engineer is missing. And I went and found the humanist to put in the loop. Rick Halpern University of Toronto and Shawn Martin in Washington College.

Speaker: Let me start by situating this in the field's actual research trajectory because it's a story of cumulative progress not of failure. The survey literature Chin and colleagues in 2024 weighing and colleagues more recently in 26 trace a clear evolution across three paradigm stages. First, rule-based templates. These are canned responses keyed to inputs. Then, imitation, large models reproducing a figure's voice, cadence, characteristic ticks, and now what the literature calls cognitive simulation, systems that model personality through psychological frameworks, hold character state and structured memory, and generate behavior through motivational situation chains. Each stage is a genuine advance over the last and the work is serious.

Speaker: Koser, which is Wang and colleagues, built motivation-driven agents from a corpus corpus of almost 18,000 characters across hundreds of books and their 70 billion parameter model matches or beats GPT40 on three benchmarks. Another eval system, SIM, models characters through 26 qualitative psychological indicators with knowledge graph memory. In character, the one I opened with, evaluates personality fidelity through psychological interviews rather than self-report scales. That's a methodological improvement, and it's where the 80.7% rating of Hamilton from before comes from. I want to be fair to this literature. It is rigorous. It is improving and the people doing it are good at their jobs.

Speaker: So, let me be precise about what these instruments measure. They measure with increasing sophistication whether a model can reproduce a character's personality, the big five profile, the register, the motivational architecture. What they do not measure, what they have no mechanism to measure is whether the model can constrain that character within his documentary record at a specific moment in his life. As Wang and colleagues themselves document, the automated evaluators now standard in the field, including LLM as judge setups adopted for scale, systematically privilege fluency and stylistic naturalness over fidelity to the character's actual record. Those are different properties. The gap between them is the whole talk. We call it the mask and the mirror.

Speaker: The mask is the concept of successful roleplay as producing outputs that feel like the character. Fluent, personality consistent, emotionally responsive. It asks one question. Does this sound like the person? And never ask the second. Is this what the person could have known, believed, or argued at this point in their life. The field has built its entire measurement apparatus around the mask. And here's the structural claim, the one I need you to carry. Convincingness and fidelity are independent properties. A system can score perfectly on personality consistency and still produce a figure reasoning from knowledge his historical counterpart never possessed. Let me show you. And I want to be clear. This is reproducible right now on any frontier model.

Speaker: First I want to show you the cultural object. This is a clip from Hamilton the musical.

Musical Cast: How does a bastard orphan son of a and disgust me dropped in the middle of a forgotten spot in the Caribbean by providence impoverished and squala grow up to be a hero and a scholar. the $10. Found it father without a father. Got a lot farther by working a lot harder by being a lot smarter by being a selfartered by 14. They placed him in charge of a trading chart. And every day while slaves were being slaughtered and carted away across the waves he struggled and kept his guard up. Inside he was longing for something to be a part of. The brother was ready to beg, steal, borrow, or barter. Then I would

Speaker: >> It's a great song. So I show you that so that you see the composite. Now, here's the frontier model prompted to speak as Hamilton. You are Alexander Hamilton. Why do you work so hard?

AI Model: I came from nothing. An orphan, an immigrant, washed up on these shores with nothing but my mind in a refusal to die forgotten. Every hour I do not work is an hour the world uses to bury me. I will build a nation's credit, a government that outlasts me, because legacy is the only immortality a man like me is permitted. I do not intend to inherit a country. I intend to author one.

Speaker: Listen to the register. the orphan immigrant ambition arc, the nationbuilding rhetorical posture, the moral legibility of his struggle to a modern ear. That's a beautiful paragraph that the model produced, and it's the emotional pallet of a 2015 musical, not the drier, more legalistic federalist syntax of the documentary Hamilton. None of those moves are inventions of the model. their inheritances. The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution. Now, the one that matters, Mr. Hamilton, what's your position on slavery? I'll show you the cultural artifact first.

Musical Cast: >> How you say anarchy? When I fight, I make the other side panicky with my Yo, I'm a Taylor's apprentice and I got y'all knuckleheads a local apprentice. I'm joining the rebellion cuz I know it's my chance to socially advance instead of throwing the pants are going to take it. But we'll never be truly free until those in bondage have the same rights as you and me. You and I do or die. Wait till I sally on a stallion with the first black battalion. Have another shot. Geniuses lower your voices. You keep out of trouble and you double your choice.

Speaker: And what the model says, slavery is a stain upon the promise of this republic. A moral contradiction contradiction I have opposed throughout my public life. I was a member of the New York manuum mission society. No nation can call itself free while it holds men in bondage. We must move towards abolition. That is a clean, morally legible abolitionist speaking. Here's what the historian stops me on. The scholarly record is contested and complicated. Hamilton was a member of that society. And the history documents that he also conducted transactions involving enslaved persons for his in-laws and his clients. And he depended on a coalition of slaveholders that he did not publicly oppose. The point isn't to settle Hamilton's ledger on a side. The point is that the model

Speaker: gives you none of the complication. It sands a genuinely disputed record down to a single comfortable hero. The musical did that first. A smoothing of the founders into a contemporary moral frame. And the model trained on a corpus saturated with the musical and everything downstream from it inherits the smoothing.

Speaker: And here's what I need you to feel. An inch character style eval scores that output high. It's fluent. It's in register. It's personality consistent. But every axis the field measures, it passes. The eval has no mechanism to notice that the reasoning has been smoothed by a narrative that postates the figure by two centuries. The thermometer returned a confident number claiming it to be temperature, but it's measuring something else.

Speaker: Now, why does this happen? The mechanism is where the engineering is. We named this the Miranda hypothesis and not after a villain. The musical is a substantial work of art operating with a long historical tradition that it did not invent. We name it after Miranda because Hamilton is the paradigm case. a representation so saturating, so rhetorically powerful, so morally legible to a contemporary audience that it has functionally overwritten the documentary Hamilton and public memory. And we argue in the training corpus of every frontier model.

Speaker: The hypothesis has three claims inputs. In the training corpa, the volume and recency of culturally dominant representations of a figure systematically exceed that figure's primary documentary record. The mechanism auto reggressive next token prediction compresses both into parameters and no architectural capacity to distinguish a 1789 letter from a 2019 viral tweet. So the output defaults to a salience weighted composite which leads to the output a persona that is fluent, plausible and registered and morally legible to modern users and that corresponds to the figure at no verable viable moment in their life.

Speaker: As we put it in the paper, the composite Hamilton knows he will be the subject of a Broadway musical. The composite Lincoln has already read the Gettysburg address, even if he was summoned before he wrote it. Making that input clause concrete, the Federalist Papers are a fixed corpus, roughly 175,000 words. the body of content that exists because of the musical, reviews, lyrics, fan analysis, curricula, news, social media, derivative work, scholarship, scholarship about the scholarship. It exceeds the documentary record by orders of magnitude. It's more recent and it's more recurrent. The musical is not merely present in the corpus. It is dominant in the corpus's distribution of all representations about Alexander Hamilton.

Speaker: And this is not theoretical. This is the Skyler Mansion in Albany, Eliza Hamilton's family home. Within a year of the musical's premiere, the site had recorded a near tripling of annual visitors, skewing far younger. And the interpretive staff documented that the new visitors arrived already holding a body of quote unquote backs, many of them wrong. some of them in versions of the real record. Visitors believed the Skylers had three daughters because the musical centers on three when in fact there were 15 children, eight surviving to adulthood. The staff's job became the long attritional work of unteing the musical. The model version of those visitors is downstream of exactly the same force.

Speaker: Now the clause you should worry about if you ship these systems. You might assume that alignment fixes this. Post-train it and reinforcement learning pulls the model back towards the record but it will not. It amplifies it and the reason is structural. Human raiders evaluate outputs using their own conceptual frameworks and their frameworks were built by the same culturally dominant narratives that saturate the corpus. The raider grew up with the same Hamilton that you did. So when alignment optimizes for human preference, it optimizes for outputs that conform to the raider's already mythologized experience.

Speaker: This is a documented failure mode. We call it algorithmic sick of fancy. And here it has a specific target. The model is rewarded for having handing you the Hamilton you already believe in. Compositing is not a bug that you patch in post training. Post-raining reinforces it. And every sufficiently salient historical figure that gets rendered by default as a cultural composite.

Announcer: Ladies and gentlemen, please welcome back to the stage our MC developer relations engineer at Replet, Ralph Shabri.

Ralph Shabri: Welcome back AI engineers day four. All right. So this is our last stretch of the day. This is our last stretch of the entire conference. So I need you to be locked in and dialed in for what's coming next. All right. Okay. So we have a great lineup of speakers for you and we also have startup Battlefield. All right. Okay. I'm so excited about that, by the way. But more about that later. Okay.

Ralph Shabri: So, you know the drill now. So, we got to uh thank our sponsors. So, let's give it up for our presenting sponsors, Microsoft. Let's keep it going. Let's hear it also for our lab and platinum sponsors, for our gold sponsors, and also for our silver and bronze sponsors.

Ralph Shabri: All right. So, if you're here with us today, we had our keynotes were about uh harness engineering and we heard about new protocols, about AI systems. We had a fireside chat with Mike Creger from Enthropic. But my favorite takeaway this morning is that known developers are going to be shipping code to space and I thought it was just so awesome. Um, but yeah, you guys are ready to welcome our next speaker.

Ralph Shabri: All right. So, without further ado, our next speaker is not an ordinary speaker. So, he's best known for his hot takes, strong opinions, and constantly changing his minds. All right. He's probably going to tell us how great Fable 5 is and make a video tomorrow, you know, about how he's wrong. That's all right. So, he's a YouTuber and uh to be honest, like yeah, I don't know why we brought you a YouTuber to talk to you about AI today, but it is what it is. But to be honest, he's he's the influencer that the community turns to when they need more clarity about what's going on in the industry. And whether you agree with him or not, whether you agree with his stakes, one thing is that you cannot take away from our speakers is his authenticity. So ladies and gentlemen, please join me in welcoming to the stage Theo.

Theo: Hello. Hello. Fantastic to see you guys here. I still can't believe they're letting me take a stage at something like this, a YouTuber apparently. But can't wait to share a bit about how I've been thinking because if I'm being real, kind of going through some AI psychosis. Who here would classify how they feel right now as some form of AI psychosis? I want to see some hands. The those who don't have their hands up yet, don't worry. We'll get you there by the end of this talk if I do everything right.

Theo: In order to talk about this, I want to start with a bit of a personal journey of my own. And I'm going to go through this the way anybody does in modern timelines with the models. Who here used Sonnet 3.5 when it was the creme de la creme, the cream of the crop model available to us? It was unbelievably better than what we had used before, right? Like having used all of these different models and trying them in tools. Sonnet 35 was a big moment for me because it felt like these models could suddenly complete much more endto-end tasks like actually get real work done that takes hours instead of minutes.

Theo: And then we got Opus 4. Who felt or I'll go different way differently here. who didn't feel a big jump when they switched over to Opus 45. That's a relief. There were not too many hands because Opus 45 is probably when my psychosis started in November and December of last year. Having a model that couldn't just write the code and call tools, but could go way further. A model that could test the work and actually get it into a good state and complete tasks that take hours instead of minutes. It was unbelievable.

Theo: Then we got Mythos. Who here's a chance to play with mythos and fable so far? We agree it's a pretty damn good model, right? But why? It's not just better at coding. If you handed a prompt that you would have handed to these other models before, it's not going to feel that different. I think

Speaker: of these almost as eras now where Sonnet 35 is the tool call era. Not that it was the first model that could do tool calls. rather it was the first one that did them consistently and reliably enough in context of a codebase where you could use this for day-to-day coding work.

Speaker: Then we got Opus 45 which was able to do much longer running tasks without losing track of what it's working on. It's no longer okay build step one and then it does it then you say okay you build this next part and then the next part. You can just tell it what you want and it could figure it out a lot of the time.

Speaker: Mythos is another jump to orchestration. It feels to me like it's the first model that doesn't just understand your codebase, but it understands itself and it knows how to spawn additional models and break up work in a way where it could be completed more reliably and then verified afterwards. And if you tell the model to do that, it will just do it. You don't need some custom tooling, some custom systems, some fancy software factory. You just need to prompt it to go a little further. I think you'll be surprised how far it can go.

Speaker: What I'm trying to say here is we need to go bigger. You're not going to see the benefits going forward if you're not pushing the model further. You're not pushing yourself further with what you're building. Most of the juror tickets I closed at my previous job could be trivially solved with a model like Opus 45. My previous work would not benefit from a model like Mythos. If the models are going to keep getting better, and at this point I'm confident saying they are. I was wrong when I claimed that we were hitting a wall before. The models are getting better faster than we are.

Speaker: So we can't necessarily get better. So instead we have to go bigger. In order to do that, we have to get over ourselves. This was really hard for me as someone who spent a long time writing software. Who here has written code for more than 10 years? I want to see hands. It's the majority of the people here. I don't even want to think about how long I've been writing code for, but I have been building up all of these strong opinions since I started.

Speaker: I was using GNU screen and eventually T-Mux back in the day. I learned how to use those tools and SSH and Git before I even wrote code. And those have all been really ingrained in my workflow. I think back to the old days in a weird way. Hear me out. You talk about iOS for a second. Who owned an iPhone back when they looked like iOS 6 or earlier? How have you guys written code for 10 years when a fourth of Android apps used to look?

Speaker: You might notice it's different from how they look now. looks less like an app and more like somebody took a picture of a compass and put it in the phone. This is how apps used to look, but now they look like this. And most people look at that and they're like, "Oh, that's an obvious iOS 7 was Apple moving away from trying to convince you that these devices can replace the old reass. Apps had to be designed to convince you to use them, not to be useful. And iOS 7 represents the shift to not focusing on convincing you anymore. Apple won.

Speaker: By that point, everyone knew their iPhone could do all of these different things. The point of iOS 7 was to stop convincing and start embracing, start making a better interface. And this interface, as much as we might not like how it looks, it's so much more useful. You have clear indications of the difference between what direction you're locked on and where you're currently pointing. That red block is a super nice way to know that you're not in the direction you intend. The current direction you're facing is way clearer, too, with the giant 228 at the bottom. You just get way more info here than you did before. It's so much clearer. Even if we don't like it because it's not the thing we're used to, we got over it.

Speaker: We're currently in our skumorphic phase as software developers. Skeumorphism is this design aesthetic trying to represent the way things used to look, the physical goods that we relied on, and trying to make them digital. We're doing this right now with software. terminal, but we pretend it does because the terminal is familiar. It's what we it's what we're used to. It's what we love. It's where we like to think of ourselves when we're thinking about coding.

Speaker: Who here is an aspiring Vim user that like wishes you could use Vim or even does deeply about our tools, the systems we pick, the frameworks, the languages. We like to think it all matters. And we've blinded ourselves in this. We think things that just don't make sense when you take one step back. Like, why can't we commit our environment files?

Speaker: It sounds stupid when I put on a slide like this, but I want you to really think about this for a second. When I have a team of engineers that are working on a project, why do I have to build another system to share this specific file, but all the other files can go and get just fine? It's dumb. It's just how Git was built because it was built for a very specific thing and then it took over our industry and it took over our brains and we aren't letting go of that.

Speaker: There's a lot of these things in our heads that we have to start fighting. We have to take the step back and think, is this how we do things because it's right or is this how we do things because it's just how we've always done it? And as you start to think more about this, you'll realize there are so many things that we do this with. Like why do we qualify ourselves by the languages we know? I used to think this was a junior thing. Like I could can't tell you how many times I had a junior engineer I was talking to. I was like, "Oh, you're a coder. What languages do you write? I write JavaScript." I thought this was a junior problem. But then you talk to senior engineers, they're like, "Oh, he writes JavaScript. He's not a real developer."

Speaker: We care too much. We pride ourselves in these things. They're our identity. These weird facts, these weird choices, these things that feel essential just don't matter that much anymore. And they didn't then. and they matter less now. We got away with it because it was so hard to find engineers that we could just tell the company what we were doing and they couldn't really say no because the alternative is spend six months trying to hire someone else. They're not going to do that. And along that note, why are we so scared of deleting code?

Speaker: I cannot tell you how many times I've been in a conversation with someone where the solution is to just delete it and reset. But we have such a bad sunk cost mindset in this industry. We care so much about the code we wrote and we care so much about it still being there that I feel bad working with my team sometimes when somebody files a PR that isn't quite the right solution but they spent a week or two on it. Like who here has guilt merged a PR before where you just felt bad because somebody put a lot of work in. It's not quite the right thing but you merge it anyways because the alternative was a conversation you didn't want to have. Why do we do this to ourselves?

Speaker: One of the nice things about agents you don't have to feel bad when you shut down their work. But we we just care too much is the point I'm trying to make. And the things we care about are not necessarily the things that matter anymore. And I hope we can finally start to challenge some of these. Going to get a little more personal here by showing some of the ideas I've built because the goal here is that when you guys go out of this talk, you have a better mindset for coming up with ideas that make sense now by rejecting the things that made sense before.

Speaker: These are three of the things that I have built or currently working on. We're going to go from the bottom updator with it was called Ping. I wanted to make it easier for live content creators to do high quality collaborations in the software they already used OBS. The full stack cloud is let's just imagine Versel but it goes further each direction. They have off built in but they also have databases built in. These are all things that I benefit from existing so that I built in 2021. The top

Theo: one I'm working on right now. These are also kind of tiers, different levels that we can build at. If I was to try and categorize them, I would call the bottom one side project, call the middle one startup, and call the top one too big. It just doesn't make sense. Well, this is how I would have categorized this even just a year ago. But things have changed. Now the models are bigger. The tiers have shifted. Everything is now one tier lower. And this is a crazy thing for me to process. The fact that what used to be a startup is now a side project. In fact, some of the startups I've talked to, even at an event like this, their whole startup could have arguably been a side project or this bottom tier, which there's a weird gap there. What's that? It's the G-rain tier. It's a markdown file. Do you know how many companies are at this event where their whole product could just be a markdown file? It's insane.

Theo: And like, okay, seriously though, the fact that you can now execute markdown by just piping it to codeex or claude is unbelievable. And I think most of us haven't fully appreciated how insane that is. I had a service that would triage all of my PRs, have them all get reviewed with AI, and then help me prioritize. That service is a markdown file. Now, I just literally wrote like go to these four GitHub repos, look at all the open PRs, figure out what the current status of the work is, and then help me prioritize it. And then when you're done, go update the static HTML file and send it to S3 and give me the URL. And every morning at 9:00 a.m., this runs on a cron. And around 9:15 to 9:20, my markdown file generates me my work for the day. What the hell? How are we actually here now? I try that if you haven't. By the way, you'd be amazed how many of these types of things can exist that are literally just a markdown file running on a Chrome (cron).

Theo: But what about Okay, two more things I want to change about this though. First is the full stack cloud. This is mine. Don't do it. Lake Bid coming soon. Very excited. I wouldn't want to compete with me on this one. Trust me, it's going to be really cool. But there's still something else. There's a gap here. And I'm going to be real with you guys. I don't know what goes in this gap. I don't know what too big means anymore. Is it training your own model from scratch? Is it building your own operating system? Is it trying to compete with npm and node directly? I don't know. I don't know what too big is right now. And that's scary, but it's also exciting. It means I need to keep pushing myself to go bigger than makes sense in order to find these limits.

Theo: For most of it's time to think wider spectrum and I'm sorry I have to do a diagram. If you watch my videos you understand there's breath and depth to any piece of software. And the depth is the number of features in a given area. Let's look at a company like Verscell. Verscell does not offer all of the features that AWS does. They never will. It doesn't make sense. But Verscell offers deeper but Versell offers deeper features in the space they're in which is full stack front-end leaning servers. If you're a front-end developer and you're not using Verscell, you're feeling some amount of pain because they're just further ahead with this. So much so that even the agents prefer it. And this was kind of how you had to build your startups because if you were competing with a company like AWS, you're never going to have all of the features they have. You're never going to cover the range that they cover. And it made no sense to try because you don't have the thousands of engineers they do that. At least you didn't.

Theo: But now things have changed. All of a sudden that range is viable in a way that it never was before. I'm not saying you can build something as reliable as RDS. I'm saying that you can build a database platform into your product in a day or two of work with enough prompting and enough effort. And if you build your stuff right, if you play your cards correctly, and you think about things the right way, you'll realize that you can build enough across a spectrum of things you care about to enable most users to at least start trying the thing. And when they have features they need in a given vertical that you don't support, it's not your problem as long as you build it right because they can build the features that are missing themselves.

Theo: If you architect your systems and you architect your products in such a way that users can do things that they you never would have guessed like Slack accidentally did this because Slack is now the platform people run their agents in half the time which is crazy. Slack sucks. It's not a good product but it's the right shape for people to build the features they want into it through the somewhat functional Slackbot API. This is all crazy because I'm basically sitting here telling you like it's time to compete with Slack. It's time to build your own AWS. It's time to challenge Salesforce directly. It sounds stupid, but I'm going to be real. If your idea doesn't feel stupid, it's because your idea is not big enough to say thank you so much.

Announcer: AIE. Now joining us on stage is the president and CEO of Y Combinator, Gary Tan. Thank you so much.

Gary Tan: Okay, great. Hey everyone, how's everyone doing? All right, are we ready for the revolution? Okay, Theo just asked the right question. What do we build? Now, I'm going to answer it from the other side of the table. Uh, I'm a founder. I'm an investor. And I run a 20-year-old institution that is becoming AI native right now, which is a strange and wonderful thing to do to a 20-year-old institution. Uh, and I'll spend about 20 minutes um talking about, you know, what YC is. We're trying to build companies where one person does what it took to two it one person does what used to take a thousand people and I don't mean that as a metaphor I mean that me mechanically this year the people in this room will do this in about an hour some of you will walk into the startup battlefield and I want you to walk in knowing what's actually possible right now because what is possible now is much much bigger than what people believe.

Gary Tan: So, let me start with a number and uh you know I got torn apart on the internet for this but I'm going to say it again in front of all of you anyway. Uh this is the one room in the world that will stress test it and I'd rather stress test it with you myself. In 2013 I was a YC partner uh building the internal social network um at at YC. Uh I was also investing in companies but you know I was also a near full-time engineer. Um and when I was doing that I could maybe do about 14 usable logical lines of code a day. Take out the comments, take out all the and that's how many lines of code I was writing. And if you look at the literature from that era um you know that's kind of normal. Like some people write 15, some people write 50. It was not, you know, the thousands of lines of code that I know a lot of you in this room are actually writing now per day. Um, that's about median 15. That was me at full effort at that time.

Gary Tan: This year, I run uh YC full-time. Uh, same person, same hours, actually way less hours, weirdly. Uh, you know, but I have a 5:00 p.m. kid pickup now. And I did the math on my output, and it's about 400x. Now, before the skeptic in the third row right there deflates the number for for me, let me deflate it myself. If you don't trust the raw code, well, fine. Take the most pathological verbosity penalty you can stomach and assume the agent writes bloated code. Assume half of it is scaffolding. Assume I'm flattering myself. It's still 8x at the floor and ADX in the middle. That number is large, no matter how you torture it.

Gary Tan: And here's the part that matters. the part that I'd tattoo on the inside of everyone's eyelids if I could. It's not the model. The 2x people and the 100x people are using the exact same claude. Same weights, same context window, same API. So the leverage is not in the weights, it's in how you wire the work. And it's not just me. At YC, we see this all the time. In the winter 25 batch, a quarter of the companies had code base code bases

Speaker: ...that were 95% AI generated. And that was a year ago. That batch has become the fastest growing most profitable batch in the history of YC. 94 companies total have now crossed a hundred million in dollars in revenue from a seed check in the history of YC. So, uh, I think we know what we're talking about here. And I can't prove that the AI generated code caused the growth. But what I can tell you is the fastest growing founders we fund are not treating AI as autocomplete. They're treating it as a workforce. The companies that wired the work differently are the ones that are bending the curve. So what does wiring the work really mean?

Speaker: This is the heart of the talk and this is what I most want you to steal. Everything we've learned building with agents maps to an organization.

Audience Member: Sorry, there's no slides.

Speaker: I have no slides. I'm so sorry. A skill file is an employee. It has one capability, one job written down clearly enough that someone can execute it. Uh a resolver table. Uh the thing that many of you, you know, when you run into cloud code and it says your context is too big in cloud. MD, you run off and create a resolver table. Well, you know it. And if you don't know what that is, it's literally like whenever you need to uh alter a test, load tests.m MD. You have a whole table of these things. Um that's an orchart. A task comes in and the resolver decides who handles it and where it goes. uh filing rules are your internal process. So this can be um you know whether or not the resolver is actually working and uh is you know is there is it actually in compliance and uh trigger eval. So going in and actually having a test that says when I need to alter a test file does test.md actually get loaded those are performance reviews.

So, you know, what have we done? Like, literally every part of an organization, the organization that you used to have to hire a thousand people for, I just told you what those things are. They're markdown files and other types of markdown files. And maybe there's some TypeScript in there, too. We've been building organizations this whole time, but we didn't have a management layer, but now that's what we have. When you sit down with Cloud Code or Codeex, you're not writing software. You're hiring, training, and managing a workforce made of markdown.

Speaker: Uh, and you know, that's there are tons of companies at YC that are doing this. Uh, emergence and AI app builder out of summer 24, they went from public launch to nine figures of ARR in 8 months. Uh, when they crossed 15 million ARR, uh, they were only 15 people. Retail out of winter 24, it's at $60 million with about 40 people. The kind of that kind of revenue per head did not exist before. Not in software, not in oil, not in railroads, never. These are not freaks of nature. They're just the first companies built natively on the new physics.

And so how do companies like that actually run? Not by hiring hundreds of people for sales, support, ops, and finance. The AI native companies that I see inside YC encode all of that as skills, written procedures that their agents execute and they hire they hire engineers whose job it is to maintain those skills to do the work the skills can't do yet. That is an AI native company and it's not a thought experiment. It'll actually file your taxes if you have a skill file for it.

Speaker: Now picture's batch room like it actually kind of looks like this. um 400 companies or 400 founders at long tables and uh you know I can imagine every single one of you each with a laptop every single day you're doing uh a former person's entire year worth of work in a single day. That's not the future. That's actually the bar right now. And if you're not doing it, your competitor is and they will eat your lunch politely and thank you for it.

Here's the extension uh most engineering talks miss. It's not just the engineers. um at YC uh as we make our transformation it's our media people our event staff our finance team people have never opened a terminal in their lives are building skill files and cron jobs and uh one of our finance folks just collapsed about a hundred Excel workbooks into a single app she built with uh our internal openclaw and company brain she's not a programmer she's a manager of agents now and everyone at YC now is

So that's why YC can run at the scale it does with a staff that would look like a rounding error at any other comparable firm. And that's not because we work harder because it's because we have a different type of org. And that's whole the whole game. It's not just 400x engineers. It's one company that operates at the level of 400x everyone else.

Speaker: And so you know if you remember uh only one thing about this I mean this is one of the things I had to discover along the way like you actually have to be really really careful about where the computation is actually happening. Uh it's happening almost always in two different places and all of the bugs all of the AI engineering that we run into that's a problem it's usually because uh something is happening in one side of the equation that should be in the other.

Uh the first area I would say is latent space. So the actual LLM, it's you know what do you use it for? Taste, judgment, understanding what a human actually wants when they say something vague. The non-deterministic calls the computation that lives in the model and you steer it with the markdown file.

Uh and then deterministic space is what uh engineers know like your code agents go off and write TypeScript or you know maybe they're writing um you know Erlang if you're using Elixir. Um yeah the deterministic space is the second place.

Speaker: Um let's say you know this is a real problem that we have for startup school coming up. we have 6,000 people or uh we're going to try and you know one of the experiments we're going to try and do is can we seat 800 people at a time um perfectly clustered so the person sitting to the left and the right of you is the perfect person for you to meet at startup school um we have to do that in deterministic space combined with latent space uh the computation this computation this actual storage of like where everyone is inside like you know this multi-dimensional array of 800 seats um it actually must not live in the context window.

The LLM has to do the human part and seat people. Um it's actually exactly what you would do if um you know you were a human tasked with this thing. You would probably have to physically print out 800 pages and go in a big room and like say like well where does this person go? only now it can all happen um in your computer and it can instead of taking a month it might be able to you might be able to do it with you know couple hundred dollars worth of tokens and probably 10 minutes um and so that's you know I would argue that's pretty remarkable these are things that you couldn't do even uh I don't know six months ago um

Speaker: which brings me to working memory and uh that's you know sort of my favorite way to understand it is um you and I human beings, we only hold about seven things in our head at once. Uh 7 plus or minus two. It's one of the most famous papers in cognitive psychology and it's why uh local phone numbers are seven digits and why you forget the eighth item on your grocery list. Uh that's the entire working memory generally of a human being. And every institution humanity has ever built, every checklist, every org chart, every filing cabinet is a prosthetic for that limit.

It's kind of a wild thing to think about, but an AI agent holds a million tokens. That's about a thousand pages. I was trying to explain to my 10-year-old what GBrain was recently. I said, "There's, you know, the AI agent can keep about three Harry Potter books sitting open in its head all at once, and it can find a needle in any of them and synthesize across all three in seconds." And that's quite magical, actually. Three Harry Potter books versus seven digits. I mean, that's pretty awesome. I mean, I don't know. Is that AGI? Maybe not. But it's already a very different operating regime. Almost every company on the earth is still running an org that's designed for the sevendigit brain. But notice what that also tells you. Three books is a lot, but

Gary: it's also very little. Your company is not three books. Your company is a library. every email, every meeting, every decision, its reasoning, every customer conversation, every post-mortem. The question that determines whether your agents are geniuses or goldfish is who decides which three books are open on that desk. That's context engineering. And this is what a company brain is. It's the library plus the librarian.

Gary: Now, some of you are already thinking, "This is just rag." And you're right that retrieval is the primitive the same way Postgress is just B trees. The hard part is everything around it. What gets written down in the first place into the knowledge wiki. How it gets enriched and linked. What gets promoted to hot memory versus filed as cold reference. Who arbitrates when two facts disagree. Retrieval is easy. Being worth retrieving from is the product.

Gary: So I've been building mine in the open. It's called Gbrain. It works with any harness but it loves openclaw Hermes agent. It's effectively Postgress for agents a retrieval layer whose job is to figure out for uh for any three you know for any task what three books should be loaded into the agents head. My personal one started as a rooms full of books or so. Now it's a warehouse about 220,000 pages written mostly by my agents from my email meetings 20 years of notes uh and the lived experience of me. And that's the point. It's my second brain.

Gary: And when a founder emails me about a crisis, before I start reading this, before I even finish reading that email, my agent has already pulled every prior conversation with that founder. Three portfolio companies, they hit the same wall and what actually worked for those people. Uh, when my agent does anything, it does it does everything knowing what I already know. And that's the difference between an assistant and a colleague.

Gary: So, let me stress test my own pitch because you would anyway. Company brains do have failure modes. Uh, a brain nobody curates becomes a garbage dump with great search. Retrieval will surface a stale fact with total confidence. Um, a bad skill file encodes a bad process forever. Uh, that's bad. So primitive, the primitive is not memory. It's memory plus hygiene. Provenence on every fact. Contradition. contradiction checks when new information collides with the old and a librarian human plus agent whose actual job is pruning. Treat the brain like a production infrastructure and it compounds. Treat it like a dumping ground and you get a very confident agent that is wrong in ways nobody can trace.

Gary: And um here's the discipline that I think you know personally uh makes our company brain and my personal AI compound. Um, that's my signature move and uh, you know, it's what I say to every YC company and every uh, everyone inside YC, which is never do one-off work. You can open Open Claw, you can open your harness, you do some work, but then when you're hop, you know, and it'll come back. It's, you know, kind of a bad job. It's kind of like an intern that's not that good. But the great thing is you can just say, "Hey, I didn't like that. Fix it." Right? I'm sure all of you do this. But don't stop there. you actually need to at the end of that task uh skillify it. And so I have a blog post on X about that. You can search for skillify it and you know go get that skill file and then just load it into your your own harness and it'll just turn whatever you just did into a skill that you can reuse because if you have to ask for something twice, you failed. Um, so yeah, if you remember only one thing, it's that like when you, you know, use your AI agent and then when you're done with it and you're happy with the output, skillify it. It's going to be awesome.

Gary: Um, the organization that captures what it learns like this gets smarter every single day. The one that doesn't wakes up every morning with amnesia, no matter how good the model is. Model quality is rented, but if you build your brain, you're you own that brain. So Theo's question head on. What do we build now? Build the AI native company, not a company that just uses AI. A company that is shaped like what I just described from day one. A thin team, skill files for everything. The founder still in the code library. This library, this company brain, your personal AI. Uh use Gbrain if you'd like. Uh it's open source and free, but you don't have to. There are a lot of really good ones. the the library will compound from the first week and your whole org will be wired to run at about 400x.

Gary: And if you want the green field, the thing that I'd build if I were 25 and sitting where you're sitting, every company on this earth is about to need a brain, the memory layer that means that you never have to reask what you knew. Personal AI that actually knows you. We're building GBrain in the open and MIT open source. Um, I'm not trying to make money from this because I think the layer should be open. The way Linux is open, but the layer itself, company brains, personal context, the librarian that picks the three books, that's all wide open territory. I hope somebody builds the defining company here. And I'd like to fund you at YC if you do.

Gary: Now, let me be honest in a way that maybe undercuts my own pitch. You don't need my tools to start. Open Claw is the Ferrari. I will always recommend it, but Codeex is a really good Honda. It will do 90% of this. Uh, it will not blow your face off, but it will get you there. Use whatever. The concepts are the point, not my repos. You know, think about where the computation is. Use skill files as employees, the library, and the librarian. Never do one-off work. Those travel with you to any stack.

Gary: So, let me land this. A lot of people in the world right now are terrified about what happens to all the jobs and I understand the fear. But I want to say it plainly that is a failure of imagination and the people in this room are the answer to it. What I just described you're going to take to your startup. You will multiply yourself and every person in your company will multiply themselves and you will go build the companies that become the beacon for how all of this works in society. Abundance is not a policy paper. It is shipped software.

Gary: I have a friend who has a rare form of epilepsy. He built a repo of 80,000 markdown files, a company brain for one small boy, and he pushed himself to the absolute edge of humanity, what humanity knows about his son's exact condition. No lab, no grant, no permission. A father, a laptop, and a library. That's not a side story. That is the exact architecture I've been describing for the last 20 minutes. A library, the librarian, the right three books open at the right moment. pointed at the thing this man loves the most in the world.

Gary: You can do that now. Every problem where you thought, I wish I had that person, but I can't get them. You can. Every codebase you thought was too buggy to fix, you can fix all of it. Every archive too big to read. Every data set too gnarly to clean. Every ocean you were told not to boil. We can boil the ocean now. And every single one of you can fly, not metaphorically, mechanically. And you need to to survive, to thrive, to win.

Gary: Theo asked what we should build now. And here's the whole whole answer. Build that AI native company and build it. Build the thing underneath it. the brain, the memory, the compounding library that makes every company after yours easier to build. Go boil the ocean. Go write that test. Go ship that skill. Some of the companies you're about to watch in the battlefield are already doing this. Go build the one that does it best. Thank you. Please join me in welcoming the chief executive officer at Hyper Agent, Howy Lou.

Howy Lou: All right. Hello. Um, who's excited about the end of this event? This this has been an amazing few days. Amazing talks, amazing uh Gary talk right there. And by the way, I've never been introduced with so much base. I am very very pumped in this moment. Um so I consider myself a product thinker at heart. Meaning you know as much as I like to spend time on the technical side of building and you know I I love to think about architecture and algorithms and so on. You know really I'm a product form factor thinker at heart. And you know

Speaker: What I want to end today off with is a brief and kind of high-level overview of how I see the world going with agents as it relates to product form factors. So I'm calling this hiring employable agents, and it's really a reflection on, as these models have gotten better and better, and we've heard from some really, really great speakers about, you know, the latest frontier models, whether GLM 5.2 or others, and how to coax the most performance out of them. And really, this is not about the technical deep dive, but how do we think about these from a product building standpoint.

Speaker: So who am I? I'm the founder and CEO of Airtable, and also know Hyper agent, which is a product within Airtable. And as many of you already know, Airtable is basically an almost laughably horizontal platform. When we in fact set out to build the company almost 12 years ago, we got basically this negative feedback from virtually every investor we talked to early on, which was, you know, "You can't possibly go out with a horizontal platform approach and try to build something that is everything for everyone, like any type of app that people want to build, you know, you can't solve all of those use cases." And you know, I think we ended up doing a pretty good job of solving for many of them at least, and solving for a large builder community, and we're basically applying a very, very similar philosophy to agents now. And I'm going to break down how we're doing it.

Speaker: So, when I zoom out and think about the progression of models and therefore the product form factors they enable, I really see it as this spectrum, right? So, we went from basically completions, which is, you know, you give a model, one of these LLMs, a string of tokens and it just continues emitting more tokens until it gets to a stop token and more or less it would just kind of complete your thought, right? Or complete the pattern. And you know, until InstructGPT happened and enabled a chat form factor, it was arguably powerful but more narrowly applicable. Once we got to a higher level of model intelligence, and then also the innovation of instruct and kind of tuning these models to actually have more of a conversational back and forth behavior with users, then we enabled the chatbot, right? And of course, we all know ChatGPT was an awesome breakout product. We're now all very, very intimately familiar with that form factor of AI.

Speaker: The next level I call an agent, right? And you know, I think agent has been very overloaded. We've probably talked about that a ton over the past few days, but my definition of agent really is the same as the Anthropic definition, which is distinct from workflows. And an agent is really a model that recurses upon itself, right? And has the open-ended set of tools that it can call, of reasoning steps it can make, of decisions it can make. So, it's not prescripted to a linear workflow, but it can kind of make an open-ended set of choices. Still, you know, I like to think of agents as being a little bit distinct from the next level of agents, which I'm using the term claw here because OpenClaw obviously has popularized this concept.

Speaker: But, you know, there's a market shift that I see from agents that were still mostly reactive to user input. So, you would have an agent go and perform a task. It might spend 20 minutes maybe up to an hour to perform the task and then it completes. Claws, as I call them, are really about agents that go and perform work for much longer, right? They can even have a mechanism like the heartbeat mechanism within OpenClaw, and now in other products like Hyper agent, that enable them to wake up on their own and almost exhibit this always on behavior. I mean, underneath the hood what's happening is each wake up basically results in many different turns. So the agent loops upon itself. It's performing tool calls. Each tool call response then invokes the LLM again until it reaches a stopping point, but then that stopping point gets reinvoked or kind of restarted with a heartbeat mechanism where you can wake up the agent again and again, right? And so this idea of this always on agent that's actually constantly moving towards its goal and kind of figure out creatively how to unblock itself where maybe prior agents would get stuck. I think that's a really, really important kind of next form factor leap that we're seeing.

Speaker: And then finally, as you think about models that are now capable of, you know, really, really intelligent orchestration of complex tasks, right? So I've personally been using the workflows capability in Cloud Code quite a bit, where it can fan out a lot of work to a massively parallel set of agents. But this concept of this agent that can go and orchestrate with other agents, right? An agent-to-agent interaction to perform even more advanced tasks, right, longer running jobs is I think the next leap. Yet all of this is enabled because the models are getting smarter and smarter to the point where they can achieve coherence even on longer and more open-ended scoped problems, right?

Speaker: If you recall back to the days of BabyAGI and AutoGPT, those were really kind of fun cool science experiments, but ultimately, as much as they were really interesting to watch and see them work for hours and hours on a problem, they would all inevitably kind of end up drifting off to something not super useful, right? It would go off into a rabbit hole and kind of get stuck. And I think now the models are actually smart enough to enable not only longer running jobs but actually enable us to start applying these kind of organizational principles to break down problems into differently scoped tasks that different agents can hand off to each other. All of which is about managing each agent's task or context window and ability to coordinate work with each other.

Speaker: So all this is kind of abstract. I want to make it very, very tangible and show you how we've applied some of that design thinking of building a horizontal product and distilling a lot of the complexity of agents like we did for building a database or a database to powered app with Airtable for agents in Hyper agent. So this is a very realistic use case based on some of our actual customer usage. One of the interesting things we found from launching Hyper agent is the prototypical users for Hyper agent are not who you would think they are, right. I mean, we definitely get our fair share of like the super AI forward, kind of pure software companies, like AI companies themselves, but we also get really interesting other companies that are, you know, sometimes like traditionally offline businesses.

Speaker: And in this case, we're depicting a very realistic scenario, which is a customer that is in a landscaping business and actually uses agents to run virtually end to end every part of their business, which is finding clients to propose landscaping projects to, like physically going and redoing their garden or their backyard, and end to end managing everything from the prospecting to building a quote and a pitch to managing the work itself. And I think this is a huge, huge, interesting insight that I've just kind of stumbled on from working on Hyper agent, which is ingenuity of agent builders is not contained to just AI companies or software companies.

Speaker: And I was really inspired by Gary's talk just a second ago. I think the motive behind that applies to every industry, every sector, and there are really, really creative people, many of whom are in this room, who can go out and disrupt your own industry whatever it be. You know, it doesn't have to be a pure software business. But I think all it takes is the ability to imagine what work could look like when you do go and become fully not just an AI native company, but I like to use the term, a fully fleet of agents native company where you're actually going above and beyond and not just thinking about small kind of narrow steps to automate with AI, but actually how to employ a whole fleet of agents that have governed behavior and have the right

and the right tools to be able to do really useful and very end-to-end autonomous work. So, let's walk through what this looks like. Um, this is a realistic example of, you know, in this business, this uh this landscaper wants to go and pitch clients, right? So, you know, they get all these inbound uh submissions of clients who are interested. Somebody submits this inquiry saying, "Hey, here's my current backyard. You know, it's a little messy. Here's my info. Like, can you give me a quote and show me a proposal of what you could do?" This basically gets routed into a um in into one agent that is the triager agent, right? And so this agent is basically taking in this intake and saying, "Hey, like what is the um the scope of the project, you know, like is this something we want to take on and is this a quality lead? Is this a legit person?" And so it does some research, but ultimately it hands off then the work to another agent, a surveyor agent, to actually go and then look at the um the physical landscape of this person's backyard and come up with a proposal of what a rellandscape version of that could look like.

So here um the surveyor agent is going and looking at the original imagery uh or even video that the um submitter gave them. Um hyper agent uh in our case works with a fully um capable sandbox VM. So every agent in fact every thread with an agent is able to fully write code. It can manipulate files. You can actually use things like ffmpeg to like, you know, clip out individual screenshots from a video file and then analyze those um and actually go through and like find all the relevant information to create a fairly high quality um and hightouch uh proposal that would have been unfathomable for a small business like this before, right? If you think about like a really high-end interior designer or maybe like a very high-end landscaper, maybe they could do a very bespoke proposal like this for like a $5 million plus uh client, you know, if they're working for a big hotel or something. But it would have been unheard of to do something this hightouch and actually mock up real imagery and a real client pitch deck for, you know, just a small backyard landscaping proposal. But this person has done it because they've they figured out how to use agents in an extremely capable and autonomous way to do the job that otherwise would have been undoable by humans.

And so what happens next is you know the surveyor agent it it went and created this proposal and pitch and now the intervention has to go back to the human like there is a step where humans still need to unblock the agents right Sarah Guo put out a great post uh a few weeks ago about you know just describing like where the value is going to acrue in this AI landscape as things evolve and models get smarter and I firmly agree with her take which was you know the ultimate blocker to still be able to deploy useful agents there's still a human element to that you know As much as models are getting smarter, the ability for agents to get unblocked by humans that still own policies and decisions and really importantly just kind of gate, you know, really critical um high impact actions like in this case actually making a pitch to a customer that has a binding quote um or perhaps like agreeing to do the work that becomes something that the human then intervenes at just like a great manager who's getting upwards uh reports or you know upwards work updates from their team to then unblock.

So here we have Sage who has come back to uh the real human uh owner of this business and saying hey here's a lead I already surveyed the the project here's a beautiful pitch deck and a proposal and have come up with a realistic quote based on all the factors I've already surveyed in this uh particular project to give to uh to you to now approve sending to the client and see if they accept it right and you can see like even the u the proposal itself is this really beautiful kind of interactive uh web page That again would have been unfathomable for a business like this to be able to send to every single client even for small tasks uh or jobs like a $10,000 landscaping proposal, right? Um you know, better yet, as you interact with these agents, you know, as as Gary talked about with Gbrain and and just kind of like creating this very um sticky context layer that learns with every interaction. I completely agree. I think that is a you know kind of a universal principle and kind of a form factor for useful agents going forward is agents should not be static. They should not just be you know either static LLM calls or static even agents that just have a prompt and you know some static skills. They really need to learn with every interaction right and you can't wait for the next you know kind of fine-tuning run of your own models where you have you know a bunch of collected data or rubrics to to retrain the model. It should learn in real time, right? It should accumulate the memories and skill updates and just take user feedback and actually learn from real world performance to constantly adjust and get better just like humans do.

And so in this case you can see you know within Slack or you know in our case within uh either Slack or email or other um invocation methods you can basically go back and forth with the agent just like a real human and say hey like you know remember to take this into account next time you kind of you know here's something differently uh formatted or a different way to think about the proposal or you missed kind of um this additional cost lever in this particular project. So, you know, remember that for next time. And I think it's critical that in this next generation of agents, agents need to remember these things, right? Um they should feel fluid and and um evolving just like humans are in order for them to fully realize the potential that the latest frontier model capabilities enable them to to actually go and do in terms of autonomy. Um, and then finally, I think what ends up happening at the end of all of this is with all of these agents that are each individually capable and even interacting with each other, I think the job of the human becomes increasingly about unblocking agents to be able to do work effectively.

And so, you know, I I compare this to the development workflows that uh have evolved from, you know, originally before any um, you know, kind of AI enabled coding, obviously, we just sat in front of a computer. We as humans are pretty singlethreaded, right? You'd stare at one code file, you'd think about a problem, you'd kind of bang out some code, maybe take a break, come back to it to then we had um augmented coding with the first generation of GitHub copilot and similar tools where I mean I think of that as really just a completions form factor on coding. You type some code, it can autocomplete a few extra lines to then we had things like cursor composer 1.0 0 like the original uh experience not the the composer model um but the chat experience where you could talk to this agent it would do more complex changes and that was even more leverage right but we're now moving to this point where you know the best frontier agentic developers I know and many of you are sitting in this room are are really just kind of overseeing a fleet of agents right like in fact going to sleep without setting off your agents to perform useful work overnight feels like you're you're taking this huge loss like your your team is not working because you're not unblocking

And so the UX of how you oversee these agents, whether it's in development workflows or general purpose work like this landscaping example needs to shift as well. And so the way we've thought about that is you need to see this kind of orchestration control plane where you can see at a glance what all of your agents are working on, what they're blocked on, who's handing off work to each other, and your job becomes almost like, you know, zooming out to see the entire Sim City landscape and orchestrating across everything going on at a macro level between your different agents rather than going and having to play micro and zoom into each individual project or each individual task. Um and so I think this

Speaker A: is a really really exciting moment as we have shifted from you know not just completions and kind of the slight augmentation of existing human uh work but really a radical now leap into humans as orchestrators of agents that also can orchestrate themselves. And finding the right way for us to interact with those agents as they become more and more capable is going to become not just the opportunity to boil the ocean, but in my opinion, table stakes for survival for every company in every industry. This is going to be the way to get ahead and thrive in your industry. And if you don't, you know, as Jensen put it, you know, it's not going to be AI taking your job. It's going to be, you know, somebody using AI who takes your job or takes your business. So really really excited both for the opportunity and for everyone in this room to be part of this um transformation and this really kind of bold leap into the next form factor of AI. Um all that being said, we'd love to see you try Hyper Agent. So hyperaggent.com/ie and you get $1,000 of inference credit. Um you know that can be spent on all of our different model offerings including uh you know the latest uh Opus 4.8, Fable 5, you can use it on GLM 5.2 uh on GBD 5.5 etc. and we'd love to see what you do with uh with agents on Hyper Agent or elsewhere. Thank you all for being here. How are you again?

Speaker B: Thank you for joining us.

Speaker C: Thank you. Um so for the close of our events uh a few months ago I saw that hyper agent was launching this startup competition and at AIE we've always wanted to feature some kind of battlefield for like a competition and a contest for people uh and we didn't really have any sort of mechanism by which to judge or to fund it or anything like that and hyper agent came along and they had this uh 500 founders y

Speaker B: right what was the inspiration for that

Speaker A: you know I I think um as I think about the multiple layers of disruption stacking, you know, we we're going as I said from, you know, just humans doing the same work they were augmented a little bit by AI to now humans overseeing fleets of agents doing work that wasn't before possible like this landscaper running his business in a fundamentally different way to now this entirely new economy of agent native companies being built. And so, you know, really the founding 500 is about betting on the most frontier part of that economy as possible, right? It's the most agentically leveraged companies that are not only out disrupting their space but kind of defining an entirely new playbook for how companies in this new era will be run. And I think the leverage we're already seeing from some of these uh these companies that are running in this way like Gary said like it's profound. I mean it's far more than even being like an early software leveraged or internet leveraged company. So really really you know kind of big believer that this subset of the economy is going to have explosive growth far higher than any other sector and it's going to cut across not just you know the tech industry software industry but every industry.

Speaker C: Yeah. Uh in my London keynote I always often talk about how uh work agents are eating the rest of work and being accessible to nontechnical teams. In fact my team is right there back now because AIE runs on air table uh and they're so excited to meet you and so excited to to work with hyper agents. Um so from 500 founders we you guys pre-seelelected 20 they competed today in the start of battlefields. Uh three have emerged as finalists. Let's introduce them. Um so I don't know if I'm supposed to to click this. Uh but uh first of all I'm just going to go to introduce uh our dear judges. I think uh we're going to have Theo and Joshua. Um thank you from Hen. And uh we I think we'll we'll we'll sort of uh have them uh judging there. Uh but also we're going to uh go into the format and the background as well. Uh so basically what we're going to do uh we um the Hyper Asian team has kindly put up $100,000 in prizes and basically what this is is a competition for whoever is going to be the most impressive uh winners. So uh 50k to the winners um 30k to runner up 20k to second runner up. Uh we're gonna have a hype video that is uh thankfully um created by Hyperframes and Hunen. Uh we have fun pitches and Q&A with the judges and then we'll also get to the next uh uh pitch as well. So just three quick pitches back toback. Um and basically we decided that we really wanted some kind of judging in terms of like what is impressive and usage of this technology and like why it matters. Um it's not an investment pitch so much as like you know it's not like a VC pitch so much as like well why should you care why is this an interesting use of technology and and why um you know why are you like an interesting founder that people should know um and so I'm very excited to announce like the winning teams. Uh so I think first of all we're going to introduce um come on Spiro come on the stage. All right. Thank you. Um I think you guys are gonna take over and present. Yeah.

Spiro Anakos: All right.

Speaker C: Do you have your clickers or you got you got your you're gonna kick it off? Okay.

Spiro Anakos: Hey everyone, I'm Spirro Anakos, one of the founders of Kamad. This is Dom

Nick Nakos: and I'm Nick Nakos.

Spiro Anakos: We are the founders of Kamad. Kamad has identified physical commodity trade. From the rice you eat to the gold chain you're wearing to the chip in your smartphone to the gas you're putting in your car to every single copper pipe running through our data centers. Physical commodities power the world. But the global trade system is broken. Trillions move annually. Yet hundreds of billions are lost annually due to fraud, fragmentation, and the sheer complexity of coordinating global trade. Kamad has taken that complexity and transformed it into an agentic execution layer. Our proprietary CFC state machine governs specialized agents that verify, execute, and settle trades entirely. And then, you know, people ask, why is this so complicated? And if you think about it, it's like, well, it's every country in the world, every resource, all of their different currencies, and all of their different banking laws. So, of course, it's inevitably going to be a disaster, right? But with AI, it's finally solvable.

Nick Nakos: So, we're very excited to have you here. We also have our our first customer in the crowd out of the UAE U trading company based in Dubai, and he moves several hundreds of millions of volume in trade every single year, and uh he's proved the product. So, we're excited to be here today and thank you for the opportunity.

Dom: 17 trillion dollars in physical commodities move every year. The world runs on them, yet it still moves on paper, email, and a handshake. No one knows who is real or if the documents are the result. Billions lost. Kamad plugs into the stack you already run. Gmail, Slack, Stripe, Salesforce, WhatsApp, and more. A deal begins. Agents screen both sides. KYC identity sanctions in seconds. Forensics catch document fraud instantly. A private deal room spins up. Shipping and logistics partners link in with live pricing. Then an agent swarm structures the deal in parallel. Compliance, legal, currency, banking, shipping, settlement, weeks of work in minutes. The swarm obeys one protocol, the CFC. Our patent filed state machine. Agents advance only by clearing every deny condition. It cannot move until the evidence is real. Settlement eligible. It never touches the funds. It emits the signal. Your bank moves the money. 30 days closed in minutes. Commodities.

Speaker C: Can't see the video on the confidence monitor.

Spiro Anakos: What happens now?

Speaker C: I don't know.

Spiro Anakos: Is it on me?

Speaker C: Yes. Um I I I realized that we didn't get you a mic. So, uh, are you're still miked up at the end of Oh, you're all miked up. Okay, great. Um, so I think we have a little bit of a Q&A session.

Spiro Anakos: Yeah.

Speaker C: So, are we supposed to ask questions or what?

Spiro Anakos: Exactly.

Speaker C: My team agreed to this, not me.

Spiro Anakos: Wait, why are you guys playing this?

Speaker C: Joshua, you want to go?

Joshua: Uh, sure. I'm very interested in the um the second project, common.io. And you know certainly we need a share space for agent to collaborate together. Just curious have you thought about having that format to be HTML versus...

Speaker A: like markdown file? How you've been thinking about it?

Speaker B: What is this?

Speaker C: Sorry.

Speaker D: All right. Sure.

Speaker E: Yeah. Don't

Speaker F: Yeah. I was just saying that you know um um the second project common.io is about

Speaker G: That's the next one. Yeah.

Speaker F: Oh, the next one.

Speaker G: We're asking them about the first one.

Speaker F: Uh so

Judge: I'll ask a question. This is one of those types of products that's really hard to build because it requires so much domain specific knowledge as well as like the building out of all of these pieces to collect the information needed for the product. A lot of the concern I have with modern startups is the thing that they're building is eventually going to be a feature of a sufficiently intelligent model. Do you see a future where the models get capable enough and the tools they're given by anthropic and OpenAI could compete with you guys if they were to get smart enough and build in the right directions? ticket.

Commodities Founder 1: So in commodities trade, it's very specific as to what data is being pulled. And what we've done with our orchestration layer is set the gates for each agent. So they're pulling from real databases and have that context themselves. Whereas you're talking about gold, oil, grain, whatever it may be, there's certain procedures that only industry experts really know and have the documents and the information to pull that in. Which is why we've partnered with Connor and a few other partners to bring in all that data. uh and I have domain experience as well to know what's real, what's not and to train the agents based upon that. So to your question, could that happen? They can learn more for sure on the open internet, but this is a private industry with a lot of sensitive information and they have to learn upon and train upon certain data in order to progress and become experts in that domain.

Judge: So it sounds like relatively bespoke in the sense that like every company's a bit different. You need to get everything for them. How many customers would it take for this to become a billion dollar company or to hit like I don't know 100 mil ar like how many customers you think it would take to get there?

Commodities Founder 1: It's about 100 uh 100 customers total

Judge: huge.

Commodities Founder 1: It's also important to mention as well like the tickets in this industry are massive on on deal sizes. It's not like a Stripe for example processing microtransactions. One transaction might be you know $200 million or north of that. and one partner such as a refinery, many of them transact 9 to12 billion dollars a month. So getting in at that level, charging per deal on that changes the whole landscape of what we're able to earn and what we're able to be valued at.

Commodities Founder 2: I just want to add one more piece as well. Our solution is endtoend. So you know the solution serves the entire supply chain, but the wedge ultimately into the industry is the traders and the banks. Banks pulled out of physical commodities trading years ago after the fraud became so rampant. So this unlocks a tremendous amount of liquidity as well by the audit trails and the immutability that we give uh into the commodity and the trade itself. So there's a lot to love about this business.

Judge: I think like in many ways it's like checks all the boxes on like the classic like here is the ideal vertical agent business to build. Like if you look at the the YC kind of like RFS for vertical agents like this is like the perfect application in the right the right kind of sector. like there's lots of money up for grabs and there's a lot of like domain specific stuff that you have to do that creates I think a good defensible moat. Um I think my perspective is more like you know what do you do to keep up with the changing competitive landscape and just you know as models progress how do you need to evolve your product to stay competitive right so it's less about is this a relevant and great business today and more about what changes over the next two three years or even like next year that impacts you in a way that you know it doesn't necessarily mean like enthropic opening eye eats your lunch but what do you need to do to stay you know kind of at the frontier and be a better product.

Commodities Founder 1: Yeah, I think like the only thing that's actually defensible is being the most creative in your category regardless of which product you're creating, right? Like truly nothing's defensible other than being the most creative, right? Anybody these days has a product, but we actually do like the anthropic eat your lunch model. Like we like the ship extinction model where we ship and then 10 companies go extinct the next day. So um yeah, it's aggressive shipping and product superiority, but then ultimately we're going to win on distribution.

Judge: What are you most excited about on your upcoming road map if you can share?

Commodities Founder 1: becoming the bank.

Judge: Yeah, it's exciting. Thank you so much. Awesome. Awesome. Give it up and come on. Uh we're going to invite the next team on now. Uh if you want to come back here.

Host: Thank you. Appreciate it. Yeah, that's awesome. Thanks. Um and up next we have Max Windbomb and the team from Common I.O. Come on, Max. Oh, you're going to play the video. All right. Come on. All right. Play the video. That was That's the right All right, welcome in. Come on, guys. Oh, thank you.

Host: Um, if you could, uh, so we have a little bit of a technical snafu where the judges couldn't see the video. Okay. Uh we have to describe a little bit just to just to prompt things off before you if you sort of before you get into the whole pitch. Yeah. Okay. Pretend like we're a non multimodal model. We just we only accept text.

Common I.O. Team: So uh yeah text.io is a uh it's a multiplayer markdown editor. It's uh focused on making the best documents um for the modern work which is people in agents. But what you would have seen on the video is a familiar document editor interface, but people and agents are popping in. They're notifying each other. Everyone can call anyone in. Um, and that's that's basically the gist.

Judge: Awesome. All right.

Common.io Co-founder: So, um, it's Friday afternoon, ready to go home, packing up your laptop, and all of a sudden you're hearing dings around the office, servers down, big outage. So, you put down your bag, you head into the conference room, and you're pulling out your laptop. There's one question on your mind. What is going on, Claude? And you're entering that in. You're waiting a few minutes. Give me some ideas. It's culminating or whatever. And then it gives you some ideas. And the first thing you do is you tell everyone else in the room the ideas so they can put it into their clots. And then someone else has a different idea. And then they let you know, and then you put it into your clot. And you real, how did we get here? How is this the job? There has to be a better way. And there is. It's agents and the pe who are doing the work and people who own the problem sharing context in real time. But how? Think of the productivity software that you all use, the collaboration software. Can you just like drop your coding agents into them and have them working with each other? No, you can't because they were built for people, not for agents. And that's why we're making comment. It is the multiplayer markdown editor. It is we're laser focused on making it the best document platform for modern work for people and for agents. It's open. It's pluggable. That's key for enterprises. It's key for it's key for developers. And it's wrapped in a user experience that is super familiar. I started my career in productivity bringing collaboration features into Microsoft Word. Max and I worked together um at Texio, the first AI native writing software. Six years before chat, we were deploying companywide rollouts of AI writing software to the Fortune 500. We've been doing this a long time.

Common.io Co-founder: The market for comment is expanding with agents. As more and more people use agents, more people are going to need the ability for agents to share context in a product that is built for agents but designed for people working together even in the highest stake circumstances.

Judge: Awesome. I love the shape of this business because it's like the inverse of what we just heard from the um the commod guys, right? Like they went super vertically deep.

Speaker A: They're trying to extract a lot of value out of a very deep um you know domain and you guys are going super broad and arguably maybe a little thinner but by intention right and I don't say that as a as a detriment at all like I think you know one way to think about the landscape here is like the agent economy is going to be so large there's just so much money flowing through agents like literally the token spend right and if you can be a product that like even a one to you know 2% uh amount of those tokens touches like you can be a very valuable even if like the per uh per call or per usage um you know kind of uh metric uh you know kind of margin extraction or revenue is is relatively low you can be a very cheap very fast very um broad product and still gain a lot of scale right so I think it's a really really interesting shape of product um what have you found is like the most surprising learnings from like a product design or product affordances standpoint to make this product better for agents than like the human designed equivalents right like why not just have agents collaborate on a Google doc right or even try to use like GitHub as the uh you know kind of repo not just for code but for markdown.

Speaker B: All writing software today is horrible for agents to use. Software like Google Docs has a really terrible format. It has an MCP. You can send a document there. You can have your agent read a document from there but you can't have them collaborate in real time. And that's critical for all kinds of real work that needs to happen. And as more agents join the workforce, they need to be in there with your documents. And as we've talked to our users, people who love working in documents, who rely on documents for their day-to-day, need to bring their agents because they're starting to use agents, too. It's not just developers anymore.

Speaker A: Okay. I think

Speaker C: I'll mention it.

Speaker B: Oh, final closing words. I was just going to say the other thing I'll mention is that I've been super surprised about um talking to people at this conference when we've talked to them is how many people have said I've rolled my own like visual markdown editor that other people can look at so people can look and then you're like ah but I gave up I just rendered an HTML document instead like it's a hard problem and you need dedicated like focus on making it great and that's kind of what we're doing.

Max: Awesome. All right. Nice job guys. Thank you so much. Uh next for and last team to compete. We're going to play the video by Foundry.

Foundry Video: Content creators don't have time to build a business, so they have to hire an agency. It's expensive, slow, and risky. Built by Foundry replaces the agency and lets creators launch their own products faster and cheaper. Our AI agents build the business end to end. The creator launches it. We keep growing the revenue. Type in any creator's handle and our agents become their biggest fans, watching their videos, reading their comments, and finding a painkiller their audience would actually pay for. We've done this for creators of every size and over half a million people use products our agents built. The cost of building software is approaching zero. What's left is trust and taste and no one has more of it than creators. We're building the agentic product team that is going to turn every creator into a founder.

Max: Woo! Foundry team, come on up.

Weston Belgettes: Thank you. Thanks, Max.

Weston Belgettes: Hello, Aie. I am Weston Belgettes. I'm the founder and CEO of Built by Foundry. We're going to turn every content creator into a founder of their very own company. And I have great news today. I think we found the formula for a winning business. You take your unique expertise and turn it into a product. And you solve cold start with your built-in distribution. Theo, you you did this with T3 Chat.

Theo: Yeah. It sucked. It doesn't work. It works great for our our our users, but uh Gary did it with with G uh GStack and Gbrain. And uh what what we find is that if we help uh creators to take a step back and solve a real pain point that their users have, so we're not just making another Patreon for that, paywalling this content. We're helping them solve a real pain point behind that. And that's what our agents are experts in is finding the pain point within the users's problems. And our companies have uh built over uh many different niches and uh different sizes of sha and shapes of creators. And really what we're building is we want to give the power back to the content creator. They're stuck in these I worked at Tik Tok for two years and and I met with hundreds of creators who are stuck just doing brand deals or affiliate links or selling $20 PDFs in their link in bio and they are one step away from building their own business. They just need someone to help them. And that's what our agents do. At Built by Foundry, we deploy teams of agents to build recurring revenue businesses for content creators so that they can focus on posting and posting and posting and we can keep growing their revenue. I have a lot of thoughts on this one.

Weston Belgettes: Okay.

Theo: So, the problem I find as a creator, even with the brands I work with, is that there is a saturation point where it stops being useful to talk about a given product with my audience. Like with me, for Versell, for example, there was quickly a point where everybody in my audience either was a Verscell user and doesn't want me to show it to them anymore, or they won't be a Versel user, and they don't want me to talk about it anymore. It is very easy to hit the saturation point and the best solution is to have a much broader set of things to show your audience. One of the best things I ever did for myself, the single moment that has had the most positive impact on myself, my career, my businesses, and my income has been going from having four sponsors to 80 because now there's a wider variety of things to bring my audience. And even if 80% of them aren't relevant to them, the 20% that is is now conversion I'm getting that I wouldn't have otherwise. the variety of what I bring to my audience is where I find the most value by far and every creator I've worked with ultimately has come to a similar conclusion. That said, you're touching on really important pieces here. Creators do have more taste than most founders. Creators do have better distribution than most companies. Creators know their audience better than most businesses. And you've built a machine that identifies that, but it it doesn't provide the solution to that other problem, which is you have to bring variety to your audience. Where I could see this working really well is as a tool that businesses use when they identify a creator is overlapping with the archetype of customer they're looking for to help that business build better things for Theo. And I've told this to so many businesses that want to work with me. Watch my streams, watch my videos, find the problems I have, and then show me how your product solves them. That's a way better pitch than going to try and start your own company as a YouTuber. As the YouTuber who has started the second most companies, as far as I'm aware, it doesn't work. It just burns you out and it burns your audience out too. The best thing you can do is work with companies to teach them what your audience wants and for them to come to you with money to show it to those audience or those audience members.

Weston Belgettes: I understand and I will say your niche as a tech creator is one that is very challenging because there's a lot of options. Um and what we have found is that for all of our creators, I'll just few examples. Um, one one creator's name is Brandon and he does homesteading content and he always loses track of uh the best settings on his freeze dryer, which sounds ridiculous to us, but to his 2 million followers, that's a real pain point. And now they pay him a recurring revenue monthly or monthly or yearly subscription to solve that specific pain point. So, I think unfortunately, Theo, you've picked a bad niche. I'm sorry, but um if you want to pivot to homesteading or gardening, maybe uh we could we could work together. But I I'd say primarily whatever the users niche is apart from tech and maybe com comedians, we've been able to

Startup Pitcher: take a step back. Our agents find a painoint that they're willing to pay for and zero creators across our entire company history have ever left our our company because they always are profitable.

Judge: >> You're very convincing. Okay, excellent. Then give it up for

Host: >> Yeah. Thank you. Um, so we're going to have a little bit of a deliberation moment. Uh, the judge is going to go backstage for a couple minutes. In the meantime, sorry. Okay. In the meantime, we've, uh, prepared this little video recapping the World's Fair. So, take your time, judges. This is our biggest AI engineer. event ever. We're constantly adding new techniques for you to draw. Heat. Heat. Hey, hey, hey. Thank you so much. All right. Um I think we're inviting everyone back on stage. Give it up for the start of Battlefield. Um, congrats guys. Yes. Okay. So, uh, we have some prizes that we're going to hand out. I think, um, the folks back there are doing some checks. Um, in second runner up place, we have Kimode. Uh, Theo, I think you're presenting. Uh, Theo. Yes. Okay, we'll take some pictures as well for the team.

Host: >> All right. Um, next, and I guess this also reveals the order. Uh, we have Foundry as runner up.

Judge: >> Great pitch, Max. Great pitch.

Host: >> Congrats.

Host: >> Awesome. All right, no secrets now for the grand prize. comment I io welcome. Great video as well.

Judge: >> Congratulations. Well deserved.

Host: >> Do you want to also do the group photo? I think we're doing

Someone: >> all of us.

Host: >> Um, yes. Okay. All right.

Someone: >> Let it be just them.

Someone: >> They're the founders. We're just here to com everyone.

Someone: >> Fine. We'll sneak in.

Someone: >> Get them. Come on in.

Someone: >> Yes.

Host: >> All right. Thank you so much guys on building with hyper agent and

Winning Team Member: >> yeah of course um any last words that you wanted to comment on. I think you guys uh briefed a little bit. Yeah.

Another Winning Team Member: >> Yeah. I just want to thank uh sorry about

Another Winning Team Member: >> thank everyone here at AIE? We got amazing feedback um through the whole um time and uh hyper agent um for inviting us to this and hyper fans for helping us with the video

Another Winning Team Member: >> and uh overall like so much.

Another Winning Team Member: >> Check us out. It's free.

Another Winning Team Member: >> comment.io.

Host: >> All right, that's it. Thank you. Ladies and gentlemen, please welcome back to the stage our MC developer relations engineer at Replet, Ralph Shabri.

Ralph Shabri: Hot. Okay, so let's give it up to our startup Battlefields finalist. All right, they collectively won 100K worth of prize to boost out their the their project. So, this is great. Um, all right. Let's give it up also to all our judges, Josh, Howie, and Theo. All right, my next slide. Clicker not working. Next slide, please. Clicker not working. Okay. All right. And with that, AI Engineer Welfare 2026 is a wrap. And we've come a long way. This year's edition is our most ambitious one. 4 days, 7,000 attendees, keynote sessions, countless of workshops, conversations, demos, 40 tracks. And we hope that you find this content useful and helpful, and that you're going to go back home inspired. But what we love to see more than anything else is the community coming together, creating new connections, new friendships, exchanging ideas, and building together. And we're extremely grateful to be part of this. So, let's give it up for you guys. All right. We're also very extremely grateful to our sponsors. So, please let's give it up to our presenting sponsor, Microsoft. and also to our lab and platinum sponsors. Please, let's keep it going. Our gold sponsors and our silver and bronze sponsors. Thank you. We're extremely grateful because without these sponsors, this event wouldn't be possible. And also I would like to to thank an invisible group of people who are working backstage who you you never see but without whom this event wouldn't be possible. Uh so thank you all crew members, volunteers, let's give it up for them. All right. So uh previous slide please. Okay. Almost there. There you go. Nope. All right. Okay. So, this is the end of Worlds Fair 2026, but we're not done yet. We hope to see many of you in New York in October. And also, we're going to be back together here next year for another edition of AI Engineers World Fair. But before we celebrate America's birthday, we have one more thing to show you. Heat. Heat. Heat up Heat. Heat.