Joining us on stage is a member of technical staff at Amazon AGI lab onjab. bar. Good morning. It's so great to be back here at the AI Engineer Worlds Fair. Just a year ago, the hard problem was getting an agent to find a button and click it on a screen, especially screens it had never seen before. Now, agents can drive browsers and they're starting to also drive desktop apps. But what we figured out, clicking was actually the easy part. What we didn't solve is the actual work. And what do I mean with this? Let's take a very simple example. A new team member starts on Monday. And maybe your job is to set up their accounts, add them to your Slack channel, book intros with colleagues, order the laptops, etc. And nobody really owns this end to end process in the company and it might be also touching five different systems. Now, agents can most likely perform each single individual step of this workflow, but agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take. And this is mostly where it all falls apart. The agent can use every single tool you give it, but it still can't do the full work. So why do we see this gap? Think about for a minute what we actually build. We taught computers to use computers. So what do I mean with this? We started building out the basics. We taught them clicking, scrolling, typing, calling an API, filling out a form, and we got those steps, these steps really reliable, and you can string them together in a workflow. And agents these days are fairly good at like operating those workflows. So, why can't you not just hand them more of your work and then literally just walk away and trust it to be completed? So all the things I talked about like using a tool models itself, tool use, stringing agents together, this is all capabilities and we mostly figured out how to add capabilities to models. Now the next hard problem is really reliability and without reliability we cannot really build up trust in those systems. So here's a quick gut check and maybe all of you can just think about an agent doing work in an end to-end workflow. How often do you think that actually succeeds these days? Maybe 60 maybe 80% of the time. And it sounds really fine, but if you look into this, if your agent one in four times deletes a database, you will never touch that agent again, right? So when you need this reliability, you really need to be it in the nines. You need to have the trust that it actually can do the work successfully. Now there's actually one place where we made enormous progress on reliability and trust and this is coding right. Think about how fast coding evolved. I still remember the first time when it started autocompleting for you, right? You just tapped autocomplete. Amazing. Then short time later, it started to write functions. And we thought that is amazing. And now look at these days. Coding agents write the code. They open up the pull requests themselves. And we had it earlier this week. Code keeps flying by. So once in a time we were able to just every single line that it generated we felt like the urge we need to really read it and make sure it's correct right I think most in the audience here can still relate to that these days I think hardly anyone is still doing that like we cannot even do that right code is generated at such a pace at the same time coding made that jump so why is that because we were able to bring it from just being capable the coding agents to actually be reliable and then trusted. So why is that? Why was coding first solved? It's because code is verifiable. You can run it, you can test it, you can check it and you can be for sure that it worked. So reliability showed up in the first place you can actually verify the answer. But here's the catch. Most of the work we do if you look at the broader knowledge work areas is not like that. Knowledge work is messy and heck the whole real world is really messy. Did the report I created land? Is the design on brand? Did it get it what I actually meant? So there is no unit test that can answer those questions. So verification really hits the wall right where most of our work lives. It's living in the seams of all of those applications we're using on a day-by-day basis. And nobody really has corrected this part yet. How do you make an agent reliable when there's no way to verify the answer that easily? And that's a field that is still wide open.

So, how can we solve this? Well, so how do do humans handle messy work? I mean, we're successful at it, right? Each of us like every day we work across different systems. We manage out how to onboard a new colleague. We do this. Well, we're doing it by figuring things out together. You grab a colleague, you jump on a Zoom meeting, you're discussing things, you're looking at the problem to solve, you're discussing pointing at systems, and maybe two minutes later, you solved it. You're done. But none of this work is actually directly verifiable. And we do this all day. So one of the things is we're looking mostly at the same screen, right? If you're jumping on a meeting with a colleague, you see the same screen, both of you, and you can actually like figure out really quickly what needs to be done. So this is what the agent these days is missing. You don't necessarily need a bigger brain. What you need is this shared context. Because if we're looking the agent and myself at the same screen, I probably have much less explanating to do. So what kind of agent do we really need to build to achieve this? And today's agent, as I said, they can already see a screen, right? and they can click and take actions in it. That part works. But if they fire off actions, what they usually do, they move on. They don't watch what happens or recover if one step didn't succeed or something goes sideways. And we need an agent that can actually work like you do, like humans work. And one example is robotics. If you just look for a moment as how robotics do it, a robot perceives what's around it and it plans what to do and then acts. So this loop here from perceiving to planning to acting, this is actually what we also would need on a screen. And it starts here really with the first word which is perceive. The agent has to take in the screen the way you do, not scrape the code behind the page, but what's actually rendered, the layout, the state, what just changed the work, what we're doing, and then do it. And it would also have to keep up in real time. Think about how we as humans work together. You jump in, you react to build on top of what each other you say. And today agents can still don't do it. What we're doing is we're sending a prompt. We're waiting. It goes away and at one point the agent come back and we might have to take a couple of turns, right? Because what the agent come back with is not exactly what we might want to do. So we're sending another prompt say, "Hey, go back do this, do this differently." And we have this long back and forth which we got so used to from our chatbot experience and from this rhythm taking turns. But what we actually would need, think about it, is an agent that can react while you're still working. Wouldn't that be really cool, right? Like at the same time you're working, it can also come up with suggestions, can help you, and there is no waiting time. So basically an agent that perceives what you perceive and understands what you mean. We call them perception agents. So why perception agents? Why do they matter? So first they complete the loop on computer use. Today's agents again they can act, they can click, they can type, they can scroll, but what they can't do well is looking at the results and whether it actually worked out. A perception agent can read the rendered screen so it can confirm its own output instead of just firing off those actions and then hoping. Second, it doesn't need an API or backend process. And that's important because it works off the rendered interface. It sees the same pixels and the structure you see. And most of today's software people use every day don't expose APIs at all. And then third, the input also goes the other way here. Instead of writing a long paragraph to describe what you want to change, let's say you're working on a website and you want to describe all the changes you want to apply. Instead of writing this really long description, wouldn't it be great if you can just point to it and say, "Hey, here this heading needs to change. Hey, can you update this section?" This is a much more precise signal and less lossy than text. and the agent can act exactly on what you marked. So this is where we started and I'm happy to share that we just recently launched the first two pieces of our perception agent harness open source. There's two pieces. There is annotation which you can use to tell it what you want.

And then the second piece, the verification part gives the agent the capability to check its own work. So let me show you the first one. So here's a very quick demo on our annotation tool. This one is a Chrome extension, so it's super easy to use. And I'm going to play here this quick video demo. So you have the extension installed and then you can just select different elements on a screen. So this example we're just drawing around the heading there marking the section. And maybe you want to change it. Why not? Let's change it to red. You could also select the elements on this page. You see how if I hover over it it finds the right element. You click it, you select it and say something maybe double the font size. And you see also how the agent here captures on the screen exactly the feedback, the location, the style elements and it creates this complete summary which you can then use and then give your agent to implement. So there is no back and forth anymore because you captured exactly what you saw on screen and the agent can see the same thing. Now let's have a very brief look at the second one at verification. So the idea of verification is that you can describe let's stay in this case of the web development. You can describe in a design MD file what your design rules are for this. And then what happens if I play this video here, the act the agent can actually check its own work against those design specs. So it will take what you defined, the colors, the components, your layout, and it turns it into those rules if you don't have it written before yet. And it does two kinds of checks. Then it does a visual check, which is really cool. So everything is on brand, for example. it's the right layout. The other part is also checking user flows. So what it does there, it actually walks through this experience through the app for example depending on the tasks available. It might add a task, it might delete a task like a real user would. So it helps you walk through those user flows as well in an automated fashion. And then once it's done, it's writing a report which you can review and it's going to call out which tests passed and it's going to tell you anything that didn't. So ultimately, you're the one that doesn't have to click through this at midnight at the end of the day because great work. The agent already did this job for you. Now there might not always be a screen, right? So I talked a lot right now. I called it perception. I talked about the agent sees what you see on a screen. But there are times in your day where you don't have a screen. Maybe you're in the office. You're walking into a meeting with a colleague.

So I did a fun experiment yesterday at the conference here. So I grabbed my colleague Giovanni who is also here and actually on the second floor there's a great like little meeting booth. We found that by coincidence. So we went in there and we had our design meeting. And the goal here is really kind of show you how perception is so much more than just the visual part. So in this example, what we want to show you is perception can also be listening in the room to what you're discussing. And you can see here on the picture, both of us are wearing our B devices. Big shout out to B for sponsoring these. Um, so we're sitting there. We have our B devices that can do a transcript. They're listening to what we're saying. And then we have this design meeting and I had a couple of great ideas how to change this website. Um, you will see them in a in a second here. So let's have a quick look how this changed the same workflow on this website using this device. So we had the discussion the be did the transcript and you can see here on the right we're pulling this meeting transcript right in there is a whole detailed summary of the meeting. There is what we discussed and then it basically captures those insights. We have them right here and we can click apply. So what this apply button does is it sends it straight to the agent. And you can see here my crazy ideas to turn the background to yellow, turn the heading to red, and also change an emoji directly applied. And it also straight kicks off the verification right away. So it creates this report and luckily this color scheme was apparently into in the approved rules. Otherwise, this would have liked like you did some weird things here. But again, you could change those rules if you don't want to have yellow backgrounds and it will make sure um that we still adhere to those guidelines. It would flag anything that's off. So, you have the judgment call if you want to either update the design specs because you actually like yellow or you take an action and say, "No, um fix this violation." But this is really the very first step. These two pieces are the very first beginning. And we're building out the rest in the open because these patterns can only get better if more people are using them, building on top of them, breaking things. So my ask here to you is go and try them out. They're on our GitHub repos, open source. Tell us what we're missing. give us the feedback what you would like to see where this should go next because ultimately none of us get smart alone and that's the whole point. We want to build AI that makes all of us smarter together. Now, if you're interested in a little bit more on human agent interactions and how we see those patterns changing, I would highly recommend this podcast by my colleague Danielle Persik. She is a cognitive scientist and runs our AGI ACI team at the lab and discusses a lot about human computer interaction patterns with experts in the industry. You can find the podcast on on any popular podcast platform. We also have more sessions this week. Um, so check them out. We have a booth down there. We have expert talks. We also have another computer use track talk coming up with my colleague Gaurav Mishra at 1:30 in the computer use track. Highly recommend checking out his talk from RL to IRL. And then ultimately come find us. We have a huge presence down at the expo hall. We would love to continue the conversation with you all. If you're not here in person, you can also check out our code on our GitHub repo and check out our website. And with that, thank you very much.