跨越80%信任悬崖：从基准幻觉到以客户为中心的Agent评测飞轮

can everybody hear me okay, perfect. so thanks for joining everyone. my name is felipe, and i am a member of technical staff at the amazon agi lab. as part of my role, i help us to build the next generation. i manage the programs to build the next generation of products our lab's building, and today i am here to talk to you about designing evals that earn customer trust. before talking about evals, i just need to give you some context about what our lab was doing the last year.

so everything started in march last year, where we launched our nova act research preview. for context, nova act is a tool that we built as part of our lab, and it's a tool that helps you to build browser agents. so basically, everything that you do in a web browser, you can use nova act to automate those tasks. as part of the research preview, our main goal was we wanted to ship a product to get as much feedback as we could from our customers. in march, we launched that. then in july, we made one additional improvement to our product, which was basically developed a whole new developer experience. this was because basically, during the first couple months, we noticed some gaps in that developer experience, and we wanted to make sure customers had the best experience building agents. then in december, we finally launched our service on aws. so today nova act is a top-tier service on aws that anyone can use to build production-level web agents. and during this period, i was basically responsible for the customer enablement piece of that. what's customer enablement just for context: it's basically i was working day to day with all of our customers to understand what was their problem and how we were helping them to solve that. so that was my main role during last year.

and then let's go to the problem that we will be discussing today. so i like to call this the benchmark illusion. so you are probably working on and building things based on benchmarks, right? it could be public benchmarks that everybody's using and everybody's trying to optimize the products on that, and also benchmarks that you are creating with your own synthetic data that you are creating and you are evaluating your product. and the main goal here is you want to optimize, right? you want to be the best as you can during those benchmarks. but then when you ship that to production, your customers start using that, what happens? it's this, right. they start seeing problems, and it's basically everything that you were not expecting them to do. they try to do that, and they break your product.

and this is happening because basically most of us, we are working on evals that are static. so what are static evals? basically, let's talk a little bit about what's the standard development process of a product, right? first thing that you do, you receive the requirements from your product team and you start building, right? everybody does that, i hope. then when you have a product in a good shape, the next thing that you will do is running your evals against public benchmarks, synthetic data that you generated, and you optimize for that. when you start feeling comfortable, you're like, okay, now i will ship that, and you ship that to the customers. but if you have static evals, what's the only thing that you can do after that? it's basically hope, right? you just hope your evals were reflecting exactly what your customers will do with your product, and if not, it breaks, certainly, right. and this is the gap. this is exactly what we will be discussing today and what we were doing in our lab to solve this problem. and just a spoiler: this is not about better benchmarks. it's about closing the customer loop. so what we want to do here is get the feedback from your customer and close the loop.

so with that in mind, what we did in our lab was basically propose an eval flywheel. so if you are not aware about the terminology, a flywheel is basically a loop that you keep repeating all the time. and the first step of this flywheel is define success. and very important here: define success here is not what you think is success, it's what your customer thinks is success. okay.

so next step, as soon as you understand what your customer really needs and the problem you are trying to solve, you go to capture signals. and we as engineers, what's the first thing that comes to your mind? instrumentation, right? let's get metrics, let's get data as much as you can about what your customer is doing. but one very important piece of that that we are missing is you should talk to your customer. like talk to your customer: go schedule meetings, visit them, and understand exactly what they are trying to do. your data might be a little bit misleading; you might be missing exactly what's the problem that your customer is trying to solve.

third step, you go to diagnose gaps. that's basically you get all these signals from customers—could be instrumentation, but also your inputs, your insights from the meetings that you had with them—and you just start categorizing those. and when i say categorizing, there are basically, at a very high level, three things that you could find from those discussions. first, it's basically inputs about your model: where your model is failing there, and this feeds into evals for your model. the second point is inputs to your engineering team: basically, what is the tooling or harness that you should fix? this is more traditional bug fixing. and the third one is also on the product side: what are you missing in terms of product? do your customers understand exactly what our product is trying to solve? is your product properly positioned? so that's very important.

so as soon as you have the gaps categorized, you just feed that into decisions. so feed that into decisions is basically you prioritize that, and you ask your science, your engineering, your product team to address those. that's basic, right? and then you just continue repeating this loop over and over.

so with that in mind, in our lab, we repeated this flywheel several times since march last year. and what i also noticed is basically, as part of my journey as a product, each of my customers have their own journey using my product. what that means: my customer evolves as they use my product, right? so there are basically four things that you can learn while your customer uses your product.

the first one is use case discovery. that's the first thing that you will do the first time that you meet your customer: basically understand what's their problem and make sure what you are building solves that. then after that, you will have a couple early adopters. from these early adopters, the main signal that you want to get is basically what's possible. so that's the time where customers will try crazy things. you will find things that you cannot help them at all, right? but you will also find things that your customers are using your product to solve, and many times you will be surprised because it's something that you would never imagine. third, this is when you start scaling: you get your early adopters, but now you are scaling, you are getting more customers, and you are able to identify some patterns, right? what are the patterns? the pattern is basically your hero use case, right? that's the problem that everybody is using your product to solve. and the last one, this is more future looking, but these are your product gaps. so when you were working with early adopters and also after you start scaling, you will also be finding some gaps in your product. your product team can prioritize those, and these are the bets for your future, right? what you should build next.

and as i said, i worked with several customers during this journey, and there are a couple learnings that i want to share. some might be obvious, but others are very insightful. so the first one is the trust cliff that i like to call. so what's the trust cliff? imagine you build a product, you are ready to ship, and you measure the reliability of this product. and you found eighty percent reliability. that seems fine, right? like, okay, i can give that to my customer and they can start using it. but what your customer thinks when you say you are at eighty percent of reliability is: i cannot trust this product. i cannot trust it. and why? because when you show eighty percent reliability, what they will think is: oh, okay, so if it's eighty percent, i still have to monitor this agent; if they fail, i will need to do some manual work. so in fact, it's actually more work for me, not less work. i need to build this agent plus monitor that and also do some manual work after. but when you get to a point around ninety percent reliability, the message changes to your customer, and what they think is: okay, ninety percent, i can actually trust this to handle my tasks, and i completely handover a workflow to my agent. so reliability is a zero to one, right? you trust or you don't; there is nothing in the middle there.

second learning, it's the eval transparency. so every time that you start working with a customer, you basically need to be transparent about three things: first thing is—that's easy, everybody likes to do that—what works. so you basically show your customer: i am really good at doing this, that's the level of reliability, you can trust my product to just use my agent, and it will give you the reliability that you need. second point is: these are the things that i am doing to improve my product. okay, so there are some things that i know are a gap, but i'm investing time right now to make this better. and the third thing, that's the tough one, nobody wants to do that: it's what is out of scope. so you really need to be transparent: if something doesn't work for your product, just go ahead and tell your customer. don't let them try; otherwise, there are so many tools in the market right now that if you show something to them that doesn't work, they will just forget about your product and they'll move to another product. so be transparent. and the main learning here is: transparency on product limitations increases trust more than higher benchmarks. that's a very interesting finding, right? be transparent about what your product is capable of doing, and more importantly, what it's not capable of. that's how you earn customer trust.

third learning is if you work very close with customers, you might find a lot of use cases that you would never have expected. so some examples here: amazon kuiper, it's the satellite internet company from amazon. working with them, we actually learned some inputs about creating features to do caching of trajectories. so instead of doing inference all the time that they use our product, we cache trajectories; if it fails, we just fall back to the model. so we optimize the cost for that. second learning is with hertz, the car rental company. so they were also using our product for qa automation, and we learned that as part of their company, they had some qa engineers that were very technical and were capable of building scripts with python and other coding languages; however, some people from qa, they are not that technical. so that was the input that we needed to actually build a whole non-technical experience for our product that would enable more customers to use it. last one, this is a startup called sola. sola is basically a rpa company—if you are not familiar with the term, rpa is robotic process automation—and they offer tools to automate processes for other companies. so imagine the variety of use cases that they might have: it's a lot of different use cases. and what we learned from them is we could give some additional flexibility in our product for them to customize, especially the actuation stack of our browser. the learning here: you will learn a lot about what your customer is trying to do here, and what should be the next features you should build for your product.

and then running this flywheel several times, we came up with four principles for this flywheel: first one is derive evals and scenarios from production, not imagination. so again, creating synthetic data might be very important at the beginning of the process, but as soon as you start getting customer signals, make sure that feeds your evals. second, categorize into actionable areas: again, product issues, engineering issues, research issues—make sure you prioritize that well to your team. third one, share capabilities honestly: again, transparency builds trust; make sure you share with your customer, especially what your product is not good at. fourth one is run the flywheel as fast as you can: basically the faster you run that flywheel, the faster your product will improve, right? that's almost obvious, but keep that in mind: run your flywheel as fast as you can.

and then with those four principles, what you will see is your evals should get smarter every week. but very important: if you are not seeing your evals getting smarter every week, that means your evals are getting stale.

and just to finalize the talk here, there are two ideas that i want you to take home, just two: first idea is, eval must reflect what the customer cares about, not what you think they care about. okay? all about getting customer input here and making sure your evals reflect what their problem is. second is evals are not built once; it's a flywheel that evolves as your customer evolves. one note here: remember that customer journey—as your customer moves through that customer journey, what they are trying to do with our product will also change, like it will get more complex, right? so your evals need to be up to date and reflecting exactly what your customer is trying to do right now. and with that, what you have to do is simply build the loop, ship evals that matter, and then you earn your customer trust. and that's all that i had today. thanks very much for joining. i will stay in our booth after for more discussions, but enjoy the conference.