Nick: Hi everyone. My name is Nick and I'm here with Ashe to give a talk about evals. We are from Lyft and we've been building Lyft customer support AI agents for a year or two now, and we've given a lot of thought to how to build evals that actually matter and scale our multi-AI agent system. Just a little bit of quick introductions. My name is Nick. I'm a data science manager. I've been at Lyft for six years, a long time lifter. Really excited to talk to you a little bit more about evals.

Ash: Hi everyone. I'm Ash. I'm in Nick's team and we've been working together on customer support agents for Lyft, improving the robustness, improving the evals, things like that. I've been at Lyft for almost four years now and very excited to be here and talk about building evals that actually matter.

Nick: Super excited to be here and super honored to be on the online track for the AI Engineer World Fair. Yeah, and let's dive in. For the agenda of today, our talk will primarily focus on evals. We will start by sharing how we think about the end-to-end pipeline for our evaluations for building a customer support AI agent system. We will go into a deep dive into each component more deeply as we go. We will start by talking about offline evaluations, online evaluations, eval harness, as well as what we are planning to build going forward.

All right, let's dive in. I want to quickly explain the high-level system of how we think about the evaluation system for AI agents. So here you can see we have the development phase and the production phase. During development, if you are building agents, you should be very familiar with managing context, building RAG pipelines to give your agents business context, defining your tools, building your agentic graph, as well as writing a system prompt. Once all of that agent engineering process is done, you have an AI agent. The way we think about this is before we launch this AI agent to production, we want to go through a rigorous offline evaluation process to make sure that this agent actually has sufficient performance before we launch this to live users. Coming from a data science and machine learning background, we've been building machine learning models for a while, and I think the way that we think about agent development is very similar to building machine learning models as well. If we are running offline evaluations for our machine learning models before they go to production, I think we should do the same for AI agents as well or any other agentic platforms and applications.

But how I think offline evaluation is different than traditional machine learning models is that typically, we're building specifically for the customer support AI use case, and we are building an agent that is multi-turn. So for offline evaluation, there will be a component of simulated conversations. You typically want to have a synthetic dataset that is representative of your production traffic, use an LLM that plays out the complete multi-turn simulated conversation, as well as have a grader such as an LLM as a judge to evaluate how good that interaction was. Then we have a launch gate, right? We want to make sure that we have certain criteria on our offline eval and we are meeting that criteria before we decide to launch this AI agent to production.

The real imperative here really is that we don't want to use our live users as test data for our AI agents. In any case, that is not good practice. So we really want to emphasize the importance of having an evaluation process. Once the agent hits production, we also have an online evaluation pipeline as well. We have our tracing tools to trace all the executions and context that the AI agent uses to respond to a real user in the production environment. We have our online grader as well that grades how well our AI agent is doing in production, as well as having a human-in-the-loop pipeline to do error analysis, identify failure modes, and feed those insights back to the development teams to continuously improve our AI agents.

Nick: I want to quickly go over three of the most common reasons why we think evaluations typically fail for different teams. The first reason is that the grader we create and the scores we generate need to be meaningfully gating something. This is what we really emphasized on the previous slide—we need to have a launch gate. If your LLM as a judge is just floating out there generating a score, but no one is really using that score as a meaningful gate for your development and production environment, then that LLM as a judge is not useful. We have also seen a lot of mishaps when people are creating their LLM judge. There are a lot of different opinions out there on how to create a good LLM judge, and unfortunately, early on in our journey, the LLM judge we created was very noisy and too generic. It would output a score, but people didn't really believe in it, or they didn't think the insights were actionable. Finally, when something regresses in production, we need to have a clear mechanism to catch that regression and identify clear owners to take actions based on the insights of our graders and regression gates.

Nick: Very cool. I want to transition to talking about our offline evaluation system. As I touched on earlier, this is the most critical piece going from the development cycle to production. We want to have a robust offline evaluation system to gain confidence in the AI agents we ship to production. We took a lot of inspiration from the paper called TaoBench, developed by the wonderful people at Sierra AI. This is specifically for customer support AI agents, but we think it is applicable for any user-facing agentic applications. Here, you can see this is an offline simulation where you have the AI agent and a user LLM interacting with each other to produce multi-turn traces. You have the agent domain policy, which is essentially the instructions/policies on how to handle different customer support issues.

Taking inspiration from TaoBench, this is the high-level approach we have for creating our offline simulator. As we mentioned earlier, we have our LangGraph agents, and we have defined instructions for our user LLM. For the simulation, we define the user intent, what this intent is supposed to represent, and the user data points or the world state of our user. For example, the driver coming to us might be a luxury driver who has been driving for us for a couple of years. Finally, to define user behavior and personas as we see with real Lyft users, we created these different personas. One example is a loyal longtime Lyft customer who is frustrated with the Lyft earning system.

So in our offline simulator, we have this LangGraph agent interacting with our user LLM and generating these multi-turn trajectories. We also built our offline grader/evaluator, where the LLM judge is a major component. We will dive a lot deeper into how to build a great LLM judge. Apart from the LLM judge, we also have deterministic evaluators, which usually look like code assertions in traditional unit tests. For example, some deterministic criteria look like this: if in a specific interaction, the AI agent is supposed to grant a concession, we write a rule checking whether the agent indeed granted a concession. We compare the agent's tool calls with the expected outcome to measure the accuracy of the agent's instruction-following behavior.

Nick: One of the big gotchas we face with offline evaluation is that creating synthetic data is a key challenge in making sure our offline dataset is representative of production data. Ideally, what you don't want to be doing is simply prompting an LLM to generate 50 different test queries for your offline dataset. Here are a couple of suggestions on how you can approach this more realistically and assemble a dataset that closely resembles your production data. We take samples from our production data. Lyft users have been reaching out to support for a long time, so we take a sample of real production corpus and supplement our offline dataset with that. Second, we mutate different criteria from our offline datasets to cover different golden paths and edge cases.

One problem with our offline evaluator was that, as mentioned, if you are simply sampling 50 test queries from an LLM, another big gotcha was using a frontier lab LLM model to roleplay Lyft users. For the most part, frontier models are trained to be helpful assistants rather than a Lyft user who might not sound very nice. For everyone who has tried customer support before, you don't typically reach out with a very nice tone. In our first pass of offline evaluation, we noticed that our LLM user sounded almost too nice. As you can see on the right, these verbatims show the LLM user very patiently explaining the issues. Our first attempt at offline evaluation gave us a 90% pass rate. This was too good to be true. In reality, real user verbatims are different. Most users are impatient and frustrated; they don't want to explain their issues like an LLM. This is what we typically see in production, which makes the AI agent much more difficult to evaluate.

So what can we do to make sure our user LLM simulates real-life Lyft users closely? We fine-tuned an LLM model with real Lyft user verbatims. Instead of speaking in a verbose and polite way, our LLM user will produce verbatims that resemble real users closely. While we did see our evaluation score go down after fine-tuning this model (making the evaluation harder), this is exactly what you want when building a user simulator. If you have an eval that is too easy, it doesn't give you real production insights. This also gives you room to tweak your AI agents to deal with "difficult" users. We also defined Lyft user personas to ground the LLM user to adopt specific personas—such as bypassers who just want to escalate immediately, refund seekers, and AI skeptics. We took inspiration from Microsoft's paper on UserLLM, which adopts a very similar approach where they fine-tune a user LLM and see evaluation scores go down, which is expected and beneficial.

Nick: All right, I'll hand it off to Ash to talk more about the LLM judge.

Ash: All right. Hi everyone. So, I'm going to take it from here. I'll talk about the second problem which we usually face when we are doing eval with LLM as a judge. Here you can see this is how pretty much everyone is using LLM as a judge to evaluate their agents. The focus can be slightly different based on the use case. Someone can focus more on safety. Someone can focus more on cost and latency. Someone can focus more on quality. But people are going to measure these sort of metrics more or less. We want to detect leaks, we want to detect safety issues, and things like that. So some parts are deterministic which can be evaluated by code and some are not which are evaluated by the LLM.

Now the problem with this approach is that these metrics are too generic and not actionable. For example, we also started our eval journey using pre-built metrics from DeepEval, which were measuring tool usage, appropriateness, response helpfulness, conversation naturalness, completeness, and things like that. We did see those metrics, but the problem was these metrics were not actionable. They were not giving us any actionable insights. If let's say response helpfulness is 0.5, then what do we do with it? Scores like toxicity score, bias, fairness, conciseness—all these are relevant. But if the metrics are just scores, we don't know what to do with them. We can use these pre-built eval metrics as a baseline, but we shouldn't use them as our core eval metrics because we want eval metrics to be actionable and tied to the business outcome or the product we are focusing on.

Ash: Okay. So what should LLM as a judge be? We collaborate very closely with domain experts and utilize their insights. Evals should be framed around a task success or failure, and a binary outcome is very easy to calibrate and train LLM judges that can consistently score your agentic trajectory. When we partner with domain experts and data scientists to build metrics that are actionable and aligned with business goals, we can see much more meaningful results and actionable insights. Not only is this more consistent, but when an agent fails an interaction, we can systematically analyze the error pattern and actually know what we can do to fix this. Here is an example of an actionable metric for our use case which is called the education rubric. Here we define the metric how it should be and we define success and fail criteria for the LLM judge. For example, the AI agent tries too many times to educate the user when it could have escalated the issue, or if it is escalating too soon without giving any chance to educate the user. Things like that come under the failure category. So we mark this as fail. But if it is the expected behavior, we mark it as pass.

Ash: Okay. With this, how do we validate if our LLM judge is working as expected? First, we need to treat it as a classifier. How we train our classification models in traditional machine learning, we can also treat our evaluation judges as those traditional ML classifiers with binary outputs. Once we have binary outputs for every metric or task based on our business goals and functional requirements, we can hand-label around 100 examples with pass/fail labels and then split the data into train, dev, and validation sets like how we used to do with machine learning models. Then we score precision and recall for our judge based on human-labeled ground truths, which will give us an actual report on how good our judge is performing.

And how do we split our data to calculate precision and recall? We split the data similar to how we used to do in training machine learning models, but the difference is the percentage of splits. Here we are not actually training model weights. We are just using the data to inform the judge's prompt. So the percentages are a little bit different. For example, in training data, we pick a few-shot examples from this set for the judge's prompt, and then we iterate the prompt against the dev set and improve our robustness or our prompt. Finally, we validate against the test set to see that we didn't overfit on the dev examples. This is the practical way of splitting the data and calculating precision and recall scores for your judge to actually know that the judge is working as expected.

Ash: All right. This is another thing which most of us ignore when we are doing evaluations, which is criteria drift and validating the validators. The key idea is that we actually discover what our evaluation criteria is by looking at the data and grading our outputs. Our sense of quality will also evolve with new data we see and more examples we grade. So the evaluations should not be decoupled from model observations. In fact, they should be co-developed with the model when we are testing the evaluator and calculating the precision and recall scores. There is always a gap when we talk about LLM as a judge. We cannot define the criteria beforehand and then evaluate agents against them. Our criteria should also be evolving as and when we see more examples, and then we should refine our metrics for our judges and then evaluate judges on top of those metrics.

Ash: So the next step, one of the things which can make the numbers we are reporting more meaningful is to add some statistical rigor to it. We report alignment rates as bare point estimates. If we add confidence intervals, do some calibration and proper sampling, the same numbers can become more meaningful. We should definitely reserve the expensive rigor for the moments when a number actually gates something, like a shipping decision, or when we are reporting numbers to company leaders. But depending on the use case, we should definitely have confidence intervals for the numbers we report because every score needs an interval. Here is just a small example to give you an insight on what this actually means. Let's say we have two evaluators: one scores 84% and the other scores 88%, and the number of traces we have used is 50. To show that this is a very small gain, we need much more than 50 examples to actually show that this gain is real. With just 50 examples and only a four percentage point gain, we don't actually know if this gain is real or not. Bigger gains and paired designs need far less samples, and we can reserve the statistical rigor for things which matter the most.

Ash: Okay. So this is a non-exhaustive list of observed eval anti-patterns. I'm not going to read all of them, but these definitely contain some low-hanging fruits. We need to put in time and effort if we need meaningful evals. We cannot rely on LLMs for everything yet. What I can say is ignoring the data is one of the most important things which we shouldn't do, and which we sometimes don't focus on due to lack of time or resourcing, but it acts as the foundation for meaningful evaluations. If you don't look at the data, you won't be able to create meaningful criteria or labels. And if you don't have labels, you won't be able to evaluate your judges. And if you are not evaluating your judges, you don't know if your agentic pipeline is working as expected. So this acts as the base. It is one of the most important things we should not ignore.

Okay. We said that we want to make metrics more actionable and standardize the pipeline, but how should we actually do it? This gives you a template to do an error analysis loop, and it is important to know that this loop is something which runs continuously. It is not a one-off audit. We deep dive into raw traces. Once we have logged our traces for our agentic flows, multi-agent systems, or whatever we have in our use case, we pinpoint failure modes. Basically, we try to identify what exactly is failing and we only keep the metrics that change a decision. We remove all the noise. We only prioritize the metrics that are tied to business use cases, functional requirements, and actually change a decision. Then we form a fresh premise to re-evaluate and then we repeat it. We can have a regular cadence of doing this pipeline. It can be weekly, it can be bi-weekly, but this is something which needs to run continuously and it is not a one-off audit.

Ash: Tracing, as we said, is one of the most important things and everything kind of depends on it. For diving deep into raw traces, we definitely need to log them first. We can use tools like Langsmith, Langfuse, etc. to log and view the traces. Here each trace captures the full graph execution: which nodes ran, what LLM saw, which tools were called, what was the token usage, what was the latency for every call, and things like that. We can also enrich traces with metadata if we want, which gives you more insights than the actual data.

And we also have annotation queues. Annotation queues are nothing but an interface which is very helpful for domain experts to label or give feedback to evaluators in an easy to understand UI. They don't have to look at the raw traces JSONs and stuff like that to figure out what to focus on. They can use this annotation queue and they can give feedback or label examples easily. We can then add these traces to datasets for offline evaluation or we can use this for calculating our precision and recall for our judges. So this forms the basis to validate the evaluation with ground truth labels.

Ash: Okay. I now I'll hand it over to Nick to close the evaluation loop.

Nick: Thank you Ash. I think the goal of having evals is to be able to feed our evaluation insights back into improving the model performance or the agent's performance. Here, I'll introduce a couple of ways that we think about continual learning for our AI agent and closing the evaluation loop. Here we have model learning, context learning, and harness learning.

Model learning is really about post-training, updating the underlying model weights, and training a custom LLM model. Context learning and harness learning are more about improving everything else other than the model. Context is improving what information the agent actually sees. This can be documents, the stored memories of the user, tool outputs, and so and so forth. Harness means updating the model's system prompt, tool schemas, control flow routing, retries, and so on.

From the error analysis that Ash shared earlier, it has really helped us understand how we can improve our agent's prompts, update our knowledge base, and tune the context management strategy of our AI agent. Identifying failure modes really helps us feed those insights into actual improvements for the AI agents.

Nick: I want to quickly talk about what's next for us. In our journey of building customer support AI agents at Lyft, we have some level of ability to run our offline simulator, but it's not repeatable. These are currently stored as scattered scripts across different notebooks and analysis repos. One thing we are looking into investing in is a systematic eval harness. Having a harness system can help run our offline evaluation in a systematic and standardized manner, and allow different people to contribute to our evaluation suite with predefined primitives and a config-based workflow.

Another thing we've been thinking a lot about is post-training. As mentioned earlier, identifying model failure modes has helped us tune the agent's context and harness. But over the years, we have gathered a lot of real user signals on our agent's performance as well. We are starting to think about how we fine-tune a model that does different tasks for our customer support AI agent, as well as framing a reward modeling problem to enable reinforcement learning.

I want to quickly share a little bit about the work we're doing with the eval harness and how we think about building evaluatable agentic applications. Evals are the scaffolding we need to run evaluations efficiently in a standardized format across all the different agents and sub-agents we have. As you can see from our offline simulator slides, our eval harness is config-driven, typically stored as YAML files that are easily editable by different contributors—not just by engineers. Analysts and data scientists can contribute to this evaluation suite as well.

With thousands, if not tens of thousands, of examples in our evaluation suite, we need parallelism and throughput to run our offline evaluations in a reasonable amount of time. To enable different users to contribute, we also define primitives around our eval harness, such as tasks, datasets, personas, LLM adapters, and evaluators.

Ultimately, the benefit of having an eval harness is that you can define the config once and run these evals indefinitely many times across different touchpoints or gates of your agent development process. This can be locally when you're developing the agent—at any point when you tune a prompt, you can run the evaluation suite and get immediate feedback on how your agent is doing compared to previous versions. We can run these at pre-commit hooks to make sure performance doesn't degrade before pushing a change to our agent service. Another area we are looking at is CI/CD, and how we can use our eval harness to build our regression test suite and acceptance test suite.

So that wraps up our presentation today. We've gone through a lot of different topics for evaluations. Truly, this is an end-to-end journey for building an evaluation pipeline that works for customer support AI agents or any user-facing agentic applications. Ash and I are very interested to hear about what you all have been working on and share any learnings you have. Feel free to contact us if you have any questions or just want to brainstorm about how to improve evaluations. All right, thank you and I hope you enjoy the AI Engineer World Fair.

Ash: Thank you.

Nick: Thanks all.