软件重构：AI工程的核心支柱
我是 Dat Ngo，在 Arize AI 担任人工智能架构师（AI Architect）。我们在全球范围内与许多大型企业合作，解决如何让 AI 真正落地运作的问题——具体涵盖可观测性（Observability）、评估（Evaluation）和实验（Experimentation）。AI 领域的发展感觉就像是软件工程的重构（software reimagined），它并非魔法，而是一套具有相同底层逻辑但表现形式不同的工程模式。基于我们独特的行业视角，我观察到了目前团队在构建智能体（Agents）和执行框架（Harnesses）时面临的最大痛点，并总结了应对这些非确定性（non-deterministic）系统的三个核心环节。
Original English
Okay, I didn't know if there's like a cut scene, but okay, so really nice to meet you all. My name is Dat. I work at Arize AI, so I'll talk a little bit about what that is. A little bit about me and kind of what I want to share today is um you know, I I work very deeply in in the space. I'm an AI architect. I work with a lot of the largest enterprises across the world to talk about, you know, we work on things like observability, evaluation, experimentation, but really it's just how do you make AI work, right? So, I do spend a lot of tokens um in this space. So, this is last OpenAI DevDay. I think I made it to probably somewhere between 100 billion and 1 trillion tokens last year. So, I do I do like to vibe it out. But I do know the space really, really well. We work with some of the world's largest companies and enterprises. So, we get to see their transformation into this space. And really what I wanted to share today was like what do I see in in the industry? So, I think we have a very unique vantage point being the company that we are. So, we get to see what every team is building, how they're building it, what are the biggest pains that they face, and really how they're trying to fix those those things. So, if I had to really distill down, you know, what we do in kind of a nutshell, it's really these three things. Maybe by show of hands, like who's who's built agents, who's building agents, who's productionized agents, who's building harnesses, and who has no idea what a harness is. Okay, we're all we're all pretty okay, all pretty cracked, so that's good. You know, I think it's really funny is that like the AI space, it really just feels like software reimagined. It's really the same set of patterns, just maybe a different flavor coming out. And it's really it feels like magic, but it's not magic, right? It's all just engineering. So, really what we're going to cover today is just three things.可观测性：解码非确定性系统
构建复杂 AI 系统的第一步是建立可观测性，以解答“我构建的系统内部到底发生了什么？”这一基础问题。当你试图修复一个自认为的 Bug 时，由于系统的非确定性，你可能会在不知不觉中引发两三个新的回归错误（regressions）。为此，我们采用 OpenTelemetry（Otel）作为核心架构。这是一种强大的工程模式，通过添加一行代码（自动注入）即可生成追踪记录（traces）和跨度（spans），从而形成智能体行为的完整审计日志。除了基础的代码级追踪，我们还需要关注会话（Sessions）——即系统与用户之间的多轮交互状态（如 Anthropic 近期发布的管理智能体论文所述），以及轨迹（Trajectories）——用于分析当模型选择特定分支或调用组件顺序颠倒时引发的系统延迟与错误。
Original English
The first one is observability, which answers the question of like what's happening in the thing that I've built and what does that look like in in whether it's a harness or an agent. Then we'll go into evals. Evals are just simply how do I derive signal from my systems in some form or fashion, right? And then as we talk about how we make improvements in this new non-deterministic world, you'll come to find out that when you make you know, what you perceive as a fix and you fix the thing that you thought you fixed, you might have actually produced like two or three regressions that you didn't really know about. And so, like in our world, we talk about observability, right? Everything that we do, something that we're super proud about is Otel is a really strong pattern for those in the engineering space, but everything we we do is through open telemetry. So, you know, it doesn't really matter, you know, what particular type of you know, harness agent model setup that you have. The good news is that, you know, being Otel first, we're really prepped for for a lot of these use cases. So, whether it's an auto instrumenter, if you know what that is, but basically you add one line of code, that one line of code will, you know, basically see what's happening around in that particular framework or SDK, create open telemetry traces and spans, and produce, you know, these views kind of here. So, if you've ever seen a trace or span, it's basically the audit record of what did my agent do? Cuz now we know that code doesn't audit agents or harnesses, it's actually the telemetry that does that. So, traces is a big fundamental part of observability. Now, there are many other different parts of observability that you should be thinking about outside of traces and spans. You can think about sessions, too. So, I don't know if you any of you read the Anthropic paper, managed agents that came out 2 days ago. Pretty awesome read. But sessions is another one about like state. So, what are the back and forth conversations, what are the back and forth states that are happening between you can think runs. For a lot of people in the enterprise, they may want to end up running I realize this isn't the easiest to see, so let me change over to light mode. A lot of folks yeah, sorry about that. Yeah. A lot of folks would like to, you know, understand like, "Hey, what is the back and forth conversations?" So, that will be something like a session. So, it's like, "Hey, what are those back and forth conversations that are that are being had?" Great. Now, people like to eval those. So, in the enterprise, you'll see a lot of folks being like, "I don't really care the deep level, you know, the agent did this tool call, that tool call." They may not care about that as much as like, "Hey, was the end user satisfied? Were all their questions kind of answered?" Now, one unique thing, I'll change this over to light mode, too. One unique thing about what we do here at Arize, this is Arize AX, is that sometimes when you think about your agent as not a a non-deterministic call, right? You want to be able to see, "Hey, what did my agent do?" For example, there's different paths that your agent could take, right? So, you know, different branches. But what if you wanted to like look over all instantiations of your agent and get a more distributional view of what's happening? These are kind of like views into like the distribution of your agent. So, what are all the possible paths and branches, also loops? It allows you to answer questions like, "What percentage of my traffic goes down one branch versus another, right? Was there a particular component in that particular, you know, branch that we took that caused whatever, a significant amount of latency?" When we start to talk about agents or different paths, you may think about trajectory evals. So, trajectory could be like, "Hey, I went down this one path and everything was really good, but for some reason when I go down this path, the the evals or the signal that I'm collecting is dropping. Why? What's the root cause? Oh, the root cause issue was that these two components are actually out of order. I did B before A and actually B has a dependency on A. So, actually like it turns out like the way my LLM decided to call these things was was mismatched, right? We need to put some context in there to say, "Hey, actually before you do this, you need to do to do that." So, there's many different, you know, views into observability. Of course, things like analytics aren't dead, either. So, you know, what a lot of the you know, folks in in the enterprise end up looking at is just they just want to build views on what their what their agents look like in real time. And so, being able to customize and build those views out, super, super helpful. And that's that's observability in a nutshell. Is like, can I see all the different layers? We have many, many different types of of layers here.五维评估：构建核心质量信号
在建立可观测性之后，第二步是获取系统的评估信号（Evaluation Signal）。我将获取信号的方式归纳为五种核心途径：第一种是广为人知的大语言模型作为裁判（LLM-as-a-judge），其复杂度远超表面概念；第二种是人类反馈，无论是终端用户还是产品经理，其领域直觉都无可替代；第三种是高质量的黄金数据集（Golden Data Sets），我们可借此对 LLM 裁判进行微调和校准；第四种是兼具低成本与高稳定性的确定性逻辑评估（如校验 JSON 负载的格式与必填字段）；第五种则是最终导向的业务指标（提升收入、降低成本或节省时间）。在企业级应用中，评估体系往往需要技术人员（AI 工程师负责构建框架）与业务专家（领域专家定义提示词与评估标准）的深度协同，并根据具体需求选择单跨度评估（Span Evals）、跨组件的多跨度评估（Multi-span Evals）、轨迹评估（Trajectory Evals）或会话级状态机评估。
Original English
Um You know, the next thing is like, okay, so I have observability, that's kind of step one. It's the same thing that happened in software, right? Now, you have to determine signal, right? And so, signal comes in actually many various forms as well. The way I like to break it down is is kind of these five flavors of signal. Um I think everyone here in the room has heard of LLM as a judge. Um and so, you know, it may seem like a simple concept, but in actuality, it can actually get quite complex and we'll kind of go through all of that. Now, you can't forget about your humans. When you think about humans, whether it's the end users using your product, it's extremely valuable signal. So, whether you're product manager or, you know, someone technical or non-technical, you do care about this signal. We've all heard of golden data sets. They're extremely valuable because it's if the third column here represents quality, you know, you trust the person who who labeled this data because they know the domain. Then you'll, you know, you'll run techniques like, "Hey, I'm going to run my LLM as a judge on some, you know, golden data set so that you can tune your LLM as a judge." You basically say, "Hey, my can I get my LLM to approximate this this thing or this person or this data set that I trust?" And then of course, we're all thinking about costs, as well. So, when we think about costs, you don't always have to use an LLM call or even humans. Determinism is super nice, so think about logic or deterministic based evals. If I go from paragraph to, you know, JSON payload, does this JSON is it a valid JSON? Does it have the schema? Does it have, you know, these fields that are non-null? And then of course, we're all building these things for some, you know, for one of three purposes, I think. So, the business metrics you care about are either some form of like, "How do I make more money? How do I save money? Or how do I save time, right?" And so, what you'll notice as you start to build really good AI products, um you'll start to have two types of personas that kind of end up coming together. So, obviously, you have your technical users, right? These are your your AI engineers, your your developers of the world. These people are extremely good at building and automating things, right? They're good at frameworking. But then you have folks who who are maybe less technical, but they understand what the AI experience should be, right? These are like the subject matter experts, the product managers of the world. These folks end up, you know, you want to relegate kind of the work of like, "Hey, this is how the prompt engineering should go. Here's the [clears throat] evals that I care about." Because you want people who can code coding and you want people who know the domain to to work in that domain. And so, in our world, what that looks like is something like this. You know, we allow folks to be able to run evals at just in a non-technical way. Of course, if you are technical, you can attach your evals and and run them kind of programmatically if you want. But in our world, we want to be able to say, "Hey, you know, I want to be able to allow a user to be able to select, you know, their their model, be able to run some out-of-the-box template or customize some eval here." And when we talk about complexity on the eval side, right? Imagine for a second that you have built some some application, some agent, some harness. That harness is got components in it. They may be called deterministically or non-deterministically. Whatever. So evals can be run on you can think a single kind of component. You know, we call that a span kind of eval. Let me kind of come here. So the scope would be one single input and and output. I'll pull up like a more complex view of this, but oops, let me close this. But you can think of the simple span input and output as um hey, I want to look at the input and output of one part of an LLM call. So that's most people understand that and that's really really simple. Now um we also have like multi-span evals. So we can think of that as like hey, in order to run the eval that I want, it actually requires data across many different components in in the system. So if I want to say hey, how well are agents passing data back and forth to each other? Well, it turns out I need the data from every single agent and how they pass data. So that's a multi-span eval and it allows you to run more complexity. If you want to look over all of the spans in in total, that's something like a trajectory eval. Did we call things in the right trajectory to to finish the business process? Um and then there's that session level eval, right? It's it's like zooming out and saying hey, what does the state machine like if I want to evaluate that state machine of um hey, let me turn this light mode. Hey, in this conversation, was the user ever frustrated? Did we answer all of their questions? So think of that as I want to evaluate the state machine. So as you're thinking about evals, it's not generally, you know, it's it's also like hey, what flavor of evals do we want to run, but at what scope and depth? So you can get very granular and then you can also zoom out. And just because you can eval something doesn't mean you always should. It's not this exhaustive thing. You want to see like hey, what are the minimal set of evals I can get away with to understand signal of like is my application working as intended because there's a cost associated with this stuff, right?自动化实验：走向自治的工程飞轮
掌握可观测性与评估数据后，下一步是通过实验改进（Experimentation）系统。实验通常意味着系统级的变更：修改提示词、切换基础模型、重构调度编排逻辑或调整运行配置。虽然目前很多团队通过人工界面分析错误日志并上传数据集进行调整，但我们深刻意识到软件开发的终局在于“极致压缩与自动化”。在未来，开发者不需要手动挑选评估指标，自动驾驶系统（如我们内嵌的 AI 助手 Alex）可以直接读取轨迹上下文，实时定位高延迟与错误节点，并动态生成测试评估方案。我们期望打造一个完全自动化的工程飞轮（Flywheel），将可观测性、评估与实验优化全链路打通。此外，为了满足不同开发者的诉求，我们同时推出了支持本地单容器部署的开源版 Arize Phoenix，以及专为 Uber、Booking、Reddit 等顶尖科技巨头打造的企业级平台 Arize AX，以支撑更大规模的 AI 商业化落地。