微软专家的AI Agent落地方法论：从模型 Harness 到基于评估准则的演进

Speaker: Well, first of all, let's bear in mind that models by themselves don't do anything. I mean, models by themselves, they predict token by token, but they themselves actually do nothing. So they have to be embedded within a harness of some sort for them to do anything. And so certainly our own M365 Copilot is one harness. GitHub Copilot is another. There are plenty of consumer harnesses that many people are familiar with.

And so really it's the interaction of the harness and the model that produces the results that we see today. And these models are indeed, as you say, getting better and better. Every day there's a new model from us or from one of the other model building houses that are out there, and so they continue to improve, for sure. At the same time, there is definitely a place for model customization. One of the things that you're going to start to see now, I think more often, is models on the edge. And in fact, there is a model that I use on this laptop on which I'm talking to you quite frequently. I use Handy, actually. Handy is an open source project for transcription; it does transcription really, really well. And so, you know, I wrote an email to you in advance of this podcast. I didn't type it. I probably dictated it. And I've gotten this dictation really dialed in, but it runs locally. This is a model that's running entirely locally on my laptop. Does not go to the cloud, nothing like that.

Now, we're going to find more and more of these things as the hardware becomes more and more capable. I have a relatively new computer here, so it's pretty capable at handling these models. But as the hardware becomes more capable of handling these models, I think you're going to start to see an explosion of models that have custom purposes that, like the transcription model. It's not a general purpose model. It doesn't do whatever; it does transcription.

Host: I think the model can do it, but the other question is why do I need this one when I can just have Whisper, for example, or GPT Realtime to do this for me? Especially like, you know, clients are asking, so like, yeah, what is the demand for this?

Speaker: Well, I mean, to be perfectly honest, what the model that I'm using locally here is Whisper. However, I have made a number of customizations to it. I mean, for example, let's say that I was commonly emailing you, and Whisper wouldn't just know how to say your name, which I may or may not be pronouncing correctly, but in any case, regardless of how I pronounce it, you know, will the model spell it correctly? I mean, maybe, maybe not, right? But if I adjust it a little bit, as I have been doing, then it's much more likely to spell it correctly.

That's right. And you'll see that, I mean, we have seen that for many years with things like custom vision models. I mean, for example, in our own Document Intelligence. So in addition to the general purpose OCR, if you have a complicated table, for example, or you have a document that spans across multiple pages and it's two columns, you know, the general purpose model has a tendency not to be able to stitch that together. But our Document Intelligence model can do that. In fact, it can do that so well that we also have a document translator. And the document translator will take the layout and reproduce it, but in some other language. If it's a right-to-left language, it'll also flip the layout. It's pretty crazy, actually. I mean, when you do it, it kind of melts your mind a little bit when you actually do these translations because it's like, how did you do that? That's nuts.

Host: Yeah. And I think by describing the usage, like the application of this model, I realized something, that is, most of us interact with new technologies because we are a consumer, but Microsoft, your job covers a very large consumer offering also, right? In terms of Xbox, Bing, Windows, and stuff like that. So yes, we do have a consumer footprint as well. But...

Speaker: But yes, it's true, at least in my division, that I am mostly talking to enterprises and governments. Although, you know, the technologies and the directions tend to be the same, but the way that enterprises and governments use our technologies. Now, one of the things that I'm going to be talking about at the AI Engineers Conference in Shanghai is agents with their own identity. And this is an emerging trend. You're starting to see this everywhere.

What does it mean for an agent to have an identity? Because up until recently, and indeed even today, our own Copilot, for example, impersonates you. So when Copilot is writing a report or sending an email or any of that stuff, it's doing it as me. So it is impersonating me. So it may have a name, but beyond that, it's a pretty shell around me. So when you have an agent that's impersonating you, it is running in your security context. It inherits whatever permissions you have. So if I have the permission to delete all the files off of my computer, and if this agent is impersonating me, it too can delete all of the files off of my computer if it really goes sideways.

Host: Mhm.

Speaker: But if I make an autopilot and it has its own identity, then first of all, it can't just delete my files. It can't even see my files because it's a totally different user, just like you're a different user from me, unless I specifically give it that access, unless I affirmatively grant it that access.

And so the way that we're using this notion of agent identities is really towards the principle of least privilege, which you probably heard of, that you know this idea that you want to give people, and now agents, the least number of privileges as possible for them to be able to do their job and no more than that, so that they can't go sideways. That becomes especially important in this age of agents for the kind of governance that businesses and governments are looking for.

Host: That's interesting because I remember hearing Acquire episode, and it actually talks about how Microsoft builds its advantage on top of the directory. Like, it's the directory that once you have the directory, all the office software just inherits that very complex and behind the scenes, you know, rights and permissions and files. Yeah. So that is actually playing into the advantage of Microsoft. But then the employees would be worried. Now I'm seeing an agent with identity doing what I can do. So like, what does it mean for the workforce?

Speaker: Well, when you declare an autopilot, for example, when you create an instance of an agent, it is a service principal. So it's an account. It has its own, you know, you can audit its access and its permissions just like you would audit anyone else.

Jobs are definitely going to change. They are already changing. As I said, my own video production workflow has completely changed. I don't use a teleprompter. In the old days, if I misspoke, I had to re-record the whole take. Today, I just edit the transcript, delete the bad word, and the video splice is done. I'm not using generative video; I make all my own videos and voice them. But it takes minutes instead of hours. The key to the success of any AI is, frankly, permission.

Host: You mentioned permission from employees. Can you elaborate?

Speaker: Well, permission from people, you know. I mean, I think one of the mistakes that some folks make—and we don't do this as much now, but you know, you can't just shove technology down people's throats. The happiest AI path is when the AI is available to everybody, enabling everybody in an organization. The devices and ways we interact with AI are still evolving, and we need people to willingly adopt it.

Host: Yeah. Sorry for, you know, all these speculative questions because, you know, when we talk about AI, we couldn't help but speculate. But now, I think we can shift gears towards like what you actually observe on the field, especially like this enterprise already adopting AI. What is a good strategy? What is a bad strategy? And what are the opportunities and new kind of business opportunities created?

Speaker: Yeah. I mean, I think the best strategy is creating AI that enables people. I think one of my favorite customer stories is that of BMW. BMW uses our technology heavily. And in particular, I went and visited them in Munich earlier this year, actually, and it was cool. They gave me a tour of the factory, all this stuff.

But there is a group at BMW called BMW MDR, Mobile Data Recorder, and they have these specially instrumented BMWs all over the world. So basically, you get if you're part of the MDR program, you get a free BMW, but your trunk is full of instrumentation because it's logging everything you can think of. They got sensors on the brakes and the engine and for the ambient humidity and everything you can possibly think of. There are literally 5,000 sensors on these cars.

And the challenge was that until recently—and I mean like a year ago—they had these like wizards, basically these people that could query all of this data because they get so much data, by the way, that the way they retrieve it is you can't just send it over the wire because there's too much. You literally send a drive to them and they put it into this big machine. It looks like, if you know Ghostbusters, it's like when they put the trap in the thing and the containment unit and they get the ghost out. It looks like that. I'm not even kidding. That's what I told them when I saw it. I was like, it's like a containment unit. It's like a terabyte or more of data that it gathers from driving around. So it's quite a bit of telemetry.

And for many years, they've just had a few of these kind of wizards that could query this information. So if I was working in the braking division, I wanted to know how the brakes on my new car design were performing, I would have to go talk to one of these kind of wizards, and eventually I'd get in the queue, and eventually I'd get my query back and I'd get some information about how my brakes are performing.

And they decided one day, they're like, "We're going to modernize this. We have agents today. We have these models that are pretty capable." But again, I mean, this is one of those places where the model by itself is insufficient because again, you got 5,000 sensors, and each one of these sensors has a different value, different thresholds, a different meaning. I mean, you know, the brake sensors are totally different than the engine sensors, right? And brake temperatures will vary very differently than engine temperatures, one would hope.

So they spent like six months going all around BMW and gathering a data dictionary. And this is the work that's still required. I mean, if you want your AI to work with your structured data or your unstructured data or whatever, it seems magical, but it's not. You still have to tell it what this stuff is. So they spent all this time with other groups at BMW compiling this information about what these sensors were gathering and also how to interact with them—like what terminology, what acronyms might you be using when you're interacting with the brake sensors versus the trunk sensor versus the whatever sensor. So they spent this time doing this stuff, but the result is they have an extremely functional agent that I was able to try myself.

Now this is me walking in the door, I have no knowledge of any of these sensors, but I was able to actually—the query I made, I remember it, I said, "Show me all the hard-braking events in wet weather in the last week." And it did it! It like made a series of queries and it was like, okay, here are all the hard-braking events in the last week in wet weather. And so if I wanted to dig deeper on that and be like, okay, what was the temperature of the brakes when that happened, or what was the stopping distance, I could have gone deeper with that. And even as a layperson, I was able to get that done.

So they spent a lot of time to make this system, which now all of the BMW engineers can use. Rather than waiting in line for somebody to make them a query, anybody can go into the system and use it just like I did, in a natural way. So again, it wasn't magical. They had to put the work in. Six months they spent developing this, putting together the data, the context that it needed. But in the end, they have built an agent that materially improves the working lives of these engineers at BMW and, frankly, improves the quality of the product itself.

Host: Yeah. Yeah. Like reducing incidents and building better cars. I have such high respects for these people. I think it's right to call them wizards. When I was at Tencent, I did kind of cleaning the data for an ad system. It was so hard—three months, and just like backend, frontend, machine learning systems, and people hard-coding things so much, so hard to clean. But I can imagine their work is like 100 times harder.

Yeah. And I think that's really the hidden part of really functional AI. Behind this really functional AI is actually quite a bit of work. And for as much as we have these coding agents now that have these really, really good coding models and those kinds of things—and yes, of course, they can write SQL queries and KQL queries and GQL queries and all kinds of queries. Yes, they can do that stuff, and that still doesn't mean that you don't have to put in the work, because it really comes down to the context.

I mean, again, like when I think about BMW MDR, it really is about the context that they gave to this agent that tells it what all this data is, what it means, what it's about, and all that stuff. And there is no super ultra magical automated way to do that. They had to gather this information from their colleagues who were making these sensors, who were using these systems. It just wasn't going to be magical. And no coding agent in the world was going to be able to look at the raw data and be like, "Yeah, that's definitely brakes. That's a hard-braking event for sure." Like, no way.

I think, yeah, it is a barrier or a chasm. When there is tacit information involved, agents just cannot know that; you have to extract that from human for agents to be any useful. But that's part of the raw material. And then the other end is evaluations, right? How can you know if the agent is doing the job properly? You queried the agent, and I'm sure they must have some way to know this information is actually accurate and gives you what you ask for.

Speaker: Yeah. Well, I mean, we could talk about how that has recently changed. The way that we evaluate agents has changed, actually, quite a bit recently. We're moving towards this notion of rubric-based or outcome-based evaluation.

Host: What is that?

Speaker: Yeah. What it means is, let's think about how it used to be before we even think about how it is today. So we have these metrics, and the original metric was groundedness. You know, you think about back in the day—which is only three years ago as of this recording. But we were building these question-answering agents, and one of the primary concerns at that time was hallucination. Thinking back, remember? Groundedness was this idea that, did it hallucinate or not?

And then fast forward a little bit, we have tool-call metrics where a tool is an agent integration. It's integrating to something, right? A Microsoft 365 tool, an external tool, or whatever. And so then we have these tool-call metrics. Did it correctly understand the task you were giving it to do? Task completion: Did it complete the task you were giving it to do? Task adherence: Did it do only that thing and not some other random stuff in its quest?

So those are the kinds of metrics that we have been using for some years now. But there is a fundamental flaw with all of those metrics: they are too generic. They're generic in terms of what it is that your agent is supposed to be doing.

So let's say, for example, I was talking to one of our customers recently, and they were looking to price apartments. They want to make an agent that prices apartments for them. It wakes up on Monday morning, it browses the local listings for their competitors, it compiles a spreadsheet, and then sends it to the manager, and now we know how we're going to price our apartments for this week.

And task completion will be like, "You're good! You asked for a completed task, you got a spreadsheet, right?" But then I open the spreadsheet. I look at the spreadsheet and I'm like, "Uh, well, you know, the one and two bedrooms, those are pretty good, but those three bedrooms, man, they're way off. Something here is not right."

But task completion as a metric would not know this because it's generic. It doesn't know how to price apartments. It doesn't know that three bedrooms in this zip code should be this range. It just knows that you asked for a spreadsheet and you got a spreadsheet.

And you can't just say, "This price is wrong." You have to say, "This price is wrong, and here's why." Because that "why"—that's what goes into the rubric down the line.

Furthermore, in business, not all mistakes are created equal. If I price an apartment a little too high, maybe it sits on the market for an extra week. Not the end of the world. But if my agent writes a listing that violates Fair Housing Laws, I'm going to get in trouble with the government for that. That is a must-have constraint. Whereas, like, let's say that I'm like, "Hey, this ATM charged me a $4 fee," and it mistakenly grants me the $4 refund—you still don't want your agent refunding fees that don't exist, but on the scale of things, it's not as bad as violating federal compliance.

So when you make your rubrics, you define these exact rules and error tolerances. And only when it passes this rubric do you accept to put this type of thing into production.

Host: Mhm. It's a trade-off. If you want to use something smart, you need to tolerate some error, but in certain areas much less than others. And sometimes you will reveal the rubric itself to the agent as a form of context so it knows what rules it's supposed to follow.

Great. And I want to summarize for my audience that this is an important mindset shift: before, a lot of people tried to tell the agents how to do certain things step-by-step. Now you just tell the agents the rules of the rubrics, what is the desired outcome versus what is not, and you trust the agent to figure out how to get there. You don't specify the rigid process because you trust the agent to find the better process.

Speaker: Sure. But: trust, but verify. That is to say, in as much as you trust an agent, you come to trust it in part because—like if I think about M365 Copilot, we have been doing rubric-based evals on our own M365 Copilot for over a year now. And I trust M365 Copilot pretty implicitly for a lot of stuff; I use it every day. And the reason why I trust it is because we have this whole eval team back here that's evaluating all of these different scenarios.

And meanwhile, all of us, the many millions of us that are using this thing every day, are stretching it in 15 different ways all the time. And in Copilot, by the way, there is a thumbs-up and thumbs-down button. If something doesn't work or doesn't work to my satisfaction, I'll give it the thumbs down, and I'll give a reason, too.

Because I know that down the line, there is this eval team, and eventually what's going to happen is my thumbs-down and the reason that I give in there is going to go into their eval set, so that in the future, this thing that I am trying to do will work much better.

Host: Yeah, I think this is intuition for managers, but for developers and individual contributors, it may not be immediately obvious. When you say trust but verify, you verify by systematic cross-check, not by manually double-checking every single line of code the agent writes.

That's super cool to see. We look forward to not only the concepts, but also your live lab demos at AI Shanghai.

Speaker: That's right. Obviously, I'll be showing our versions of these things at AI Engineers Shanghai, which I think have come out quite well in our products. A picture says a thousand words, and live demos even more so.

Building evals is a crucial habit, and increasingly you do find folks that are focused on that as a job. We have seen new titles emerge over the past couple of years: Eval Maintainer or Evaluator. "Calculator" used to be a job title decades ago before electronic computers, and now new roles are emerging again.

Host: Along with these new roles, for enterprises thinking about adopting AI, what are the best practices you've observed in the field? Can you name a few?

Speaker: Well, in terms of general best practices: One, as I did say, building rich context and rubrics that are adaptable.

Two, and I always say this for AI: you kind of need to have an abnormally long pilot period. We all have this pressure; we all want to release these things tomorrow because it's easy to put them together. But the ease of putting these things into production has resulted in people putting a lot of bad ideas into production, too.

So you want to have a pilot period. You want to have your users interact with this stuff, really get their feedback, and then go back and incorporate that feedback into the rubrics again, so that when you do these evals, it's acting the way your users truly want it to.

Host: With AI being probabilistic rather than deterministic, and requiring this long pilot period, do you think enterprises need more services than pure software products? Such as FDEs (Forward Deployed Engineers) to help enterprises adopt AI, build evaluations, and establish rubrics?

Speaker: Sure. Well, there it is: another job that didn't exist before the AI revolution is Forward Deployed Engineers. We have that at Microsoft also; that's called the Microsoft Frontier Company. It is a bunch of FDEs that go into our customer sites and help them work on the best practices for putting AI into production.

That helps, but at the same time, I think every engineer should have a baseline understanding now of what AI is, how to use it, how to provide context for your agents, and how to write these rubrics.

The challenge is it's changing very quickly. For me personally, it's my job; I'm literally in this all day long and most nights hacking on this stuff. It can feel insane because every day there's something new. But nonetheless, I think you owe it to yourself to stay abreast of these things.

And what I always tell people: mess with everything. And I mean everything you can get your hands on! Try Microsoft Copilot, try other coding agents, try consumer stuff, try Suno and make yourself a song. I think Suno is cool. I play piano, but the guitar I'm still learning back here. Try all these things, because it really gives you a feel for what they are and how they work in a way that no amount of podcasts or YouTube videos can give you—firsthand experience will.

Host: Yeah. Just this morning I was giving a lecture at Columbia, and I told the students: treat learning AI as a craft, like learning swimming or snowboarding. You can never learn that just through watching videos; you have to do it.

If I summarize our talk today, starting with the question of frontier models and harnesses to the actual product: we talked about evaluations, agent identity and permissions, building the infrastructure to extract tacit information and feed that context to AI. And at the center, it's really the people who do all this and grant permissions.

Thank you so much! Look forward to having you at AI Shanghai and watching your live demos of all these amazing things.

Speaker: I look forward to being there. Thank you.

Host: Thank you.