构建抗打断的类人语音智能体：LiveKit 的端到端工程实践

Um all right, we're going to talk about voice agents. And most of the agents that we deal with on a day-to-day basis, um they they do things in the background, right? They they write code for us, they do research, we give them something to do, a task, and we walk away, we come back, and we don't really care if they take an extra 10 seconds, an extra 10 minutes, maybe sometimes 10 hours, right? Um and we expect that. But at some point in the last year or so, you have probably interacted with another kind of agent, a real-time agent. Maybe when you called your pharmacy, maybe uh when you called the airline. I talked to one just the other day when I called a car dealership. And at some point in that conversation, you did the the most human-like thing possible. You spoke over it. You interrupted it. You confused it. And the entire conversation fell apart, it just didn't feel natural. And then you did what I do every single time, operator, hit zero, like get me to a human, right?

So, how do we fix that? How do we make these agents as human-like as possible?

So, we're going to talk about how can we build human-like voice agents for financial services.

Well, latency is part of the equation. Latency has a finite budget, and we'll talk about what that is.

Um but the keyword here is human-like. How do we make these voice agents as human-like as we can? That's the goal.

So, there's quite a few people that are talking about voice lately. Um some of these might look familiar. We've got this this guy Sam. Um he's excited about voice models. He says uh that they're starting to change the way they that we interface with AI. Isn't that true? And then this this um weird name here in the bottom right, Swyx. I don't know if you've heard of him. Um he says oh uh that this is the OG modality. Um the correct modality for aligning AGI to our intents is very good voice AI engineering. That's that's that's a pretty bold statement there. And then like get here says we learn to speak before we learn to read. Right? Voice is the most natural interface that we have. So, how can we make it as natural as possible and human-like as possible? Quick show of hands, who who here has ever built a voice agent and deployed it to production? All right, I see a few hands. All right, so some of you know the challenges that you might face when doing this.

Okay, so that's that's cool. Segue here that OpenAI actually chose LiveKit to power that that functionality there. So, it's a very good example of a good use case or a good uh good example of a good voice agent. Um LiveKit is the open-source real-time framework and infrastructure for these human-like agents. Um agents that can hear, see, and speak to a person on the other end. And so, we're not just talking about voice, actually. Um there's there's multiple ways to interact with with agents, voice, video, data streams, uh even physical AI. We have a robotics division as well, but voice is the entry point voice is is the most natural modality, and voice is where there's a lot of challenges. So, we're going to focus just on voice today.

Hundreds of models, dozens of providers, and quick show of hands here, who has ever picked a model based on a leaderboard? Like you pick them whatever model's at the top of the leaderboard, that's what we're going to use, right? Well, we're going to talk about why that does not work for real-time agents. We can't just pick models off the top of a leaderboard. Um because leaderboards, they score these models, these components, but we're shipping a conversation. We need to be able to score them together. So, we're going to talk about how we can do that.

So let's let's say that we fall into this leaderboard trap. We just pick a model. Um then the the first question that people ask is always, you know, which model should I pick? So they use a leaderboard, they pick a model, but that the actual question that we should be asking is which stack should I ship for my use case? And so that is what we're going to figure out. How do we answer that question?

Let's uh say you're you're testing your TTS provider. And TTS provider on the dashboard, it's it's uh reporting time to first byte. And that looks really fast, 210 milliseconds time to first byte. That's super fast. Um but we've actually tested lots of popular providers that are shipping up to 3/4 of a second of silence after that first byte. So a byte does not equal audio. So what we're actually looking for is time to first audio. How much time does it take for our user to actually start hearing something? So time to first audio is what we should be measuring, not time to first byte. Um so imagine, you know, the payment went through and and and you you're asking, you know, um you know, did my payment go through?" And then you hear 2 seconds of silence. That's not very reassuring, right? So, we need to be able to measure this this gap that the caller hears.

Now, the same story for lang- uh large language models. So, time to first token is what we get in text. It's it's a text-based measurement, but tokens do not equate to audio, either, right? Your TTS can't start speaking on a token. A TTS model text-to-speech model actually needs an entire sentence to start speaking. Um and so, that's what we need to measure is actually time to first sentence. Um because a an LLM that is very fast at producing tokens is not always the same LLM that produces the first sentence the fastest. So, time to first sentence is what we should be looking at.

Well, we can't. That The answer is we can't. But ask yourself, what would a human do in the situation? So, is a human just going to go silent while the system is doing something? No, they're going to say, you know, "Give give me a second. I'm looking that up. Um system's running a little slow, almost there." Or they're going to move on and they're going to ask another question while the system is doing something in the back end. So, that's that that's the human-like behavior, multitasking, right? We can do that. So, why can't agents multitask and do that?

Well, that was a trick question because they can. They should. They should be able to. And this is the shape of that. Um this is the only code that I'm going to show you uh in this in this talk, but this is a normal tool call and the the one line is what makes the difference here. Um, this one line, this update, this async await, this awaiting this tool call is is the whole trick here and this is what's handing the mic back to the agent. This is allowing the conversation and the tool to stop sharing a thread. They're just they're separate now. And so um, this is what again makes the agent more human like because it's able to keep the conversation going as something is going on in the back end.

Thank you for calling The Gilded Rose Hotel. How can I help you? Yeah, I'd like to book a room. What dates are you looking at? Uh, next week, Wednesday through Friday. Next Wednesday, July 30th to Friday, August 1st? That's right. How many guests will be staying? Two. I have queen two beds, king double queen, a suite or the penthouse? Would you prefer a city or ocean view? Ocean. May I have your name for the booking? Jesse Hall. Is that Jesse Hall? Yes. What's your email? me@me.com Is me@me.com correct? That's right. And your phone number? 555-123-4567 Is that 555-123-4567? That's right. I'll need a credit card to hold the room. What's the the number?

We'll stop here.

So, again, latency, that's a budget, and and we know how how to fit into that, but how can we be as human-like as possible? That's the goal.

So, this is what we have to do. We test the conversation. We take a complete working agent So, that hotel receptionist that you heard just a minute ago, we took that. It has a real back end. Um it's an actual It's not just a demo. It actually works. Uh and it's open source. You can check it out. So, we take that agent and we put simulated callers against it. And these callers can pursue goals. They can change their mind. They can interrupt. They can mumble. We can add background noise. We can make sure that this is a real-world scenario, and we run all of these simulated users against that agent. Um we can run text-based simulations, which just tests our business logic. Or we can also run audio-based simulations that tests the speech recognition capabilities. It tests turn-taking. The entire um the entire voice pipeline is what's tested there. And so, we take the scenarios, and the scenario should describe what the caller wants and what actions the agent should take, what the outcome should be.

And then we score that. We score those outcomes. And so, in the hotel benchmark, you know, did the correct reservation hit the database? In a financial benchmark, you know, did the correct authorized amount get charged, right? And so, we we check these agents. We make sure that every all the outcomes are met. And then we don't score it based on just the completion alone, right? Because we know that agents can game completions, right? We have to make sure that they took the right steps. Did they take 10 steps when they only should have taken two steps, right? Did they hallucinate a function call? These are the things that we have to test, the entire conversation there. And then, we swap the stack out. We We have the same agent, the same scenarios, but we change the models. And then, we run that loop again. The identical input Every model sees identical input byte for byte. We swap the models, we run the loop. We swap the models again, we run the loop continuously over and over, and we are building at that point a benchmark, a personalized benchmark, for our use case, for our agent. So, there are as many benchmarks now as there are agents. You run this against yours, and you find out which models work best for your situation. That is the only way to find out which model is best for you.

And so, here is the results of that benchmark on our hotel agent. This is the LLM that ranked the highest for our hotel receptionist agent. That's Gemma 4 31B on LiveKit inference. You notice on the second line, the same model, Gemma 4 31B on another provider, was slower. So, LiveKit optimizes its inference serving for voice workloads. And LiveKit is model agnostic, so you can use any model that you want. So, again, you plug those models in, and you figure out which one is work it will work best for your use case.

And so, here's a checklist. You can take a take a photo here. We need to set our latency budget that the caller hears. Measure time to first audio, time to first sentence. Confirm critical dates. Anything that's financial money, we need to make sure that we have those those checks in place. Verify identity before an account is accessed. Keep listening while slow tools run. Keep the conversation going. And then run the complete conversation repeatedly through simulations. Test the final state, privacy safeguards, unnecessary tool calls, turn-taking. All these things and then when you find these failures, you turn those into regression scenarios and you keep testing over and over again.

So, start with one one of those workflows or failures that you care about. Keep Keep iterating then. Run it. Listen to the conversation. Check what happened on the back end. At some point in the very near future, I hope, that we all call one of these voice agents and we talk over it. We confuse it and it just rolls with it and it's and it's a very human-like experience. That's That's our goal, but here's what I need from each and every one of you. I need you to go build that. Thank you.

Woo!