Today, I'm going to talk about voice agents, which is a very hot topic this year, because people believe the technology is now ready for practical use and commercial deployment.

Today, I will share two of our past projects based on voice agents, highlighting some lessons and best practices on how to help customers utilize them.

Let's get started.

First, what exactly is a voice agent?

Basically, it's an agent with a voice interface.

Users interact with the agent through a voice interface.

We hope this offers a more natural way to interact with large language models.

There are two key aspects here.

First, it's about real-time interaction, which is somewhat different from video generation, which can be done offline.

For voice agents, the response must typically be within one second; this refers to the end-to-end latency.

Secondly, in most cases, people aren't looking for casual chit-chat or companionship; instead, they have specific tasks in mind.

For example, you might want to provide customer support, retrieve information, or sell a product.

Even companionship can sometimes be a task with specific goals.

It's not just casual chat; you have particular goals, perhaps telling a story or doing something else.

So, this is essentially what a voice agent is.

In the remainder of this talk, I will present two examples demonstrating how we built these agents and the lessons we learned.

Here's the first example.

and I really need to find a way out of here fast.

Okay, I guess you're cutting straight to the chase here, but at least can you tell me a little bit more about yourself? Like,

okay, let's see.

All right, I'm a sophomore at Mariana University studying astrophysics.

So, this is a game we started about two and a half years ago.

So this is the app. Um this is the user.

So this app is like this guy is called Stella. Um the user going to do voice interaction with Stellar to help her.

So you can think that the whole story now is driving by how the users uh inputs.

The task here is open water game.

So like this this sentence is copy from the game which is already launched uh two two months ago.

So basically Stella landed on a alien um planets. So it's that the aircraft is crashed here.

So then she asking the players to help like okay it's so like I so um strange word uh so many options and I feel a lot of emotions here.

So you're using voice to interact with player to let the player to help her to escape um the planets.

So that's a very large world setting. That's only the first game. Uh the is a kind of trial game in a very large series.

So what voice agent play here? The role here is you need to be a both a game designer and actor.

The role here is you need to be a both a game designer and actor.

The game designer means that you want to design the the story which is make sense is fun to play.

So then the agent want you want to create a dialogue that match the character setting.

then for the game designer you need to guide the um when you when the user interact with the game you want to guide um the story line which is if it's just a single line story is not agent right now kind of complex structure or even graph structure and some something have freedom here.

if it's just a single line story is not agent right now kind of complex structure or even graph structure and some something have freedom here.

The the issue here is that you want to have really good game like if if you for how to write a book, how to write a game plot, it have a lot of uh principle there like you want to have uh the all the stage order pace all the all of the things make the the story looks uh interesting.

you want to have uh the all the stage order pace all the all of the things make the the story looks uh interesting.

The other things like it's a game you expect people to have thing to upure input with you like the T and the player to try all the boundaries the agent must be within their setting sometimes like this game on a sci-fi world like two maybe 2,000 year later and you have a random chat setting if you say okay what's a movie you uh you uh you you watch recently if you pick up a movie right now is Maybe wow you watched the movie like 1,000 years ago.

sometimes like this game on a sci-fi world like two maybe 2,000 year later and you have a random chat setting if you say okay what's a movie you uh you uh you you watch recently if you pick up a movie right now is Maybe wow you watched the movie like 1,000 years ago.

So and but the thing is like all the language model trend on the current data now you want to how to move all the settings to your future words.

So I share a particular uh like earlier log here um show how um some Chinese task.

So this is the lock uh on the very early earlier states um is still in Chinese.

So the idea here stellarify some foods asking the player to say okay which food I going to choose the player the the player settings here I will help you.

So I think it cannot eatable you cannot eat anything. Um so then oh the standard first thing like you need to find some meat to read.

So you have a rack to search how to play how to catch animals here but the the the answer is you cannot this you didn't see any animal yet.

So you have a rack to search how to play how to catch animals here but the the the answer is you cannot this you didn't see any animal yet.

So the stellar response like okay I want you to eat med but I can only have uh vegetable here.

Um the player don't want to help. Stella say okay I really need you to help.

Uh player still like I don't want to help.

The thing is like if you stuck here then the story can not move on.

So um the prompt is saying like uh okay after three trials just move um choose something by yourself.

And so, but then it's a random choice. Stella is dying.

So, say, "Okay, he's dying."

Um, but the the the player say, "Okay, you got you're going to die. It's not a nice guy here, but you need to be nice here."

So, the challenge here is like it's open the game. Um, but your response should be make sense like it's a open word again. It's like three or maybe one 2,000 year later.

Not every word setting specified lot of thing like you can uh when you develop the game the game designer cannot write anything for you.

You need to think okay that thing make maybe makes sense make sense in a 2000 year letter also need to be engaging and fun.

So that's that's again that's not the chatbot. So that's all the um all the challenges here.

So what do we do is like the project launched two two years ago at that time you have GBD4 but it's very expensive and we did some calculations thinking if you use a GP4 like okay that's a huge loss of the revenue and at that time the best model is lama 2 right now uh at that time lama 2 is not strong enough so uh what we did at that time is we actually pre-trend 30B model uh with kind of five trillion tokens so but this tokens enriched on the fiction G role play data the performance kind of match on the llama 2 on general task little bit better on the role play and but the the lesson we got is like okay pre-trend a model take a few months and even that you can of outperform nama 2 but you have nama 2b so like if spend too much time on pre-trailling uh you maybe this the the progress isn't so great so that's kind of the lesson we got That's we're going to say why that's maybe a bad choice.

The project launched two years ago, at that time you have GBD4 but it's very expensive.

And we did some calculations thinking if you use a GP4 like okay that's a huge loss of the revenue.

And at that time the best model is lama 2 right now uh at that time lama 2 is not strong enough.

So uh what we did at that time is we actually pre-trend 30B model uh with kind of five trillion tokens.

So but this tokens enriched on the fiction G role play data.

The performance kind of match on the llama 2 on general task little bit better on the role play.

And but the the lesson we got is like okay pre-trend a model take a few months and even that you can of outperform nama 2 but you have nama 2b.

So like if spend too much time on pre-trailling uh you maybe this the the progress isn't so great so that's kind of the lesson we got That's we're going to say why that's maybe a bad choice.

Another thing we did is like okay because GPU is so expensive that's two years ago and we spend a lot of effort to actually build data center by ourself.

So if you own a data center the cost is much lower.

And then we move to post training.

The post training is that the key thing here you have very complex story line workflow.

That's the example that not the real one and the real one is much more complicated.

And then we have pens of two 20 labelers.

We need to train the labeler to be a good game designer because like is particular way how you response.

Um then ranking and evaluating all the model uh preference.

So using these two we spend kind of quarter here and can outperform GP4 on this particular scenario for all the scan we use human to uh play.

So you can outperform GB4 but the question here is like okay that's a single again that's a the tiny bit of a whole open world again.

I how what if you want to do multiple games.

So we get another face is that we want to expand to a broader range of games and characters.

So you can less reliance on proper engineer at that time proper engineer is very complicated right now it's like and even different versions of GPD4 is very sensit sensitive to proper engineer.

At that time proper engineer is very complicated right now it's like and even different versions of GPD4 is very sensit sensitive to proper engineer.

So the thing is like yes we can help but you want to design game designer to write um prompt engineer.

So the idea here at that time is like it's still like one and a half year ago. The idea at that time is like you need a pre-train a real world model can distinguish which one's good which one's bad and because you cannot rely on humans to do it.

So at that time we first train a re reward model to tell you which response is good in this game setting then you can post train another model for it.

Um the one important lesson we got is like even this is for game sounds like simple but it's still a lot of things like instruction following you need to make sense.

So the model still need to be very general enough.

So even that you train the in domain model the model must be good in general task.

If you think okay the best open air or best like closed uh API is like the score is 90 but in your application the general task near be 85.

If you're lower than that one you're going to think you have a you have a sitting flaw on on your task.

So that's the f you first guarantee the general task is good.

Then for your particular task we kind of create a in domain evaluation.

So you have lot of um character settings a lot of scene settings uh for different gen settings you want to make sure like under this setting you follow all the settings and the response is good follow the instruction follow the sync and then once you have this uh benchmark you can tune the model so that you can be be the best uh compared to others.

And then once you have this uh benchmark you can tune the model so that you can be be the best uh compared to others.

I think that's a very general pattern that you care about in domain performance.

The key thing here like you really want to develop a really good in-domain evaluation task.

So you can say I I can see the model improve on this one but at the same time guarantee your model performs well on all the other general task.

So the lessons we got is like the intelligence came from pre-training.

So after we finish the whole projects but then we think backwards all the all the big improvements from pre-training on massive data.

So it makes us think rethink maybe we give give up on the pre-training maybe a bad idea. We maybe spend another quarter on pre-training.

So that's kind of the lesson we got and but still like it's it's still limited.

The dialogue quality declines after 50 terms is still right now like given the complex setting after 50 terms conversation then you think that maybe the model kind of be can the intelligence much lowered also the models nowadays still drug uh struggle with complex world setting and you have multiple car uh characters so it's still hard right now even like if you look all this voice model all this video model kind of two to three uh characters that's the limit even for the the text part like if you have four set four characters it's very challenging right now.

given the complex setting after 50 terms conversation then you think that maybe the model kind of be can the intelligence much lowered.

also the models nowadays still drug uh struggle with complex world setting and you have multiple car uh characters.

so it's still hard right now even like if you look all this voice model all this video model kind of two to three uh characters that's the limit.

even for the the text part like if you have four set four characters it's very challenging right now.

The other thing is like in the demo you see the latest is big is a term by term based.

So all the projects we here is focus on the larger language model itself.

Then the lesson we got like if you really want to truly humanlike interactions you kind of need to tune the architecture a little bit is not by the traditional three component and architecture.

So that's become our next projects.

Is this a good time to talk?

Actually, I'm just about to head out the door.

No problem, John. I know how busy things get. If you'd prefer, I can give you a call back at a time that's better for you.

To be honest, I think I'm all set. I already have health insurance through my job.

That's great, and I'm glad you're covered. A lot of folks I talked to do have something through work, but many don't realize there are options that could lower their out-of-pocket costs.

Okay, that's a very different one.

We sell insurance.

So before it's again right now, we sell insurance.

At the first time, I think maybe sell insurance like you can be very creative. You can do whatever you want to sell something.

But in reality, it's pretty very formal.

Right? First, first of all, you cannot call anyone you want to call the the user you called must be submit some information. So, shows uh their interest on your products.

Secondly, it's very highly regulated the whole insurance industry.

So, um let me explain this problem.

So, right now we do AI telemarketer.

The agent row is like is telemarketer.

So right now this particular example is like we sell health insurance through phone and but on multiple countries the thing the requirement you have two requirement you ma you must pass the telly marketing certification like the human you have 80 score to pass this one must be pass this certification before you can launch secondly you have some performance metrics you you're able to sell at a particular threshold like DV you like 1,000 uh customers you call them you must be able to sell particular number of sales also the complaints must below particular number if the people say okay I feel so bad like like you said in not true information or the experience is bad they they're going to complain that insurance company really care about it so the capacity you need here first of all you need to be intelligent you need follow the sales playbook with precise answer.

the requirement you have two requirement you ma you must pass the telly marketing certification like the human you have 80 score to pass this one must be pass this certification before you can launch.

must be pass this certification before you can launch.

secondly you have some performance metrics you you're able to sell at a particular threshold like DV you like 1,000 uh customers you call them you must be able to sell particular number of sales.

also the complaints must below particular number.

if the people say okay I feel so bad like like you said in not true information or the experience is bad they they're going to complain that insurance company really care about it.

so the capacity you need here first of all you need to be intelligent you need follow the sales playbook with precise answer.

We're going to show what is precise answer means.

You need to able to use tools because insurance you have a lot of internal tools to query and some mass like you need to lot of compilations here.

Also need to be very humanike.

And if you call someone maybe this guy's out of door like lot of noise and maybe have some accent and also you need have some um the the voice need be realistic. It's not so rob uh robotic.

And the last one is entering the latency.

When I finish my sentence, your response must be within one second. Otherwise, you feel like it's a little bit not so responsive.

So the precise response like for example, if you respons you can cover up to $600. That's wrong. Totally wrong. you failed this exam.

because the precise answer the precise one is like cover $400 for some common one and like um the $600 only for the front teeth.

So that's it's on the play uh product information.

So if you have any issue with your teeth that's not right.

If you have particular like AB diseases with with your teeth that's the right answer.

If you respond this one it's going to be you you failed the exam.

The other thing the other more challenging thing here similar to the gaming like when some customer and try to asking okay can you can we grab time to talk about the you going to try three times if you cannot hand out you cannot reschedu before three times or you can reschedu later so for example I try the mar try first one the ten mark try first one no thank you then you try second time no then you you try the Third one if the user say uh-huh.

you going to try three times if you cannot hand out you cannot reschedu before three times or you can reschedu later.

so for example I try the mar try first one the ten mark try first one no thank you then you try second time no then you you try the Third one if the user say uh-huh.

So you may be thinking uhhuh maybe interesting like if you think the emotions interesting because you you change the world uh you change like can you tell you okay like explain how it can could benefit your personality p personal.

then you maybe think okay is maybe the user is interest but in reality it's like you need to think you need to find out the voice is impatient.

then given the contact you can think okay I already have tried three times I need to reschedu.

So that's the whole that's the cutting when you have the uh audio as inputs.

So then one key question here how do we do real time?

So I kind of um show examples how different model architecture we have right now.

The first one is the more f the fanciest one is called end to end food duplex.

It means like you have user you have your model is single model.

the user speak to you which is all the waveform come in then you listen to the waveform and response anything during the sync dur during uh the interactions.

So in this case it's easy for user to interrupt also easy for the model to uh do some filling words.

like user say something hey uh say a long sentence you can say yes yes that's right so that's the most natural way.

um but none of this system is in deployed right now that's one or two demos you can try but it's very feels not so controllable.

so in most cases like if you're using uh even GBD40 I think they're using the end to have duplex.

it means that when the user speak you have a voice active detectors dete detect if the user speak or not.

So you have trunk of chunks go to the model and the model going to response the uh previous one.

So that's the um um that's the half uh duplex.

The another one is a ch solution.

still similarly you have turns but here you have two models not just single model.

So these two models the first is understanding model give the audio in generate the text response then the text goes to the generation generate the audio outside.

The last one um is the called uh ch three components like you have um just do ASR which is transcribed audio go to the log gen model and just and then get a response go to the TTS which which is generate audio.

So for this different one, this one is humanlike very human like you because the model can interrupt you.

And the last one is like uh if you go that direction is easy to customize because like you can much easier to adding a new capacity into the agent.

So what we typically use for customer is using the two component chain solution.

So um you have uh for example we using 30B understanding model to generate response but if the if the user query is complex maybe using a fine-tuned larger model to do syncing as a tool use.

so then it goes to one bit generation model to generate the response.

Nowadays all this model is based on single is the same large language model all based on the same uh LLM but you kind of either continually pre-trend or fine-tuned with different data mixture.

For example for the understanding you need to have many of hours of really different quality of audios you maybe want to have a lot of low quality audios also because you want the understanding model generate a response you want to have a lot of text tokens to to to continue as well.

Otherwise, it's just the audio model.

The generation model, you want to have an even more high quality a um hours of audios.

The larger language model, you kind of want to train on some domain specific data.

So this this architecture makes easy to customize because this is kind of like understanding and the generation is kind of general purpose one.

um you you have this model you can maybe can use in different scenario but if you go to particular scenario you just fine-tune this model.

the how to get both intelligence and low latency that's a key for voice agent.

once there's bunch of idea here.

first of all you want to lessen talk and sync at the same time like you you listen and you generate a response sentence sentence.

and then between that while you call the large generic model to think maybe I want to respond uh better maybe I want to do some search better but all the thing can be uh asynchronously.

the other one is like you want to do context engineers like it's a one step be beyond prompt engineer.

that's because for your problem you maybe have a very long context like the product information and also all the playbooks kinds of like maybe uh 100k tokens.

you want to do engine you want to dynamically generate um construct the content the cont context generate uh the prompt.

the Another thing like you have uh you have organizer which is handle different strategy like okay this kind of what kind of user you think this uh user is and then think about different strategy and also do intent analysis for example how to count the hunt and some do uh live task tracking.

so all the thing a lot of uh together you can get both intelligence low latency.

so that's kind of the project progress we did like uh it started this year we partner with uh Fortune 500 insurance leader.

it started this year we partner with uh Fortune 500 insurance leader.

Um so we start here January February you have using chap GD4 you got this like kind of uh 55 score.

but the thing is like you need to pass this line this human is a human performance it's 80 you must pass this line to be launched.

you can see like you struggle a lot and but then you have steadily progress into and how you you can match human it take hands of half a year or three quarters.

actually the lessons here is that the evaluation of the end to end voice agent is pretty challenged like because you need have a real human to make a call.

actually the lessons here is that the evaluation of the end to end voice agent is pretty challenged like because you need have a real human to make a call.

Once you have a core, it's much harder to do like uh like automatic evaluation.

and but that's a key if you don't have this one is it's really hard to know the whole end to end performance.

and this is ongoing is that handling complex product compilation in real time still pretty difficult.

like for insurance you have lot of product compilations how to handle them.

the price different maybe I I okay that's too expensive for me and I want a cheap solution then you need to pick up the right one for them.

The last one the high security setting make the cost higher.

We have the panel discussion talk about if there's only open maybe dominant words under the 2B areas not the reason is because for insurance if it launch in different country the model cannot go the data cannot go out of this country or even more the data cannot go outside the security group of the company.

the reason is because for insurance if it launch in different country the model cannot go the data cannot go out of this country or even more the data cannot go outside the security group of the company.

So either way you can rent um GPT model off uh on your uh account running on your account or you need to you need to develop your own model.

So that's why all the things struggling here.

Um so also that that's why we spend so many efforts to develop the whole models by ourself rather than just maybe proper engineer open source uh just API.

So I showed two examples how we developed voice agent in the past kind of two years.

The lesson we got like the voice agent are pretty highly scalable.

even that the GAN setting the insurance settings are very different but the technology wise same model architecture same technologies here.

only things like maybe data is a little bit different and the evaluation is a little bit different you need to spend a lot of people on that one.

but um the model architecture and also the methods how you post train how you pre-trend how you like all the things it's the same same from game to tenn marketer.

it's very different one game you want to be fun tenant marketing you want to be very precise but handle the user input very carefully.

but still like I think right now it's able to long landing in this areas but still on the day one setting.

the reason is like for game it's just a very simple game right now single character a small word setting.

for tele marketing right now we can maybe sell kind of five different in health insurance for particular company with some certain of compilation compilations.

is how to sell general purpose.

I think in generally this tele market is really good for any product between $500 to $5,000 that's r is really good for this tele mark to sell.

but right now uh if you use this trend model to sell any arbitrary new products you still need a lot of tuning right now.

also there are a lot of other scenarios like before it's like customer service all the things based on larger lo uh just text large model right now you can adding a voice interface to this application.

so there are a lot of applications here.

so I think that's why I think we can we able to land to product right now but still on day one.

so we have maybe another few exciting years to.

Lastly, if you're interested to work with us or partner with us, just contact us.

We have a booth. You're going to be here. Our co-founder will be here.

Welcome to talk to us.

Yeah, that's all.

Thanks everyone.