Our mission is to unlock the unrealized potential of voice AI. So basically we train voice models, speech-to-text, text-to-speech, speech-to-speech, whether it's transformation, translation, speech-to-speech dialogue, and so on and so forth. We want to be main model provider for voice for everyone building voice agents and voice solutions. So we are not working on orchestration, we are not working on specific verticals. We just make building blocks for people who want to build voice AI.

This is a spin-off from a non-profit lab we created 2 years ago with the funding from philanthropists including Eric Schmidt, Rodolphe Saadé and Xavier Niel. The main idea was to create a lab that has open research and we focus mostly on speech. So we developed Moshi which was the first speech-to-speech model for conversation, speech-to-speech translation, Pocket TTS most recently a CPU model. And we decided to also create this for-profit structure to make a product that can be used in production beyond open source.

Speech-to-text, LLM, text-to-speech, that's the classic Cascade. In our case, we do streaming speech-to-text, streaming text-to-speech with voice cloning, semantic VAD, the classic stuff. In a human conversation, you need the entire stack of understanding, producing an answer, and pronouncing it to be around 200 milliseconds. None of this will allow to have a conversation that sounds human.

Today, if you have a voice agent that is supposed to use a tool, you're going to wait. The tool call, we are fighting for latency of the TTS, trying to grab 10 milliseconds, 20 milliseconds, and then you have a tool call or open router that is going to have a latency between 500 milliseconds and 4 seconds. I think now the main bottleneck is becoming the tool call, which is very unpredictable. One solution to do that is to have fillers. So basically, your LLM it splits into two things. It sends a tool call. And while it waits for getting the result back, it can keep the conversation going in a natural way, and then it retrieves the result, and it tries to insert it back naturally in the conversation.

Speech to speech is the idea that now instead of having the three blocks, you only have one that does everything together. That reduces latency a lot, but that's still not a human conversation. In particular, because every single speech to speech model except Moshi is half duplex. What it means is that the model is either listening or it's speaking. And it cannot handle the ambiguity of human conversation where you can have overlap between people speaking on one another.

Full duplex—that's a human conversation. In Japanese it's a sign of politeness to do a lot of back channeling. You will say mhm, uh-huh constantly when the other person is speaking and you get up to 20% of the time that is overlapped. Paralinguistic understanding is understanding all the cues that come from the way people speak. Technically, that is in Moshi, that is in any speech-to-speech model because this information is not lost. However, if you don't exploit this information to make your model say relevant things, it's never going to exploit it.

Voice is very expensive. The voice mode of most hyperscalers is run at a loss. It's a gigantic multimodal model, and they lose money every time you use it. LLM now is almost nothing in terms of cost. TTS is really what is going to consume most of the money. I saw people burning their fundraising in TTS bills. Another aspect is privacy. You will be more comfortable if all your private data is local.

To solve that, our first step is Gradion Phonon. It's an on-device TTS. For us, on-device means it runs on a smartphone CPU. It's a very small model, less than 100 million parameter, and for its size, it works quite well. It's better than all the existing on-device models. Kokoro is a good one, but it doesn't have voice cloning. This runs on the smartphone CPU, which means that you can use that to power any kind of voice application without paying a single cent of API fee.