Good morning everyone. Thanks for being here to the voice and vision session. I'm Hervé Bredin, chief science officer and co-founder at pyannote.ai. I'm going to talk to you today about understanding conversations and what you can do on top of transcription.

I've been an academic researcher all my life until two years ago when I started this company. I worked on this topic called speaker diarization, which I'll introduce a bit later. Over the years I built an open source toolkit called pyannote which focuses on speaker diarization, and that became quite popular in particular since OpenAI released Whisper speech-to-text open source models. Whisper was some kind of revolution in terms of STT—the fact that it was free, that it was very good—but it didn't provide the actual names and tags of the speaker. So people naturally turned to pyannote to combine the two. (By the way, I happen to turn 45 in just one week, and we are almost at 10k stars on GitHub, so please give me a birthday gift by adding your star!)

Let's go to the core of the presentation. You all know what transcription or speech-to-text is. Basically, that's answering the question: "what was said" from the recording of a conversation. You get a sequence of words as output. That's what Whisper does, but without the actual names of the speaker, the transcripts are not really understandable. The next step is to attribute a speaker tag to each of the words. That answers the question: "who said what."

For some applications, that's enough—like for meeting note takers to make a summary and assign action points. But there are many cases where knowing who said what is as important as what was said. For instance, automatic video dubbing: you want the voices to be consistent across languages. Same for podcast intelligence, like tracking a speaker or finding the same guest across different episodes.

But knowing who said what is sometimes not enough to really understand how the conversation goes. Knowing "when" they said it brings even more information. Without precise timestamps, you can't detect that a speaker interrupted another. You can't really detect the small backchannels a speaker makes during the speech of another. If you miss it, you don't really understand that the listener is actually agreeing or nodding. Knowing exactly when this small word was pronounced is very important. Same goes for pauses in between two speech turns, which might convey additional information about my state of mind or the point I'm trying to make.

We can go a step further: knowing who said what, when, and "how" they said it. Are you laughing because I said something stupid, or was I actually funny? Maybe I'm coughing because something makes me uncomfortable and I'm hiding behind that. Stress, disfluency, prosody—stressing particular words in a sentence might completely change its meaning. If a child tells you "the dog ate the cake" versus "the dog ate the cake", the meaning is slightly different. Being able to have a voice AI actually understand this stress and these low-level details brings much more information to any downstream LLM or tools you use after transcription.

Having a higher-level view of the conversation also opens up a new bunch of applications. All of this happens in an acoustic environment that gives you context—whether it's in a quiet room or if I'm in the street. This brings more context to the game so you can take a better decision about what's going on and why a person is saying what they're saying.

This is basically what we are trying to work on at pyannote. But we started with a smaller problem called speaker diarization, and that's really what I want to talk about today.

Speaker diarization is usually just as important as what was said. You can tell by going to the Hugging Face model repository and filtering the audio models by number of downloads. Among the top seven models, the first three are actually related to speaker identity and speaker diarization, and the others are STT. That shows speaker diarization is actually quite important in the community.

Going into a bit more detail, speaker diarization answers the question: "who speaks when." Starting from the recording of a conversation, the first step people usually do is Voice Activity Detection (VAD)—telling whether anyone is speaking at one time or not. Then you can segment those speech regions into smaller speech turns. This segmentation includes finding speaker change points and finding regions where people are actually interrupting each other or making some kind of backchannel (saying "hm" or "okay"). You don't want to miss these small speech turns because sometimes they convey the most important information of the conversation. If you miss a small "yes," you have no idea what the state of mind of the speaker is.

But that's not yet speaker diarization. Diarization goes all the way to assigning a speaker identity to each speech turn. We usually don't have any prior knowledge on the actual number of speakers in the conversation. If you are a meeting note taker, you might have the list of attendees, but that doesn't mean two people didn't join from the same channel, or an uninvited attendee joined. That makes the problem difficult because we don't know in advance the number of classes we're supposed to detect, contrary to classical machine learning problems. Also, we don't know the identity of the speaker. It outputs "speaker one," "speaker two," and they can be permutated and still be correct from the point of view of the evaluation metrics.

Even though the community has been working on this topic for a long time, it's still not solved. There are other reasons: we have to handle overlapping speech, take into account very short speech turns, handle imbalance between the speech time of multiple speakers, and deal with acoustic condition problems that any speech processing task has to deal with.

To show you how diarization systems are evaluated, you'll often stumble into the acronym DER, which stands for Diarization Error Rate. In my demo, comparing a community diarization pipeline to a manual reference, we see three kinds of mistakes: confusion (it got the speaker wrong), false alarms (it detects speech when there's actually no speech), and missed detections (often happening during overlapping speech when we only detect one of the two speakers). The DER is basically the sum of confusion, false alarms, and missed detections divided by the total duration of speech.

I'm often asked how well state-of-the-art speaker diarization works today. It's a difficult question because it really depends on the use case. For conversation telephone speech, the best system goes down to an 8% Diarization Error Rate. But if you are in a restaurant with many friends and lots of background noise, even the best system reaches a 41% Diarization Error Rate. So it's far from being a solved problem, but we are working on that.

Once you have speaker diarization and transcription, it shouldn't be that hard to actually find a speaker-attributed transcription, right? It's just a matter of assigning a word to a speaker. But actually, that's not that easy.

The first reason is that most speech-to-text models are actually trained on single-speaker data. As soon as you apply them on multi-speaker data with overlap, speaker change, and all kinds of mess, they fail miserably. For instance, if you look at the Open ASR leaderboard on Hugging Face, the Nvidia Parakeet model reports an 11.4% word error rate. But when we apply the very same model on the AMI dataset (meetings between 4-5 people using a distant microphone in the middle of the table), we get 26%. The difference is because on one side you have single-speaker speech, and on the other side you have multi-speaker and distant microphone speech. That's why it degrades a lot.

So when doing speaker-attributed transcription, it might go wrong because STT doesn't work great and doesn't generalize well to multi-speaker recordings, distant microphones, cross-talk, interruptions, or code-switching. But it also might be because of the actual reconciliation between diarization and STT timestamps. STT does not transcribe overlapping speech well, the timestamps often disagree between STT and diarization, and sometimes diarization will detect speech that the transcription will not transcribe (and the other way around).

In our demo with Nvidia's Parakeet model and our precision diarization model, the question is how to assign a speaker to each word. Sometimes the timestamps are a bit shifted. Or there's a word in between two speech turns, and which one do you assign it to? According to diarization there are two speakers speaking, but only one word from the transcription.

That's why at pyannote we came up with a new STT orchestration system that takes care of the reconciliation between the two. The trick to actually solve this problem of reconciliation is partly available in our open source community model, which we call exclusive diarization. Basically, what we do is we find a way when there is overlap to actually select the most likely of the two speakers that will be transcribed by the STT model. That simplifies the reconciliation between the two. This is not part of the model training—we really plan to support any kind of STT without having to change or retrain the STT model itself. It's supposed to work with any STT, even fine-tuned ones you might have internally for your particular use case. It correctly manages to interleave the overlapping words from the speakers.

This demo and tools are available in the pyannote tutorials GitHub repo, where you can play with the pyannote open source toolkit, pyannote.metrics for evaluation, and our interactive visualization widgets. Thank you very much!