The loudest voice on the call is usually not the user

Our voice agent can tell every speaker apart and still spend the whole call talking to the wrong person. Let's know why the loudest speaker is often not the one your agent should follow, and how we solved it.

Bhavana Praneeth · Oct 21, 2026 · 15 min read

The loudest voice on the call is usually not the user

The call that broke our first design

This transcript is synthetic, but it is close to real calls we have handled:

[00:04] Speaker 2: "So I told him about the offer, and he said the price was..."
[00:11] User:      "haha"
[00:12] Speaker 2: "Right? And then we got to the paperwork..."
[00:19] User:"yes"
[00:20] Speaker 2: "exactly. So anyway, the next step is..."
[00:27] User:      "okay"

The person on the phone said three words. Most of the audio came from someone standing next to them.

Our first rule picked the main speaker by looking at the start of the call: whoever took most of the first few turns won. In the demo that was almost always right. The demo had only one caller, no background noise, polite turn taking, and a script that begins with "Hi, am I speaking with...?" On this call it picked Speaker 2. Even if we generated test data and played it back, the rule might work for that scenario but not for others.

The mistake started from there. The agent aimed its next question at the wrong person. The call summary attributed Speaker 2's statements to the user, and the outcome label was wrong. The user, who had said "yes" to the one question that mattered, was treated as background noise.

What made it hard to catch was that the diarization looked fine. The model put the two voices in different buckets, kept the labels stable, and reported a low error rate. We identified who spoke and when, but we still failed at the most important task: identifying the person the agent was actually talking to.

Real calls do this all the time. The user says "yes", "no", "okay", "haha". A background speaker leans in and answers for them, or a family member takes the phone. The agent itself talks for half the call, because it is doing the explaining. And the first thirty seconds are usually hold music, greetings, IVR prompts, and "can you hear me?", none of which is the conversation.

Four problems hiding in one ticket

Duration seems to be a proof. It’s easy to explain, easy to measure and easy to get right in most cases. But that only tells you who was active during the call, not who matters in the conversation. The agent does most of the talking in a support call. Usually the one who decides is not the one who speaks the most. And the only person you need to identify could be the one who says only three words.

Once we noted the problem down, the confusion cleared up. "Find the main speaker" was really four questions :

Problem	Question
Diarization	Who spoke when?
Speaker identification	Which known voice is this?
Role identification	Is this the customer or the agent?
Main-speaker selection	Whose intent should the agent follow?

The first three are inputs. The fourth is a decision.

A diarization model clusters audio frames that sound similar. It doesn't know who the main speaker is, who is a background speaker and who is a person in the back seat. Even a perfect model would leave that blank, because the answer isn’t in the audio alone.

What we tried before that

We didn't see it at first. Diarization is the “who spoke when” tool, so we tried it three ways .

We started with the diarization that is built into a speech-to-text provider, with an LLM filter on top. It only worked with one provider, and often split a single caller into multiple speaker labels.

It was correct but every single call had to have the GPU service, another audio stream and an extra hop over the network. On a CPU it took about real time, which leaves no room on a live call. We parked it.

Then we made an experiment of skipping labels and instead cleaning the audio with a speech enhancement model. Enhancement is supposed to preserve all speech, so the background voice was there. It also cut parts of the caller’s own speech in our tests in Hindi and Hinglish.

The agent does not need to know who everyone on the call is. It wants one little answer at a time: is this the caller?

Our approach: three signals and one memory

For every call, we created a voiceprint. It works with any speech-to-text provider, operates on the agent's CPU, and fails open, meaning that the turn is retained if something goes wrong. The system maintains a single belief about the caller's identity, updates it with evidence, and modifies it only when the evidence is compelling enough, rather than selecting the primary speaker at each step.

That belief is supported by three types of evidence.

The first is acoustics. A small speaker encoder turns each turn into an embedding, a list of numbers that describes the voice. We compare it with the caller's voiceprint, which builds up over the call, not with whoever spoke last.

The second is audio quality. We only embed the speech in a turn, never the silence around it, and a turn needs enough speech before it can teach us anything. A clipped "okay" is weak evidence even when its embedding looks similar, so very short turns are kept without being judged.

Thirdly, there is the conversational context. When acoustics do not suffice for making the decision, we provide a context to an LLM and a narrow question: Is this utterance directed at the agent? The LLM knows that a response of “yes” would answer the agent’s question; “he is not interested” would be considered a side conversation, while “let me pass you to my colleague” – a handoff. However, it never listens to the voice, and therefore does not know the identity of the speaker. This is why it is allowed a vote and not a veto.

We thought that loudness could help, too, because the person holding the phone would be generally the loudest one. We were mistaken. The caller can sometimes speak louder and sometimes quieter; also, a microphone near another speaker would make him louder than the person holding the phone. In this layer, loudness is just logged; it never makes any decisions. (It turned out to be quite problematic in its new role.)

How the system works

The memory is the caller's profile. We learn it from the caller's first real turns, which is called enrolment. During enrolment every turn is kept, and the LLM votes on whether each one is addressed to the agent. A voice only becomes "the caller" when its turns sound like each other and the LLM agrees.

From the enrolment turns we also learn how much the caller's own voice varies. Each enrolment turn is scored against the others, never against itself, which gives us the caller's normal range. Every later turn gets a drift score: how far it falls below that range. Small drift keeps the turn. Large drift drops it. Anything in between goes to the LLM.

What happens to each finish turn?

A dropped turn never reaches the LLM that runs the conversation. The agent politely asks the caller to move somewhere quieter instead of answering the background voice.

The profile has to be able to change. If the user hands the phone to someone else and the profile never moves, the system stays confidently wrong for the rest of the call. So the profile has an exit rule, and we want more than one signal before we accept a handoff:

the new voice keeps showing up across several turns, not one;
the context agrees that the new person is now answering the agent;
ideally, someone says so out loud: "Hold on, let me pass you to my colleague."

One label change is not a handoff, and neither is one LLM answer. A false handoff is usually worse than a late one, because the agent starts ignoring the person it was supposed to help.

Back to the call at the top. The user's "haha", "yes", and "okay" are too short to judge, so they stay with the active profile. Speaker 2 sounds different, and the context marks those turns as side conversation rather than answers to the agent. The profile stays with the user.

Now, just make one line different. Let’s say that Speaker 2 says, "Hang on, let me transfer you to my colleague," and there’s a new voice that responds to the agent’s next query. This new voice sounds different, it’s clear that the call is being transferred, and the pattern continues from there. That is when the speaker profile should shift.

The first production version taught us a few things the demo never did. It enrolled one caller from near-silent "Okay" and "Yes" windows, learned what silence sounds like, and then kept dropping the real person. Another time we scored turns against a profile that already contained them, so every normal caller looked like they were drifting. Each fix was small, such as never enrolling silence and never scoring a turn against itself, but we only found these bugs on real calls.

The second problem: the bot kept getting cut off

The voiceprint fixed what the agent understood, but the bigger complaint was something else. A TV, a horn or a person in the room would cut the bot off in the middle of a sentence.

Turn-based checking is far too late in this case. The bot gets disrupted by the voice activity detector (VAD) that detects sound before words even start. Our speech-to-text transcription does not provide partial transcriptions, so a delay on words will only cost extra seconds in every disruption. It needs to be done from audio data when VAD triggers.

We split it into two questions. The first is whether the sound is a human voice at all. A small audio classifier, YAMNet, answers that in a few milliseconds on CPU. On phone audio, sirens, horns, engines and fans score close to zero, while caller speech scores high.

The second question is whether that voice is the caller, and for that we needed a model that works on tiny slices of audio. A personal VAD gets the caller's voiceprint and then labels every 30 milliseconds of audio as the caller, another voice, or no speech. We trained it on IndicVoices for real speaker variety, on our own phone calls for the line and the codec and the Hinglish, and on MUSAN noise and music, all mixed together.

Short phone clips are hard for any voiceprint. Half a second of speech is only two to four syllables, so the embedding mostly describes the sounds being said, not the person saying them. Phone lines also keep only the 300 to 3,400 Hz band, and most speaker models were trained on wideband audio. So we fine-tuned a stronger encoder, ERes2NetV2, on phone-filtered speech. To check that it learned voices and not phone lines, we ran a test where both clips went through the same simulated line. It held up. Then we retrained the personal VAD to match the new encoder.

Loudness came back here. The idea was simple: the person talking into the phone is usually the loudest voice at that moment. So the personal VAD also got the caller's normal speaking level as an input. The first model learned the wrong lesson from it. When a caller got louder than they were at the start of the call, it decided they were someone else. The retrained model mostly fixed that, but it still struggled with quiet short replies, the "haan ji", "no" and "theek hai" a caller says when the line is bad. Most of those were the caller.

How the frame-level check decides whether to let an interruption through

All of that was first executed in shadow mode. In shadow mode, the model operates in production on calls but does not make any actual modifications and records what would be done. And it is also when we discovered the issue with quiet replies without muting anyone. This also affected the design of the project. Rather than muting the audio, the next step will be allowing all utterances to go through speech-to-text and giving the personal VAD a chance to have its say during an interruption. If it feels that the speaker is another person, the bot will continue its monologue. If it is not sure or the person is not enrolled yet, then everything remains the same.

How we evaluate it, and what it costs

We do not measure diarization error rate. DER is time-weighted, which means that a mistake of 300 milliseconds on “yes” will not affect it much, even if that mistake leads the agent to chase the wrong person. That disparity was highlighted by AssemblyAI in their 2026 write-up, where a very poor speaker assignment on the transcript gave a 15.1% DER and 30.7% cpWER, while a transcript that was perfect got 51.5% DER and 0.0% cpWER.

So we track the decision. Did we pick the right human? How many times did the identity change during the call? How many turns did it take to settle? How often did we say "unknown" instead of guessing? Did the agent's next action, and the summary, use the right person's words? For the frame-level model we track two numbers against each other: how much of the caller's own speech it would wrongly mute, and how much background speech it catches. Improving one by giving up the other is not progress.

We also review every call where the decision changed, and check whether the change was justified as well as whether the final answer was right. A switch caused by a two-second glitch is a bug even if the system ends up on the right person.

The results: the main speaker stopped flipping with every diarization prediction. Short replies stayed with the active profile instead of being absorbed by whoever spoke before them. A two-second acoustic glitch no longer moved the conversation to a different person. The agent asked its next question to the right human more often, and the call summaries became easier to trust. In shadow, the retrained frame-level model would have muted much less of the real caller than the first one.

None of this is free. The LLM call only runs when the acoustics are ambiguous, but when it does run, it sits on the critical path. We cache, batch, and keep the prompt small, because the agent still has to reply before the caller gives up. The fine-tuned encoder is several times heavier than the one it replaced, so it only runs when the caller enrols and a few times after that, in a background thread.

There is a privacy cost too, since we are sending transcript text to a model. Redaction, an approved provider, and logging only what we need to debug are requirements for us.

Some edge cases are still hard. The text check can be fooled by quoted speech, sarcasm, or a second person saying "yes" in the background. Overlapping speech is difficult, and a background voice that sounds like the caller can slip through. Cold starts are the weakest moment: until the caller has enrolled, nothing is protected, and that is when the agent is most likely to make a bad first impression. A fast handoff, where the new person starts answering right away, can take a couple of turns to confirm. The approach also assumes a small number of humans on the call, which fits most phone calls but not multi-party meetings.

The failures look different now, though. "The agent talked to the wrong person for the whole call" became "the agent took two turns to notice a transfer," and we will take that trade.

One limit on everything above: there is no public benchmark for main-speaker selection. Public benchmarks such as Benchmarking Diarization Models measure diarization, which is the input to this decision, not the decision itself. Our evaluation is internal, and in this article it is qualitative.

Five lessons for voice AI teams

First comes the question before the choice of the model. "Who spoke when" and "who am I talking to" require different systems, and the latter one is a policy built on top of the former one.
Consider similarity to be evidence and expect the evidence to be very poor for short phone clips. Cosine score above the threshold is an answer that the voices are the same; it doesn’t imply that these are the same people.
Have a single hypothesis and write its exit strategy on the first day. The majority of flapping vanishes as soon as the decision gets memory, but the lack of an exit strategy makes it just persistence.
Don't believe in loudness in order to identify the speaker. Every single individual may change their volume, and the model trained to work with the loudness feature will gladly learn the wrong message from it.
Quantify the decision and test it in shadow mode. The beautiful diarization report will easily coexist with a call in which the agent spent ten minutes talking to someone else, and you can identify the truth only with the real calls.