Speaker diarization is the task of working out who spoke when in an audio recording. A diarization system takes a conversation and splits it into speaker turns, assigning each stretch of speech to a speaker label: Speaker 1 said this from 0:02 to 0:14, Speaker 2 replied until 0:21, and so on. Without it, a transcript of any multi-speaker recording is a wall of undifferentiated text.

This guide covers what diarization is and is not, how systems do it, how speaker labels should appear in transcripts, why diarization remains genuinely hard, and what the training data behind a good system looks like.

What speaker diarization is (and is not)

Diarization answers who spoke when, using anonymous labels. It does not know or claim who the speakers are; it only separates them. That distinguishes it from two neighboring tasks. Speaker identification matches a voice against a set of known voices to name the speaker. Speaker verification confirms whether a voice matches one claimed identity, the pattern used in voice authentication. Both are covered in our speaker recognition dataset guide. Diarization needs no enrolled voices at all, which is why it works on any recording but can only ever say Speaker 1 and Speaker 2.

In production the tasks often stack: a meeting tool diarizes first, then identifies the speakers it can match to enrolled voice profiles, and falls back to anonymous labels for guests.

How diarization systems work

A modern diarization pipeline has four stages. Voice activity detection finds the stretches that contain speech. Segmentation cuts the speech at likely speaker changes. A speaker embedding model converts each segment into a numerical voiceprint that captures the characteristics of the voice. Clustering groups segments with similar voiceprints, and each cluster becomes one speaker. Some newer systems fold these stages into a single end-to-end neural model, which handles overlapping speech better but needs more labeled conversational training data.

Accuracy is measured by diarization error rate (DER), which adds up three failure types: speech that was missed, non-speech that was marked as speech, and speech attributed to the wrong speaker. As with word error rate in ASR, vendor benchmark numbers are measured on clean test sets, and your audio will differ.

Speaker labels in transcripts

Diarization output usually lands in a transcript as speaker labels: a consistent tag on every turn, with timestamps. Conventions vary, but good practice is stable across formats:

  • Consistent identifiers. The same speaker keeps the same label for the whole recording, and across recordings if the dataset tracks recurring speakers. Labels like S1/S2 or SPEAKER_00 are standard; real names only when identity is known and consented.
  • One label per turn, with the turn's start and end time, so text can be traced back to audio.
  • Explicit handling of overlap. Simultaneous speech is marked as such rather than silently assigned to one speaker.
  • Unknowns marked honestly. A segment the annotator cannot attribute gets an unknown tag, not a guess.

These conventions matter twice: once for the readability of the transcript, and again when the transcript becomes training data, because inconsistent labels teach a model inconsistent boundaries. Our guide to audio annotation covers the wider labeling workflow.

Why diarization is still hard

Diarization performance collapses in predictable places. Overlapping speech is the biggest: real conversations are full of interruptions and simultaneous talk, and clustering-based systems must assign each moment to exactly one speaker. Short backchannels (mm-hmm, yeah, right) are often too brief to embed reliably. Similar voices, especially same-sex siblings or colleagues with the same accent, blur cluster boundaries. Far-field and telephone audio strip away the spectral detail embeddings rely on. And the number of speakers is usually unknown in advance, so systems must guess how many clusters to form.

Every one of these failure modes is a data problem before it is an algorithm problem. A system trained mostly on clean two-person studio interviews has never seen a noisy four-person meeting, and it shows.

Training data for diarization

Diarization models and the speaker embeddings under them are trained on multi-speaker conversational audio with precise, human-verified speaker turn labels. The data that moves the needle is spontaneous conversation with natural interruptions and overlap, recorded across the devices and environments the product will face, with enough distinct speakers to teach the model what voice variety looks like. Read speech does not help here; the value is in the mess. Our conversational speech data guide covers collecting exactly this, and our audio annotation service produces the turn-level speaker labels to train and evaluate against.

If you are buying rather than collecting, check the dataset for real overlap (not just clean alternating turns), speaker counts per recording, device and environment metadata, and consent that covers commercial model training. Our speech data licensing guide explains what to look for in the paperwork.