Speaker diarization assigns speaker labels to intervals in an audio recording: for example, Speaker 1 speaks from 0:02 to 0:14 and Speaker 2 replies until 0:21. Labels describe who spoke when within the agreed recording scope; they do not, by themselves, name a person. Overlapping speakers may need parallel intervals.
This guide explains anonymous turn labels, speaker identification, the open-source Whisper transcription workflow, and diarization error rate. It also provides an annotation brief and an ungated evaluation template so a buyer can define the reference, scoring protocol and acceptance decision before commissioning work.
What speaker diarization is (and is not)
Diarization answers who spoke when by assigning anonymous labels to time intervals. It does not require enrolled identities, and diarization alone does not name a person. The related but separate task of matching a voice to a known identity is speaker identification, covered in its own section below and in our speaker recognition dataset guide.
The tasks can be combined. A meeting product may diarize first, then map some anonymous clusters to enrolled speakers and leave other speakers unnamed. The buyer should therefore specify whether the required output is anonymous speaker attribution, role labels such as agent and caller, or identity fields supported by a separate rights and verification process.
Diarization versus speaker identification
Diarization and speaker identification solve different problems, and buyers who conflate them end up scoping the wrong project. Diarization needs no enrolled identities: it assigns anonymous speaker labels, such as Speaker 1 and Speaker 2, to speech intervals within a recording. Nobody has to have provided a reference sample in advance, and the output never claims to know who Speaker 1 actually is.
Enrolled speaker identification matches an incoming voice against known reference speakers. Speaker verification checks a single claimed identity rather than searching a roster. These are additional tasks beyond anonymous within-recording attribution; a reference representation and decision rule need their own evaluation. The pyannote metrics identification documentation distinguishes scoring with prior speaker models from anonymous label mapping. Our speaker recognition data guide covers the corresponding data brief.
Anonymous labels do not establish that the underlying audio, transcript or representations are anonymous. The European Data Protection Board distinguishes pseudonymisation from anonymisation: reduced linkability does not mean the link to a person has been removed. Assess source permissions, applicable privacy requirements, access, security, retention and deletion for the actual project and jurisdictions. Identity matching may require an additional biometric-use assessment. Neither a Speaker 1 label nor this technical guide establishes a legal exemption or a universal consent rule.
State which output the product needs: anonymous speaker turns, supported roles, identification, verification, or a combination. Ask what reference material, rights, evaluation and delivery fields each task requires; do not commission identity matching solely because the product needs readable speaker turns.
How do you correct speaker labels in a transcript?
Check the audio, segment boundaries and speaker assignment together. A boundary crossing can pull in a neighboring speaker; an identity swap can persist even when timing is correct. Split or merge turns where the agreed schema requires it, correct the anonymous ID, and review overlap against the chosen representation. Keep uncertainty visible when the evidence does not support attribution. Do not assign a known wrong neighboring speaker merely because an acknowledgement is short.
Record corrections by error type, including boundary changes, label swaps, missed overlap and extra speakers. Choose review depth using representative audio, product-critical slices, observed uncertainty and the agreed acceptance/rework rules; a low aggregate correction rate alone does not establish that spot-checking is sufficient. A managed pass through audio and speech annotation can assess an agreed correction guideline, reviewer requirements and QA plan. Use the same conventions for corrected reference labels and the evaluation template below.
How diarization systems work
A modular pipeline can detect speech, segment turns, compute speaker representations and cluster them. Joint models also exist: Fujita et al. (2019) describe end-to-end multilabel diarization that represents overlapping speech directly. That historical example establishes an architectural option, not expected accuracy on a buyer's recordings. Evaluate the complete chosen pipeline on target channels, speaker counts, languages, noise and conversational behavior.
Speaker labels in transcripts
Diarization output usually appears as a speaker label and start and end time on each turn. The annotation brief should define:
- Stable identifiers. The same person keeps the same anonymous label within the agreed scope. Cross-recording identity is a separate requirement.
- Boundary and timestamp rules. State the units, precision, minimum turn duration, silence rule, and treatment of clipped or uncertain boundaries.
- Overlap. Decide whether simultaneous speakers receive parallel spans, an overlap flag, a dominant-speaker label, or another documented representation.
- Backchannels and short turns. State whether brief acknowledgements are their own turns and how they are reviewed.
- Unknowns. Preserve an unknown or uncertain state rather than forcing a guess.
- Role or identity fields. Include them only when the project, source records, and applicable rights support them.
These conventions affect transcript usability, training labels, and the reference used to score a system. Our guide to audio annotation covers the wider labeling workflow.
What to put in a diarization annotation brief
| Decision | What to specify |
|---|---|
| Input population | Languages, speaker-count range, mixed or separate channels, codecs, devices, environments, and expected overlap. |
| Label schema | Anonymous IDs, turns, overlap, backchannels, unknown speakers, timestamps, roles, and any cross-session rule. |
| Reference process | Annotator and reviewer qualifications, independent review, adjudication, guideline version, and disagreement record. |
| Output | JSON, JSONL, RTTM, CSV, TextGrid, or another schema, plus manifests, checksums, metadata, and version history. |
| Acceptance | Representative held-out audio, scoring implementation, slices, threshold, evidence, failure review, and rework trigger. |
| Rights and controls | Source, permission, permitted use, privacy, security, access, retention, transfer, and deletion requirements. |
The managed audio annotation service can assess these requirements for buyer-provided or newly collected audio. Supported labels, reviewers, QA, security, formats, schedule, and price are confirmed in the written scope.
Does OpenAI Whisper perform speaker diarization?
The open-source Whisper transcription implementation does not provide built-in speaker clustering or speaker labels. OpenAI's Whisper model card describes speech recognition and translation; its transcribe implementation includes optional word timestamps. Timestamps alone are not attribution. This description concerns the released open-source transcription workflow, not every adapted model, combined product or OpenAI audio API.
One integration option runs a separate diarizer and assigns transcript words or segments using their timing relative to speaker turns. Define how the merge treats boundary words, concurrent speakers and uncertain attribution. Other integrated architectures or output conventions need their own checks; separate processing is not a requirement for every speech system.
Inspect the final speaker-labeled transcript around speaker changes and overlap. Timing disagreements or a merge policy that selects only one active speaker can produce an attribution error even when a component appears acceptable in isolation. For a concrete output choice, the pyannote Community-1 model card documents both regular and additional exclusive diarization, the latter intended to simplify transcription-time reconciliation. Choosing that representation does not establish overlap completeness or final transcript accuracy.
Evaluate the merged transcript under a stated word- or segment-attribution measure, alongside transcription quality and component diarization results where they help diagnose errors. Include boundary words, short turns and overlap in review. For telephone audio, test the actual codec, sample rate, resampling and channel handling of the chosen pipeline instead of assuming a fixed accuracy penalty. Our Whisper and 8kHz phone-audio guide is the separate channel-planning owner; the pilot must establish performance on the intended input.
Conditions to include in a diarization pilot
Our blog post on where speaker labels break covers the diagnostic checks worth running on a demo before trusting it, such as comparing the speaker count a system returns against known metadata and looking at segment-duration distributions within a cluster. The list below is the buyer side of the same problem: what to specify in a brief for each condition, so acceptance testing is built around it instead of discovered after delivery.
- Overlapping speech. Specify parallel labels, an overlap flag, a single-speaker output or another representation, and test that the implementation actually supplies it. Document information excluded by the chosen representation.
- Short backchannels. Define whether brief acknowledgements are separate turns. If attribution is uncertain, use the agreed unknown/review rule. Any duration-based exclusion must be explicit in labels and scoring, with its impact reported.
- Very short turns. Include short-turn examples and agree how they are labeled, reviewed and scored; establish any operational minimum on representative evidence.
- Similar-sounding speakers. Inspect actual label confusions on the target population and capture setup. Do not infer an error ranking from demographic categories; include justified evaluation slices with sufficient evidence.
- Channel bleed on shared microphones. Document microphone placement, room conditions, crosstalk and any source-separation step. Assess the resulting attribution and residual interference rather than promising complete recovery or declaring it impossible.
- Far-field and telephony conditions. Include the intended room microphones, compressed calls or telephone channels. Test deployment conditions rather than treating a close-microphone result as evidence for every channel.
- Unknown or changing speaker counts. State whether a count or range is supplied to the pipeline. Include recordings with joining/leaving speakers where relevant, inspect extra or merged clusters, and specify how count errors are reviewed.
Performance depends on the complete pipeline, the training and adaptation data, reference-label quality, scoring configuration, and how closely the evaluation resembles deployment audio; a clean two-person benchmark does not establish performance on a noisy multi-party meeting or a mixed call-center channel.
How to evaluate a diarization sample
Start with representative held-out audio and a human-reviewed reference created under the same written convention you expect in production. Freeze the scoring settings before results are seen, then report the headline and its missed-speech, false-alarm, and speaker-confusion components where DER is used.
Diarization error rate (DER) combines missed, extra and confused speaker activity after speaker mapping within the scored regions. Its denominator is scored reference speaker-time: simultaneous reference speakers count separately when overlap is included. The pyannote component implementation weights intervals by speaker counts; its label matcher separates missing speakers, extra speakers and wrong mapped labels. A missed speaker can occur while another is detected; false-alarm speaker activity can occur while a reference speaker is also active.
For a hypothetical one-second scored interval with no collar, two reference speakers overlap. If the system reports only one of them correctly, with no extra or confused speaker, there are two reference speaker-seconds and one missed speaker-second: DER is 1/2, or 50 percent. This is an arithmetic illustration under those assumptions, not a measured model result. Because extra speaker-time can grow beyond reference speaker-time, DER can exceed 100 percent. Report the components to make the failure understandable.
Use a protocol checklist before comparing results: reference version and labeling rules; boundary collar and its implementation; inclusion or exclusion of overlap; speaker mapping; scored regions, including exclusions; and scorer/version/configuration. The pyannote DER documentation describes mapping and overlap options. These are practical compatibility checks, not an exhaustive universal scoring standard. Save commands or configuration with results so another evaluator can reproduce the setup.
Do not rank vendors from headline DER values obtained on different references or protocols. Use the same held-out material and compatible scoring, then review the failures that matter to the intended product.
- Report results by product-critical slices such as overlap, speaker count, channel, noise, language, device, and short-turn conditions.
- Record the distribution across files, not only one aggregate that can hide severe failures.
- Inspect speaker-count errors, label swaps, boundary errors, overlap, and examples with high product impact.
- Keep speakers, sessions, source recordings, and prompts separated from training or tuning where the evaluation design requires it.
- Agree the acceptance threshold, minimum slice coverage, evidence, and rework rule in writing rather than borrowing a universal target.
Use the ungated speaker diarization evaluation and acceptance template to record the reference, scoring configuration, held-out design, slices, deliverables, rights, and decision.
Training data for diarization
Training and adaptation may use multi-speaker conversation with verified turn labels, real overlap and interruptions, varied speakers, and channels and environments that represent deployment. Read or scripted speech alone is insufficient for testing natural turn-taking and overlap; the useful mix depends on the model job and failure analysis.
Separate-channel capture may help trace speaker sources, but microphone isolation, crosstalk, routing and processing determine what each channel contains. Check whether the chosen pipeline preserves channels or downmixes them: the Community-1 input documentation, for example, specifies automatic averaging to mono for multichannel files. This is a model-specific behavior, not a rule for all systems. Our conversational speech data guide covers collection choices, and the full-duplex collection offer describes a custom scope when new recordings are needed.
For any source, verify the speaker and session coverage, channel and condition metadata, annotation convention, reference QA, provenance, permitted model use, restrictions, and evaluation separation. Our speech data licensing guide explains the corresponding diligence questions.