A voice assistant trained on clean, one-speaker, read-aloud audio behaves beautifully in a demo, then meets two people talking over each other. It mishears the interruption, drops the backchannel, and treats a false start as a finished sentence. The gap is almost always conversational speech data: spontaneous, two-speaker recordings that contain the things scripted corpora are designed to strip out. If you are building assistants, agents, or meeting transcription, this is the data that decides whether the model survives contact with real users.

This guide covers what makes spontaneous dialogue hard, why models trained on scripted reads fall over on it, and how paired conversational audio is collected and labeled before it becomes a corpus you can train on.

What conversational speech data actually is

Conversational speech data is audio of people talking to each other rather than reading at a microphone. The distinction sounds small and it is enormous. Scripted speech is planned: the speaker knows the next word, so prosody is smooth, pauses fall at clause boundaries, and there is exactly one voice on the channel. Spontaneous speech is composed in real time, so it carries everything planning removes: false starts, mid-sentence repairs, fillers, trailing off, and the constant negotiation of who speaks next.

The version that matters most for assistants is two-speaker audio, a genuine exchange between two people, captured so that each turn, overlap, and interruption is preserved. Some teams call this a conversational AI dataset or a dialogue dataset. It sits at the opposite end of the spectrum from the single-narrator reads used to bootstrap early speech models. For the broader taxonomy of recording styles, our guide to what speech data is lays out where conversational sits among scripted, prompted, and elicited speech.

Why scripted audio breaks on real conversation

A model learns the distribution it is shown. Train it on studio reads and it gets excellent at studio reads, then goes brittle the moment the input stops looking like one. Conversation breaks scripted-trained models in a handful of specific, repeatable ways, and each one maps to a labeling decision later, so they are worth naming.

Turn-taking is the first. People hand the floor back and forth with almost no silence between turns, and they project the end of a turn before it arrives, which is why interruptions land where they do. A model that has only seen isolated utterances has no notion of a turn at all. It cannot tell whether a pause means keep listening or respond now, so it either talks over the user or sits there waiting.

Overlap is the second, and it is the one augmentation cannot fake. In real dialogue, speakers overlap constantly: agreement, interruption, finishing each other's sentences. On a single mixed channel that becomes two voices competing for the same frequencies, and a model trained on clean single-speaker audio has no representation for it. Third is disfluency: the um, uh, false starts, and self-corrections that fill spontaneous speech and that a scripted corpus has none of by construction. Fourth is the backchannel, the mm-hm, right, yeah a listener drops in without taking the floor. Treat those as turns and your assistant interrupts a user who was only signalling they are still listening.

Underneath all of it is prosody. The rise that invites a response, the flat continuation that says I am not finished, the emphasis that carries meaning: real conversational prosody is generated by the act of conversing and cannot be read off a page convincingly. That is also why scripted data is a weak base for expressive synthesis, which is part of why sizing a collection for natural dialogue is harder than for reads, something our guide on how much speech data you need gets into.

How paired conversational data is collected

Capturing real conversation well is harder than recording reads, and most of the difficulty is in the setup rather than the talking. A few choices shape everything downstream.

The first is how speakers are paired and prompted. You want genuine spontaneity, not two strangers stiffly trading rehearsed lines, so the work is in giving contributors a reason to actually talk: a scenario, a task to finish together, a topic they have an opinion on, while staying out of the way enough that the speech stays natural. Demographic pairing matters too. If your assistant serves a specific market, the ages, accents, and dialects in the conversations should reflect it. Sourcing those speaker profiles, particularly across Nordic and European languages and harder-to-find dialects, is much of what we do at Spirelight through a global contributor crowd.

The second is the channel layout, the single decision that most affects how usable the data is later:

  • Separate channels per speaker. Each voice is recorded on its own track, so overlap is preserved but the speakers stay cleanly separable for labeling and analysis. This is the most flexible format and the one most worth paying for.
  • Single mixed channel. Both speakers share one track, which matches what a far-field device or a single meeting microphone actually hears. Realistic, but harder to annotate because overlap has to be untangled by ear.
  • Far-field and device capture. Recording across a room, through the kind of microphone your product ships with, so the acoustics match deployment rather than a headset no user owns.

The third is condition. If your assistant runs in a kitchen, a car, or an open-plan office, the conversations should be captured there, with the real noise and reverberation, rather than recorded clean and degraded afterward. Added-noise augmentation has its place, but real overlapping speech in a real room carries artifacts a mixing script does not reproduce. The use cases we work on, from in-car voice to call analytics, each come with their own acoustic constraints worth specifying before a single recording is made.

How conversational audio is labeled

Raw two-speaker audio is not training data until it is segmented, attributed, and transcribed to a consistent standard. Labeling is where a conversational corpus earns or loses most of its value, and it involves a few layers that read-aloud data never needs.

Speaker diarization comes first: marking who spoke when, so every region of audio is attributed to a speaker. On overlap this is genuinely hard, and how the convention handles two voices at once (separate spans, a marked overlap region, a dominant-speaker rule) has to be decided up front and written down. Turn segmentation sits on top, defining where one turn ends and the next begins, including the cases where they do not cleanly alternate.

Then the transcript itself, with conventions scripted corpora can ignore. Are fillers and false starts kept verbatim or cleaned? Are backchannels transcribed and tagged as backchannels rather than turns? How is unintelligible crosstalk marked? Are laughter, breaths, and non-speech events labeled the same way every time? None of this is cosmetic. An assistant that needs to tell an interruption from a backchannel can only learn it if the labels drew that line consistently. Two annotators who quietly disagree on these rules inject noise that looks like model error much later. Our guide to what audio annotation involves walks through these layers, and quality-checking each pass against a documented style guide is what keeps the corpus consistent at scale.

Because the transcript is the supervision signal, its error rate sets a hard floor under the accuracy your model can reach. That holds for read speech and holds harder here, where the audio is messier and the temptation to guess at an overlapping word is constant. The same coverage and verification thinking from our ASR training data guide applies directly, with diarization and turn labels added on top.

What to look for when you buy conversational speech data

When you evaluate a conversational corpus, push past the headline hour count. Ask whether the speech is genuinely spontaneous or lightly scripted role-play, because the disfluencies and overlaps you need only show up in the real thing. Ask how speakers are separated on the channel, since per-speaker tracks are far more flexible than a single mix. Ask whether overlap and backchannels are labeled at all or quietly cleaned away, because cleaned conversation has had exactly the hard cases removed. And ask whether the language, accents, recording conditions, and turn dynamics match where your assistant will actually run.

If your deployment is narrow, a specific market, a specific device, a specific noise environment, a custom collection scoped to those conditions usually beats stretching an off-the-shelf corpus to fit, which is what our custom collection work is for. To see what is already available by language, recording style, and conversational format, browse the spontaneous and paired conversational sets in our speech datasets catalogue.