A model that hits a strong word error rate on a clean scripted test set and then falls apart the day it meets real customers is one of the most predictable failures in speech AI, and it's still one of the most common. The cause is almost always the same: the team trained and validated on scripted or read speech and shipped into a product where people talk the way people actually talk. Those are not the same acoustic or linguistic distribution, and no amount of held-out accuracy on the wrong kind of speech tells you how the model handles the right kind.
Our guide to custom speech data collection covers how to plan a collection effort end to end. This is the decision underneath it that gets skipped: scripted vs spontaneous speech data isn't a stylistic preference, it's a determination your target audio should make for you, and getting it wrong is expensive to discover late.
Why the two aren't interchangeable
Scripted or read speech comes from a fixed text: a speaker articulates complete, grammatically well-formed sentences, usually in a controlled recording setting, at a pace and prosody shaped by the fact that they're reading rather than composing on the fly. That's exactly what LibriSpeech, most TTS training corpora, and most internal "read this script" recording sessions produce. It's clean, it's predictable, and it aligns tightly with whatever language model you pair it with, because the vocabulary and sentence structure came from written text in the first place.
Spontaneous speech is produced live, and it comes with everything that entails: false starts, filled pauses, self-corrections mid-sentence, overlapping turns, variable pacing, and sentences that trail off or restart because the speaker changed their mind halfway through. None of that appears in a script, because a script has already resolved all those decisions before anyone opens their mouth.
The gap is measurable, and it's not small
You don't have to take this on faith, the ASR literature documents the gap directly. On Switchboard, telephone conversations between strangers in a relatively formal register, modern systems reach around 5.5 percent word error rate. On CallHome, calls between friends and family in a casual, spontaneous register, the same class of system lands closer to 10.3 percent, roughly double, a gap Microsoft's own research team highlighted when describing their conversational speech recognition milestone, and documented in detail in "English Conversational Telephone Speech Recognition by Humans and Machines". That's the same underlying model architecture and roughly the same acoustic channel, telephone audio in both cases. The difference is register: formal versus casual, more scripted-adjacent versus more spontaneous. If formality alone produces close to a doubling in error rate, training purely on read speech and deploying into open conversation is a larger gap still.
This is why spontaneous corpora exist as their own category
This isn't a new problem the field just noticed. The Linguistic Data Consortium built the Fisher English corpus, close to 2,000 hours of unscripted telephone conversation between strangers, specifically because DARPA's EARS program needed conversational data that scripted corpora couldn't provide. Nobody handed participants a script and asked them to read it. Callers were given a topic and left to talk, false starts, interruptions, and all, because that was the only way to get the acoustic and linguistic phenomena a scripted recording session structurally can't produce. If the field needed a dedicated, purpose-built collection effort to get spontaneous data at scale two decades ago, it's not a corner your own project can round off by reusing whatever scripted set was easiest to gather.
Which one your model actually needs, without hedging
Here's the actual test, not a rule of thumb: pull ten minutes of audio from your real target use case, not an internal script, not a demo recording, and listen for disfluencies, false starts, backchannels, and interruptions. If your product is a controlled interaction, dictation, a voice command with fixed phrasing, an IVR menu selection, voice search queries, people tend to speak in short, deliberate utterances close to scripted style, and a model trained mainly on read speech will likely hold up. If your product is open-ended, a customer service call, a meeting transcription tool, an assistant listening to natural conversation, disfluencies aren't an edge case, they're most of what you'll hear, and a model trained mainly on read speech will degrade visibly and predictably, in exactly the direction the Switchboard-to-CallHome gap points.
That ten-minute listen is the decision. Not the paper that says conversational data helped someone else's product, not a default because scripted data is easier to collect. What your users actually sound like.
What training on the wrong register actually breaks
The failure mode isn't just a worse number on a scorecard. A model trained mostly on grammatically complete, read sentences carries a language model prior that expects sentences to resolve cleanly. When it meets a genuine false start or a filled pause, it often doesn't transcribe the disfluency, it smooths over it, substituting a plausible completion for what was actually a stumble or a restart. The transcript reads fine. It just isn't what was said.
That's cosmetic in a casual context and serious in a compliance one. If you're relying on a transcript to verify that an agent delivered a required disclosure word for word, a model that quietly cleans up hesitations and false starts can hand you a transcript that looks compliant regardless of what was actually said on the call. The literal words matter more than the readability of the transcript in exactly the situations where this failure is most likely to occur.
Most real products need both, in a ratio your audio should set
Very few production systems are purely one register. Scripted-style data is still useful for anchoring vocabulary, product names, and domain jargon cleanly, so the model has seen those terms articulated without disfluency noise. Spontaneous data teaches the model to handle hesitation, overlap, and natural pacing without hallucinating a cleaner version of what happened. The ratio between the two isn't something to borrow from a paper trained on a different product with a different user base. It's something your own ten-minute listening test should set, and something worth revisiting if your product's usage pattern shifts, say from short commands toward more open conversation, since the register your users default to can change as the product itself changes.
If that listening test tells you your current training data leans scripted and your product needs spontaneous, that's a collection problem worth scoping properly rather than patching with augmentation. Browse Spirelight's conversational and spontaneous speech datasets, or talk to us about commissioning collection matched to your actual product.