Speech data is recorded human voice paired with the labels a model needs to learn from it: a transcript, who was speaking, and the details of how the clip was captured. If you are training an ASR system, a text-to-speech voice, or a voice assistant, this is the raw material the model is fitted on, and its shape decides what the model can and cannot do once it ships.
This guide covers what speech data actually is, the main types you will meet, what arrives in the box alongside the audio, where it comes from, and the line between training-grade material and audio someone scraped off the internet. It is written for the engineer or data lead doing the homework before the budget gets spent.
What speech data is, precisely
At the simplest level, speech data is audio of people talking. For machine learning, that audio is rarely useful on its own. What turns it into data, rather than just sound, is everything attached to it: a verbatim transcript aligned to the audio, who the speaker was, what language and dialect they spoke, where and how the clip was recorded, and what device captured it. A WAV file with no transcript and no labels is a recording. The same file with a checked transcript, speaker demographics, and recording conditions is a training example.
Format matters more than most people expect. Training pipelines generally want lossless or lightly compressed audio (WAV or FLAC), a known sample rate (16 kHz is typical for ASR, 24 kHz or higher for TTS), and consistent channel counts. Audio that has been through heavy lossy compression, resampled twice, or normalized in ways you cannot undo will quietly cap the quality of whatever you train on it. Good speech data preserves the original signal and records every transformation done to it.
The main types of speech data
When teams ask about the types of speech data, they are really asking how the speech was elicited, because that one choice changes how the audio sounds and what a model learns from it. Four categories cover most of what gets licensed or commissioned.
Read or scripted speech
The speaker reads prepared text aloud: sentences, paragraphs, or phonetically balanced prompts. It is clean, predictable, and easy to transcribe because you already know the words. Read speech is the backbone of most TTS corpora and of ASR systems that mainly handle careful, deliberate speech. The catch is that nobody talks the way they read, so a model trained only on scripted audio tends to fall apart on real conversation.
Prompted speech
The speaker responds to a prompt without a fixed script: answering a question, naming an object, giving a command, or reading a digit string in their own rhythm. This is how you collect wake words, voice commands, and short utterances at scale while keeping enough natural variation to be worth having. It sits between scripted and fully spontaneous.
Spontaneous and conversational speech
Here people talk freely, alone or with each other. You get disfluencies, false starts, overlapping turns, filler words, and real prosody. This is the hardest speech to collect and transcribe, and it is exactly what a voice assistant or a meeting transcriber meets in production. Conversational speech, recorded between two or more real speakers, is what teaches a model to handle interruptions and the messy middle of a sentence.
Far-field and in-the-wild speech
The microphone is not held to the mouth. Picture a smart speaker across the room, a car cabin at speed, or a phone face-down on a table. Distance, room acoustics, background noise, and reverberation all chew up the signal. Far-field speech is collected deliberately, with controlled noise and varied mic placement, so models built for the living room or the dashboard see something close to where they will actually run.
Most serious projects blend several of these: read speech for coverage, prompted speech for commands, conversational and far-field audio so the system survives contact with real users. Choosing the mix is one of the first real decisions in a collection, and it deserves more thought than the raw hour count. Our note on how much speech data you need goes deeper on sizing a mix instead of chasing a number.
What ships alongside the audio
The value of a speech dataset lives as much in its labels as in its waveforms. The transcript is the headline. A training-grade transcript is verbatim where it needs to be, time-aligned to the audio (often at the segment or word level), and consistent in how it handles numbers, punctuation, casing, and non-speech events like coughs or laughter. The conventions used to produce it should be written down, because a transcript is only as trustworthy as the rulebook behind it.
Beyond the transcript, expect speaker metadata: a stable speaker ID, plus age band, gender, accent or region, and native-language status where consent allows. That is what lets you balance a corpus and check whether your model performs evenly across groups instead of quietly failing for one accent. Recording metadata covers the device, the environment, the sample rate, and the noise conditions. Many corpora also carry annotation layers on top of the transcript: speaker turns for diarization, emotion or intent tags, sound-event labels, or phonetic alignment. If those layers matter to you, our overview of audio annotation walks through how they are produced and checked.
One label is easy to overlook and expensive to get wrong: consent and licensing provenance. For every speaker, a usable dataset can show that the person agreed to have their voice recorded and used for the purpose you intend, including AI training. Without that paper trail, the audio is a liability no matter how clean it sounds.
Where speech data comes from
Broadly, there are three sources. Public and open corpora (academic releases and community projects like Mozilla Common Voice) are a fine starting point for prototyping and benchmarks. They are cheap and quick, but everyone trains on the same material, the licenses vary, and coverage of specific languages, dialects, or recording conditions is patchy.
The second source is commissioned collection: a vendor recruits speakers to a defined spec and records exactly the speech you asked for. This is how you get a balanced set of, say, Norwegian speakers across age bands in car-cabin noise, or 200 distinct voices reading domain-specific prompts. It costs more and takes longer, but you own the spec and the result fits the model you are actually building. Spirelight runs this kind of collection through a global crowd of more than 10,000 contributors across 50-plus languages and dialects, with particular depth in Nordic and European languages and in hard-to-source speakers and noisy conditions. If a custom collection is where you are heading, our data services page lays out how a project gets scoped.
The third source is scraped audio: voice pulled from videos, podcasts, or calls without the speakers agreeing to it. It is tempting because it is abundant, and it is the riskiest of the three. You usually have no consent, no clean metadata, no control over quality, and no license you could defend. That is the line worth drawing carefully.
Training-grade speech data versus scraped audio
Training-grade is not a marketing word. It points at a set of properties that scraped audio almost never has: consent and a documented license for AI training; verbatim, convention-driven transcripts rather than auto-captions nobody reviewed; real speaker metadata, so you can balance and audit your corpus; controlled, documented recording conditions instead of whatever a random video happened to contain; and a quality-control pass where a second person checks transcripts, flags clipping and background contamination, and confirms the audio matches its labels.
Scraped audio can pass a quick listen and still cost you later, through transcription errors baked into the model, demographic gaps you cannot see, or legal exposure that surfaces the moment your product ships. The economics are deceptive. Cleaning and relabeling a scraped pile to a usable standard often costs more than commissioning the right speech data in the first place, and you still cannot fix the consent problem after the fact. For the wider buying picture, see our guide on buying AI training data.
The practical move is to define the speech you need before you shop for it. Specify the language and dialect coverage, the speaker distribution, the recording conditions, the transcript convention, and the license terms, and treat that as the spec a dataset has to meet. Generic speech data and the right speech data for your model are not the same purchase.
When you want to see what a structured, consent-cleared collection looks like in practice, browse the ready-made speech datasets and use them as the reference point for what training-grade should mean.