What is the difference between TTS and ASR training data?

They have nearly opposite goals. ASR data succeeds by covering variation (many speakers, accents, and noisy conditions) so the recognizer generalizes. TTS data succeeds by removing variation: one consistent voice recorded cleanly in a steady style, because the synthesizer will reproduce any imperfection in the source audio.

How many hours of audio do I need to train a TTS voice?

It depends on the approach. A strong single-speaker neural voice typically needs from a few hours to a few tens of hours of clean, consistent recordings. Zero-shot voice cloning needs far less per target voice but relies on a large multi-speaker base model trained beforehand.

Can I use audiobooks or podcasts as TTS training data?

For research and multi-speaker pretraining, found audio can help. For a production voice it is risky: inconsistent acoustics, compression artifacts, variable mic distance, and licensing you often cannot clear for synthesis. The cleanup cost frequently exceeds recording cleanly from the start.

Why does transcript accuracy matter so much for TTS?

TTS learns a precise mapping from text to sound, so the transcript must match the audio character for character, including punctuation and normalization of numbers and symbols. Mismatches corrupt the alignment most training depends on and teach the model the wrong pronunciations or timing.

What audio quality and sample rate should TTS data use?

Aim for clean, dry recordings with minimal reverb, no background noise, and consistent loudness. Sample rate should match your target model: 22.05 kHz is a common baseline, while 24 kHz and 48 kHz are typical for modern neural vocoders that produce higher-fidelity output.

TTS Training Data: Build Natural Text-to-Speech

A text-to-speech voice is only as natural as the recordings it was fitted on, and the requirements for tts training data are almost the opposite of what you would collect for speech recognition. Where a recognizer wants mess (accents, noise, overlapping speakers, every condition production might throw at it), a synthesizer wants the studio: one consistent voice, recorded cleanly, read in a steady style, with transcripts that match the audio character for character. The model is learning to reproduce a sound, so any inconsistency in the source becomes an artifact in the output.

This guide covers what separates usable tts training data from a generic pile of audio: studio-grade versus found recordings, single-speaker versus multi-speaker design, phonetic and prosodic coverage, recording consistency, and the transcript precision that decides whether the model learns the right mapping. It also touches on voice cloning and expressive TTS, where the rules bend.

How tts training data differs from ASR data

It helps to be blunt about the split, because teams who have built a recognizer often carry the wrong instincts into their first voice. An ASR corpus succeeds by covering variation: more speakers, more accents, more noise, more of the long tail. A TTS corpus succeeds by removing variation. You are not teaching the model to generalize across a thousand voices, you are teaching it to faithfully reproduce one, or a controlled handful, so consistency is the whole game. A new microphone halfway through a session, a cold the speaker caught on day three, a room that sounds different in the afternoon: each of these is signal the model will dutifully learn and then emit at random.

The other inversion is what you optimize the audio for. Recognition tolerates lossy, real-world capture because that is what it must decode in production. Synthesis is generative, so every imperfection in the training audio is a candidate output. Background hum, mouth clicks, breaths in inconsistent places, a slight room reverb: a recognizer learns to ignore them, a synthesizer learns to produce them. If you want the broader framing of how recorded voice becomes a training asset, our explainer on what speech data is sets the foundations, and the companion piece on ASR training data shows the mirror-image requirements.

Studio-grade versus found audio

Found audio (podcasts, audiobooks, scraped video) is tempting because it is abundant and already paired with rough text. For some research and for large multi-speaker pretraining it genuinely helps. But for a production voice you intend to ship, found audio carries problems that are expensive to discover late: inconsistent room acoustics, compression artifacts from whatever codec the source used, variable mic distance, music beds, and licensing you often cannot actually clear for synthesis.

Studio-grade capture solves these at the source. A quiet, treated room, a fixed microphone and gain, a consistent distance, and a speaker who can hold the same energy across long sessions give you audio where the only thing varying is the speech itself. In our experience the cleanup cost of found audio (denoising, re-leveling, trimming, re-aligning transcripts) often exceeds the cost of recording properly the first time, and it never fully removes the artifacts. Aim for clean, dry recordings: minimal reverb, no background noise, consistent loudness, and a sample rate that matches your target model, commonly 22.05 kHz or higher, with 24 kHz and 48 kHz typical for modern neural vocoders.

Single-speaker versus multi-speaker design

The first design choice is how many voices the dataset holds, and it follows directly from what you are building.

Single-speaker. The classic path for one branded, high-quality voice. You get the most natural result per hour because the model concentrates entirely on one timbre and speaking style. The cost is that the speaker has to be available, consistent, and willing to record enough material, often several to tens of hours of clean speech for a strong neural voice.
Multi-speaker. Many voices in one corpus, labelled by speaker, let a single model produce several voices and, with the right architecture, generalize to new ones. This is the foundation under modern voice cloning and zero-shot TTS. Each speaker can contribute less individually, but the collection demands tight metadata and consistent recording standards across everyone, which is harder to enforce at scale.

A practical middle path is a large multi-speaker base for general prosody and pronunciation, then a smaller, pristine single-speaker set to fine-tune the specific voice you ship. The base teaches the model how speech behaves, the fine-tune set teaches it who to sound like.

Phonetic and prosodic coverage

Even one voice needs the right material, not just a lot of it. Phonetic coverage means the script exercises the full sound inventory of the language, including rare phonemes and the awkward transitions between them. If a diphone or a consonant cluster never appears in training, the model has to guess it at inference, and guesses are where synthesis sounds wrong. Good TTS scripts are deliberately balanced for this, which is why phonetically rich sentence sets exist rather than just reading a novel front to back.

Prosody is the harder half. A flat read of balanced sentences gives clean phonemes but a lifeless voice. The data has to contain questions, statements, lists, emphasis, and the natural rise and fall of real reading, so the model learns intonation, rhythm, and stress rather than a monotone average. Coverage also means the things people actually type at a TTS system: numbers, dates, currencies, abbreviations, acronyms, and named entities. How those are spoken, and how they appear in the transcript, decides whether your voice reads 2025 as a year or four digits, and whether Dr. becomes doctor or drive. Decisions like these belong in a written normalization spec, not in each annotator's head.

Recording consistency and style

Consistency is the quiet requirement that makes or breaks a single-speaker voice. The same speaker should keep the same distance from the mic, the same energy, the same speaking rate, and the same emotional register across every session, because the model treats the average of all of it as the voice's identity. Drift in any of these shows up as instability in the output, a voice that subtly changes character between sentences.

Style has to be chosen and held on purpose. A neutral narrator, a warm assistant, a brisk newsreader: each is a different target, and mixing them in one single-speaker set muddies the result. Breaths and pauses deserve a policy too. Consistent, naturally placed breaths can make a voice sound human, while random or clipped ones make it sound spliced. This is exactly the kind of controlled collection a custom data collection is built to enforce, with the same speaker, room, and direction held steady across the whole corpus.

Transcript precision for tts training data

For TTS the transcript is not a loose label, it is the exact text the audio must align to, so precision matters more than it does for recognition. The model learns a mapping from characters, or phonemes, to sound, and any mismatch teaches it the wrong thing. If the speaker said a word the script did not contain, or skipped one it did, or the punctuation implies a pause the audio does not have, the alignment that most TTS training depends on degrades.

That means transcripts should reflect what was actually said, with accurate punctuation that mirrors the real phrasing, consistent normalization of numbers and symbols, and clean alignment between text and audio segments. Many pipelines also add a phonetic layer, mapping text to pronunciations through a lexicon or grapheme-to-phoneme model, which is what lets the voice handle homographs and unusual names correctly. The annotation and verification work behind this is its own discipline; our guide to audio annotation covers the layers, from segmentation to pronunciation checks, that keep a TTS corpus honest.

Voice cloning and expressive TTS

Two directions bend the rules above. Voice cloning, especially zero-shot cloning from a short reference clip, leans on a large multi-speaker base so the model has already learned the space of voices and only needs a few seconds to place a new one inside it. The reference still has to be clean (cloning faithfully reproduces whatever flaws the sample contains), and the consent and licensing around someone's voice are not optional. Cloning a voice you do not have clear rights to use is a legal and ethical problem, not just a data one.

Expressive and emotional TTS deliberately reintroduces the variation a neutral voice removes, but in a labelled, controlled way. Instead of one flat style you collect the same speaker reading in distinct emotions or registers (happy, sad, urgent, calm), each tagged, so the model learns style as a controllable dimension rather than noise. The discipline is the same as before: vary one thing on purpose, label it precisely, and keep everything else constant. Found audio rarely gives you that control, which is why expressive datasets are almost always purpose-recorded with explicit direction.

When you are ready to put this into practice, you can license ready-made single-speaker and multi-speaker voices, or scope a custom recording in the exact style and language you need, in our speech datasets catalogue.

TTS Training Data: Datasets for Natural Text-to-Speech