TTS training data pairs speech recordings with the text spoken in them. A custom single voice benefits from consistent capture, while multi-speaker or expressive systems deliberately include variation. The training recipe determines the text processing, audio preparation and any required speaker or style information.
This guide covers recording quality, single-speaker and multi-speaker design, phonetic and prosodic coverage, and transcript conventions. It also explains why a ready-made conversational corpus and recordings commissioned for a specific studio voice serve different requirements.
How TTS training data differs from ASR data
ASR learns to map sound to text; TTS learns to generate sound from text. For a custom single voice, consistent recording conditions help keep that voice stable. A multi-speaker or expressive system deliberately includes variation, with speaker and style information so the model can distinguish it from capture artifacts.
NVIDIA NeMo's TTS data documentation describes audio-text manifests and optional speaker information. Start from the target model's required fields and preprocessing recipe, then choose the speakers and recording protocol. A conversational ASR corpus should not be presented as a ready-made studio voice simply because it contains transcripts.
Studio-grade versus found audio
Existing sources such as podcasts, audiobooks or video may contain useful speech, but capture conditions, transcript quality and permissions vary. Inspect representative original files for room acoustics, compression, microphone distance, background sound and unwanted variation. Ask the supplier to document the source, applicable permissions and restrictions for the intended synthesis use; record unresolved questions for project-specific review.
Purpose-recorded collection can control capture conditions specified in a brief. Confirm the proposed room, microphone, gain, speaker direction and checks rather than assuming a studio or voice is available. Compare the preparation needed for an existing source with a scoped recording project. Choose sample rate, channel layout, preprocessing and acceptable noise or reverberation against the target model and vocoder recipe. There is no universal sample-rate floor for every TTS system, and resampling does not restore detail absent from the original recording.
Single-speaker versus multi-speaker design
The first design choice is how many voices the dataset holds, and it follows directly from what you are building.
- Single-speaker. One target voice with capture and style rules chosen for the intended model. Confirm the proposed speaker, permitted uses, session consistency and recording volume through a representative first batch. A generic hour count does not establish what a particular training or fine-tuning recipe needs.
- Multi-speaker. Many voices in one corpus, labelled by speaker, let a single model produce several voices and, with the right architecture, generalize to new ones. This is the foundation under modern voice cloning and zero-shot TTS. Each speaker can contribute less individually, but the collection demands tight metadata and consistent recording standards across everyone, which is harder to enforce at scale.
A practical middle path is a large multi-speaker base for general prosody and pronunciation, then a smaller, pristine single-speaker set to fine-tune the specific voice you ship. The base teaches the model how speech behaves, the fine-tune set teaches it who to sound like.
Plan the first TTS recording batch
Before commissioning more hours, turn the model requirements into a brief that a supplier can assess. Separate requested characteristics from evidence already checked. An unknown speaker, recording location or permission should remain an open question until it is confirmed in the proposal.
| Decision | Put in the brief | Check on the first batch |
|---|---|---|
| Model and voice design | Training or fine-tuning recipe, intended synthesis use, voices, languages, accents and speaking styles. | The proposed speaker and style labels match the agreed design; feasibility is confirmed. |
| Capture and text | Required audio format, recording conditions, script coverage and transcript-normalization policy. | Inspect original capture, listen to complete utterances and compare the spoken content with the agreed text representation. |
| Delivery and splits | File naming, durations, speaker/style identifiers, manifest fields and training/validation separation. | Load a small manifest with the intended recipe; check referenced files, units, required fields and the stated split rules. |
| Acceptance and rights | Review method, correction rules, intended uses, source and speaker permissions, security and required records. | Confirm the applicable evidence and unresolved questions before approving more recording. |
Use the TTS recording scope and acceptance worksheet to prepare those decisions. It is an original blank planning aid, not a ready-made corpus, recording sample, contract or statement of available voices. A supplier proposal still needs to confirm feasibility, rights, deliverables, schedule and price.
For a text-policy discussion, consider this invented English example: Dr. Lee paid $12.50 on May 4.
One explicitly chosen reading could be Doctor Lee paid twelve dollars and fifty cents on May fourth.
Record that choice in the guideline instead of asking reviewers to infer it. This is written example text only, not a recording or a universal normalization rule.
The manifest follows the chosen recipe. NeMo's documentation, for example, describes an utterance per JSON line, optional normalized text, speaker identifiers and separate training and validation manifests. Those fields are an implementation example, not a mandatory schema for all TTS models. Validate the first delivery with the actual loader and review the audio; a successful file import alone does not establish recording or synthesis quality.
Phonetic and prosodic coverage
Plan scripts against the target language, model recipe and intended reading styles. Review coverage of sounds, word positions, transitions and pronunciation contexts, then use the first batch to identify gaps. A duration total or a list of sound counts does not by itself establish the resulting synthesis quality.
Include the utterance types the system needs, such as questions, statements, lists, emphasis, numbers, dates, currencies, abbreviations and named entities. Agree how they should be read and represented in text. Keep intended prosodic and style variation distinct from unwanted capture artifacts, and evaluate the choices with the chosen model rather than assuming that one script design guarantees a particular voice quality.
Recording consistency and style
For a selected voice, write a capture protocol covering the microphone, room, distance, gain, session checks and intended speaking styles. Compare the pilot and later sessions for unwanted drift. The model and voice design determine which variations are useful; identical energy, rate and emotional delivery across every utterance are not universal requirements.
Separate capture controls from deliberate style variation. Specify whether breaths, pauses and corrections remain in the audio and text, and how styles are identified if the recipe uses them. A custom speech collection assessment can establish whether the proposed speaker, setup and direction can meet the brief. Confirm that scope and its checks in writing rather than assuming that a particular speaker, room or facility is available.
Transcript precision for TTS training data
The text representation must match the spoken content under the target recipe. That can mean graphemes, phonemes or a defined normalization step, rather than a character-for-character copy of a script. Agree how to represent numbers, abbreviations, punctuation, pauses and corrections, then check omissions and additions against the actual recording.
That means transcripts should reflect what was actually said, with accurate punctuation that mirrors the real phrasing, consistent normalization of numbers and symbols, and clean alignment between text and audio segments. Many pipelines also add a phonetic layer, mapping text to pronunciations through a lexicon or grapheme-to-phoneme model, which is what lets the voice handle homographs and unusual names correctly. The annotation and verification work behind this is its own discipline; our guide to audio annotation covers the layers, from segmentation to pronunciation checks, that keep a TTS corpus honest.
Voice cloning and expressive TTS
Voice-cloning approaches have model-specific reference and adaptation requirements. The VALL-E paper (2023) describes one approach to conditioning synthesized speech on a reference; its experimental setup is not a universal data requirement. For a chosen model, confirm the required reference length, capture conditions and preparation, then evaluate the output. For a synthetic-voice project, record the intended use and arrange project-specific review of authorization, source rights, privacy, personality or publicity rights, compensation, security, transparency and contract terms.
An expressive project may deliberately include different styles or registers. Where the model recipe uses style labels, agree their definitions, coverage and review method. The recording protocol should distinguish intentional variation from capture problems rather than require every delivery characteristic to stay constant. For an existing source, inspect style coverage, capture variation, label consistency and permissions against the target.
For a purpose-recorded voice, request assessment of a custom TTS recording brief. If the experiment instead calls for conversational speech, review the conversational speech catalogue and its actual source, rights and specifications. That catalogue is not presented here as ready-made studio TTS voices; suitability for a particular synthesis recipe requires separate review.