Every avatar, lip-sync, or talking-head generation model is trained on the same raw material: synchronized recordings of a human face and voice. The decision you face is where that material comes from. The public options, GRID, VoxCeleb2, MEAD, HDTF, CelebV-Text, are thoroughly documented, and nearly every paper in the field trains on them. Almost none of them can follow you into a commercial product, because their license terms, and more importantly the rights of the people on camera, were never designed for one.
This guide covers what a talking-head dataset has to contain, how the established public sets compare, where their licenses actually stop, and what a custom, consented collection changes. It ends with honest numbers on how much data different modeling approaches need.
What a talking-head dataset actually contains
A talking-head dataset is a corpus of video recordings in which one visible person speaks, with the audio track synchronized to the face. Because the model learns the mapping between speech and facial motion, both streams and the alignment between them are the product. Six specifications decide whether a given corpus can train your model.
Identity coverage comes first: a model generalizes across faces only if it was trained across faces, and a corpus of three people trains a model of three people. Head pose range is next, because frontal-only footage produces avatars that break the moment the head turns.
Resolution and frame rate set the ceiling on output quality; 1080p at 25 or 30 fps is the practical floor for generation work, and what matters is pixels on the mouth region, not the frame as a whole. Sync accuracy is the quiet killer: misalignment of more than a frame or two measurably degrades lip-sync training, broadcast and web material routinely drifts beyond that, and scraped pipelines have to filter clips through sync detectors before use.
Expression range matters because neutral read speech trains a neutral face; expressive segments have to be captured deliberately. Finally, listening segments: a dialogue avatar spends half of every conversation not speaking, and natural footage of nods, backchannels, and attentive silence is the rarest ingredient in public corpora.
The public dataset landscape
Five corpora account for most of the published work in talking-head generation and audio-visual speech. They differ less in format than in provenance, and provenance decides what you may do with them.
| Dataset | Size | Source | License restriction |
|---|---|---|---|
| GRID | 33 speakers on video, 1,000 sentences each | Studio recording, fixed six-word command sentences | CC BY 4.0, but 720x576 video and a constrained sentence grammar |
| VoxCeleb2 | 6,000+ speakers, 1M+ utterances | YouTube interviews of public figures | CC BY-SA 4.0 on the metadata only; the compilers no longer distribute the data, and rights stay with the original video owners |
| MEAD | 60 actors, 8 emotions at 3 intensity levels, 7 camera views | Studio capture by the authors | Stated terms of use: non-commercial research only, no redistribution, access by approved request |
| HDTF | About 16 hours, 300+ subjects, 720p to 1080p | YouTube URLs plus crop and timing annotations | Source videos downloaded from YouTube by you; rights remain with the uploaders |
| CelebV-Text | 70,000 in-the-wild clips with text descriptions | Web-collected face video | Non-commercial research use only |
The GRID, VoxCeleb2, and HDTF rows are as published on the GRID record on Zenodo, the VoxCeleb2 page at Oxford's Visual Geometry Group, and the HDTF repository; the MEAD and CelebV-Text terms come from the datasets' own distribution pages. The pattern to notice: the studio-recorded corpora are small, constrained, or gated behind research terms, and every corpus large enough to pretrain on was scraped from the web.
License terms are the trap, not quality
Only two of the five, GRID and MEAD, were recorded by the people who distribute them, and their terms show how far provenance alone gets you. GRID ships under CC BY 4.0, which permits commercial use of its constrained 720x576 recordings.
MEAD ships with a stated terms-of-use agreement: the actors signed formal agreements with the producers, but the terms limit use to non-commercial research, prohibit redistribution, and gate access behind approval. Even the best-consented studio corpus stops at the laboratory door. Everything large enough to pretrain on was scraped, and a scraped corpus has a structural problem no license text can fix: the compiler can only license what the compiler owns.
VoxCeleb2's Creative Commons label covers its metadata, and its compilers have withdrawn distribution of the data itself; HDTF is a list of YouTube URLs and annotations you download yourself. In both cases, copyright in the underlying footage stays with whoever uploaded it, and the people on camera never signed anything at all.
That last gap is the one that matters. Under GDPR, biometric data processed to uniquely identify a person is special-category data, and face video is the textbook example of data that supports such processing. A talking-head model goes further than identification: it learns to reproduce likenesses, which is precisely the use a participant should be able to accept or refuse. Research-only terms, withdrawn distribution, and scraped provenance therefore converge on the same practical answer: these corpora support papers, not products. The general mechanics of dataset rights are covered in our speech data licensing guide; face video simply raises the stakes, because the data identifies a person by construction.
What a custom collection changes
A commercial-grade collection is built in the opposite order: rights first, cameras second. Every participant signs a release before recording that names likeness use, biometric processing, and model training as purposes, so the consent chain covers the product you intend to build rather than a vague research abstraction. That document is the one thing no scraped or research corpus can retrofit.
Controlled capture then buys back everything the scraped sets lack. Camera, lighting, framing, and background are fixed, or varied on purpose rather than at random. Audio and video are synchronized at capture instead of repaired afterward. Pose, expression, and emotion are scripted into the session plan. Sessions run in monologue mode, one speaker to camera, and in dialogue mode, two people in natural turn-taking, which is the only way to record genuine listening behavior. Demographics are recruited to a specification, so age, gender, and language coverage match the users the avatar will meet. This is the model Spirelight's talking-head video collection follows: recorded to order, consented per participant, and delivered with capture metadata attached.
How much data you actually need
The honest answer spans three orders of magnitude, because it depends on the approach. Few-shot personalization of a pretrained generator adapts to a new identity from seconds to a few minutes of video. Person-specific avatars, the classic route to one high-quality digital human, are typically trained on several minutes to an hour of a single speaker. Generalizable pretraining is the expensive regime: published models train on corpora from roughly sixteen hours (HDTF) to thousands of hours (VoxCeleb2), spread across hundreds or thousands of identities.
The practical consequence is that few teams need to rebuild VoxCeleb2. The pattern that works is a consented corpus of tens of identities and tens of hours, used for fine-tuning, domain adaptation, and evaluation on top of whatever pretraining the team can legitimately use, plus a held-out consented test set regardless of how the training data was assembled. The step no team can skip is confirming that every identity inside the shipped model's training set agreed to be there.