Visual Speech Recognition Datasets

Every team building a lip reading or audio-visual speech model faces the same early decision: which dataset to train on. The public options are well documented and available for research, and they are the reason the field moved as fast as it did. They are also nearly all English, nearly all frontal, and, for the strongest of them, licensed for non-commercial research only. That last clause is easy to miss and expensive to discover late: a model trained on BBC or TED footage under a research agreement cannot quietly become a product.

This guide maps the public visual speech recognition datasets, what each one is actually good for, where the license lines sit, and what to specify if you commission audio-visual capture of your own.

Why audio-visual models beat audio-only in noise

Visual speech recognition, lip reading in most paper titles, decodes what a person says from video of their face. Paired with the audio stream it becomes audio-visual speech recognition, and the pairing earns its cost exactly where audio-only models fail: overlapping speakers, street noise, machinery, moving vehicles. Articulation is unaffected by the acoustics of the room, so the visual stream still carries signal after the acoustic one has drowned.

The benchmark record is consistent. AV-HuBERT, Meta's self-supervised audio-visual model, brought lip reading on the 433 hour LRS3 benchmark down to a 26.9 percent word error rate, and its joint representations improved audio-only recognition as well. Auto-AVSR then showed that transcribing unlabeled collections such as AVSpeech and VoxCeleb2 with off-the-shelf ASR keeps scaling audio-visual accuracy with training hours. Both results rest on the same input: large volumes of faces speaking on camera, gathered under research terms. That input is the constraint this guide is about.

The public lip reading dataset landscape

Five datasets carry most of the published lip reading and audio-visual ASR work, and they divide by unit. GRID and LRW are constrained word and command sets that models saturate quickly. LRS2 and LRS3 are the sentence-level benchmarks that state-of-the-art systems report on. AVSpeech is an in-the-wild collection built for speech separation and enhancement, distributed as segment references into YouTube rather than as media. The Oxford VGG lip reading pages host the LRW and LRS families.

Two rows need footnotes. GRID is genuinely open, but its six word command grammar, sentences like "put red at G9 now", is so constrained that results on it transfer poorly to continuous speech. And AVSpeech's size is nominal: distributed as URLs, its retrievable fraction shrinks as source videos are deleted, and what is retrieved carries no license to the underlying footage.

What the public sets do not give you

Commercial rights come first. LRW and LRS2 sit behind a BBC data-sharing agreement that restricts them to non-commercial academic research. LRS3 carries no BBC agreement: built from TED and TEDx talks, it is released under the Creative Commons BY-NC-ND 4.0 license, which rules out commercial use just as firmly. Either way, the strong sentence-level sets cover a paper and a benchmark table and nothing more. AVSpeech conveys no rights at all: it is a list of pointers into other people's uploads. A production model trained on any of them carries a licensing question that engineering cannot answer. The speech data licensing guide covers how to read these terms before training rather than after.

Coverage is the second gap. The sentence-level benchmarks are English, so teams building for other languages inherit almost nothing. Broadcast and conference audio is close-miked and clean, which means the noise robustness that motivates audio-visual modeling is usually simulated by mixing noise in afterward, not the same thing as capture in a car, a kitchen, or a crowd. Presenters and interviewees face the camera, so profile views past roughly 45 degrees are scarce, and in-cabin or head-mounted geometries are absent.

The third gap is consent, and it is structural. Nobody in a BBC broadcast or a TED talk agreed to have models trained on their face. Face video sits close to biometric data under the GDPR, and for a European deployment the provenance of every face in the training set is a question you should expect to be asked. More scraping cannot patch this; only capture with documented permission can.

What to specify when commissioning an audio-visual speech dataset

Where the gap is rights, language, angle, or noise, the alternative is capture scoped for the purpose. The specification decides whether the footage trains well, and most of it costs nothing extra if agreed before recording starts:

• Sync tolerance. State the maximum audio-to-video offset and how it is measured, for example within one frame at 25 frames per second. Pipelines that assume sync instead of verifying it find the drift in their training curves.
• Mouth region. Specify the minimum pixel size of the mouth crop at delivery resolution. A 4K wide shot can still yield lips too small for the input crop of a lip reading model.
• Camera angles. Frontal only, or deliberate profile coverage at stated angles, rather than whatever head turns happen to occur.
• Lighting and noise. A controlled baseline, plus the low-light levels or real noise conditions the deployment needs, with signal-to-noise ranges stated when noise is captured live.
• Languages and speakers. Speaker counts, dialects, and demographic spread per language, not one pooled figure.
• Consent. Likeness, biometric, and model-training permission documented per participant, with the intended use stated in the release each speaker signs.

This is the model behind Spirelight's audiovisual speech data collection service: conditions and consent language are agreed before recruiting, so the rights review happens once, up front. Benchmark on LRS3, the research license exists for exactly that. Train what you ship on footage whose speakers agreed to the use.