Automatic speech recognition (ASR) is the technology that converts spoken audio into written text. It is the engine behind voice assistants, dictation software, call transcription, in-car voice control, and the captions on your videos. You will also see it called speech-to-text (STT) or simply speech recognition; in practice these terms all point at the same task.

This guide explains what ASR means, how a modern recognizer actually works, how accuracy is measured, why systems fail on real-world audio, and what it takes to make one better. It is written for teams evaluating or building voice features, not for researchers, so it stays practical.

What ASR means

ASR takes an audio signal containing speech and produces the most likely text transcription of it. That one sentence hides real difficulty: the system has to cope with different voices, accents, speaking speeds, background noise, microphones, and vocabulary, and it has to do it fast enough to be useful. A recognizer that works beautifully on a podcast studio recording can fall apart on a speakerphone in a moving car.

ASR answers the question of what was said. It does not answer who said it; that is speaker recognition, a separate task covered in our speaker recognition dataset guide. It also does not interpret meaning or intent; that is language understanding, which usually runs on top of the ASR output.

How automatic speech recognition works

Classic ASR systems were built from three separate parts: an acoustic model that mapped audio frames to sound units, a pronunciation lexicon that mapped sound units to words, and a language model that scored which word sequences were plausible. Each part was trained and tuned separately, and each was a source of error.

Modern systems are end-to-end neural networks. A single model, usually a transformer-based encoder-decoder or a transducer, learns to map audio directly to text from large volumes of transcribed speech. The model listens to the audio, builds an internal representation of it, and generates the transcript token by token. Whisper-style models and the recognizers inside major voice assistants work this way.

The practical consequence of the end-to-end shift is that the training data now carries almost all of the weight. There is no lexicon to patch and no separate acoustic model to adapt; the recognizer knows exactly what its training data taught it. Accents it never heard, terms it never saw, and noise conditions it never trained on all show up directly as errors.

How ASR accuracy is measured

The standard metric is word error rate (WER): the number of substitutions, insertions, and deletions the system makes, divided by the number of words actually spoken. A WER of 10% means roughly one word in ten is wrong. Modern systems reach low single-digit WER on clean benchmark audio, and considerably worse on spontaneous, noisy, or accented speech. We cover the metric in detail, including where it misleads, in our article on what WER really measures.

The number that matters is not the benchmark WER a vendor quotes but the WER on your audio: your users, your acoustic conditions, your vocabulary. Always evaluate on a test set that looks like production.

Why recognizers fail in the real world

Most production ASR failures trace back to a mismatch between the training data and the deployment conditions. The common gaps:

  • Accents and dialects. Models trained mostly on standard varieties of a language degrade on regional accents and on second-language speakers. Our guide to multilingual speech data covers this in depth.
  • Noise and distance. Far-field microphones, traffic, kitchens, and overlapping talkers all push audio away from the clean recordings most public training sets contain.
  • Domain vocabulary. Product names, medical terms, street names, and industry jargon are misrecognized when they were rare or absent in training.
  • Spontaneous speech. Real conversation is full of false starts, fillers, interruptions, and half-finished sentences. Systems trained on read speech stumble on it.
  • Code-switching. Speakers who mix languages mid-sentence break recognizers trained on one language at a time.

Making ASR better: it comes down to data

If you are building or fine-tuning a recognizer, the levers are mostly data levers. Fine-tuning a strong pretrained model on a few hundred hours of well-transcribed audio from your actual domain routinely beats a generic model, and the gains are largest exactly where generic models are weakest: accents, noise, and vocabulary. What that data needs to look like, and how much of it you need, are covered in our guides to ASR training data and how much speech data you need.

Quality matters as much as quantity. Transcripts must reflect what was actually said, timestamps must align, and metadata must let you slice the data by speaker, device, and environment. Our speech data quality guide lists the checks worth running before you train on anything.

Where ASR is used

Voice assistants and smart speakers, dictation and clinical documentation, call center transcription and analytics, in-car voice control, meeting transcription, media captioning and subtitling, and voice interfaces in apps and devices. Each of these has its own acoustic reality and its own vocabulary, which is why teams shipping them end up caring about training data long after they have picked a model architecture.