Fine-tuning Whisper for phone calls means training it on transcribed audio that matches the channel it fails on: 8 kHz, narrowband, compressed, echoey, two speakers on separate legs. Done right, published results show real word error rate reductions from a few dozen hours of matched call audio. Done wrong, you spend GPU time teaching the model to forget clean speech.
The standard tutorials cover the training loop on read-speech corpora like Common Voice. Almost none cover the decisions that make telephony fine-tuning work: how to prepare dual-channel call audio, how many hours you actually need, how to avoid catastrophic forgetting, and how to evaluate so the number you get predicts production. This page covers those decisions in order.
It assumes you have a Whisper-based pipeline in production or close to it, and that its transcripts on real calls are bad enough to justify engineering time.
Why phone audio breaks Whisper
Whisper's front end converts every input into a 16 kHz log-Mel spectrogram before the encoder sees it. Feed it an 8 kHz call recording and the audio gets resampled up to 16 kHz, but resampling restores nothing: the frequency content above 4 kHz that the telephone channel discarded stays gone. The model receives input shaped like wideband speech with an empty upper half of the spectrum, a signal it saw comparatively little of during pretraining.
Real calls stack more problems on top of the structural mismatch. Telephony audio commonly arrives mu-law encoded (G.711) and must be decoded to linear PCM before any processing; interpret it as linear audio and you get noise. Then come codec compression artifacts, transcoding along the call path, line echo, and cross-talk between speakers. The combined effect is why reported real-world word error rates on phone audio run several times the clean-benchmark numbers, and why the durable fix is matched training data rather than a cleverer preprocessing chain. The full mechanics are in our guide to why Whisper struggles with 8 kHz phone audio.
Prepare the data: channels, alignment, chunks
Data preparation is where telephony fine-tunes are won or lost, and it differs from the read-speech tutorials in four specific ways.
Split dual-channel calls per leg. Most call platforms can record the agent and the caller on separate channels. Train on each leg as its own mono stream instead of diarizing a mixdown. Each leg carries one speaker, so transcript alignment is unambiguous, and you avoid feeding diarization errors into your training labels as noise. If all you have is mixdowns, fixing the recording path forward is usually cheaper than fixing the labels afterward.
Align transcripts per channel. Each leg needs its own transcript with timestamps referenced to that leg's audio, not one interleaved conversation transcript. An interleaved transcript forces you to reconstruct who said what, which is exactly the ambiguity channel separation exists to remove.
Chunk to 30-second windows on silence. Whisper trains on 30-second segments. Cut each leg into windows of up to 30 seconds at silence boundaries, using a voice activity detector or a simple energy threshold, so words are never split across a chunk edge and every chunk's transcript is complete.
Keep the audio at native bandwidth. Do not denoise, bandwidth-extend, or otherwise improve the training audio. Production traffic will arrive at 8 kHz with its codec artifacts intact, and the entire point of the exercise is to make training conditions match inference conditions. Enhancement that will not run in production does not belong in training.
How many hours of call audio you need
Less than you might fear. Published results put domain adaptation for Whisper at roughly 10 to 50 hours of matched, transcribed audio. A peer-reviewed study that fine-tuned on about 170 hours of Indian-accented English reported word error rates in the 15 to 18% range, and published low-resource experiments have moved word error rate with as little as 8 hours.
| Matched telephony hours | What published results show |
|---|---|
| About 8 | Enough to move word error rate in published low-resource experiments |
| 10 to 50 | The range published results report for domain adaptation on matched, transcribed audio |
| About 170 | A peer-reviewed fine-tune on Indian-accented English reported word error rates of 15 to 18% |
The practical recipe: start at 10 to 25 hours, measure on your held-out set, and expand only where the measurement points, more hours of the accent you still miss, not more hours in general. For the broader question of dataset sizing across speech tasks, see how much speech data you need.
Avoid catastrophic forgetting
Fine-tune Whisper exclusively on narrowband calls and it gets better at calls while quietly getting worse at everything else. If the same model also serves meetings, voicemail, or dictation, that regression will surface as a mystery bug weeks later. Three defenses, used together:
- Mix domains. Blend a slice of clean wideband speech from the original training distribution into every epoch, so the model keeps seeing what it already knows.
- Keep the learning rate low. Aggressive learning rates buy fast telephony gains at the price of overwriting pretraining. Small, patient steps preserve more of the base model.
- Evaluate on both distributions every epoch. Track a clean wideband test set alongside your telephony test set. Stop when telephony error flattens while clean error starts to climb: that crossover is your signal.
For the training loop itself, feature extraction, tokenization, the trainer, and the word error rate metric, the Hugging Face fine-tune Whisper guide is still the reference. It targets Common Voice rather than telephony, but the loop is identical. What changes is everything around the loop: the data, the mixture, and the evaluation.
Evaluation that predicts production
Build the held-out test set from real calls, never synthetic ones. Simulated narrowband audio, clean speech pushed through a codec, does not carry the echo, transcoding, and speaker behavior of a live call path, and a model can score well on it while failing in production. Hold the set out completely: no chunk of it, and no other chunk from the same calls, appears in training.
Then report word error rate by slice, not as one blended number. At minimum, break results out by SNR band and by accent. A blended figure can hide a model that is accurate on quiet, familiar-accented calls and unusable on the noisy, accented segment your support tickets actually come from, which is usually the segment that motivated the project.
Finally, run evaluation through the exact production preprocessing: the same mu-law decode, the same resampling path, the same VAD settings. An evaluation pipeline that differs from deployment measures a system you are not shipping.
When fine-tuning is the wrong first move
A fair share of complaints that Whisper is bad at phone calls are pipeline problems, and pipeline problems are free to fix. Before you collect a single training hour, check four things:
- Channel handling. Transcribing a two-speaker mixdown when separate legs are available creates overlap errors that no amount of fine-tuning removes.
- VAD tuning. An aggressive voice activity detector chops words at segment edges and manufactures deletions.
- Endpointing. Endpoints that trigger on natural pauses truncate turns mid-sentence, which looks like the model missing speech it never received.
- Decode path. Mu-law audio interpreted as linear PCM produces garbage transcripts with no other symptom.
Fine-tune when the errors that survive those fixes are systematic: domain vocabulary the model has never seen, an accent it consistently mishears, or the telephony channel itself. Random, unpatterned errors point back at the pipeline, not the model.
Where to get the training data
The bottleneck is rarely compute. It is matched, transcribed call audio with clean rights. Describe the language, codec, and channel layout on the call-center speech data offer; the team manually selects a relevant sample before pricing a fine-tuning starter set. What to verify in that sample is covered in our telephony speech data guide.
And if the honest conclusion is that you would rather not own this loop at all, that is a legitimate engineering decision too. The Spirelight STT API serves a tuned endpoint for teams that want the accuracy without running the training pipeline themselves.