Affective computing studies systems that recognize, interpret or respond to human emotion. Speech is one possible input, alongside text, facial expression and physiological signals. The availability of a recording does not establish that it can be used for a new analysis purpose, or that a model can reliably infer the state a product cares about.
This guide covers the signals involved, the gap between benchmark and deployment conditions, and the data and purpose-specific checks to make before building a speech-based affective feature.
What affective computing covers
The field is broader than voice. Facial expression analysis reads action units around the eyes and mouth from video. Physiological sensing reads heart rate variability and skin conductance, methods used in research and in some wearable products. Text sentiment analysis reads the words people write. Gesture and posture systems read movement. Each modality gives a partial, noisy view of an internal state that has no external ground truth, and research systems that report strong results often combine two or more of these channels rather than relying on one.
Why speech carries the commercial pull
Support systems and voice agents may already process an audio stream, making speech a candidate input without adding a camera or another sensor. That is an infrastructure observation, not permission to infer emotion. The EDPB's voice-assistant guidance addresses explicit processing purposes and consent choices, including purposes beyond responding to a voice request.
Before adding affect analysis, assess the intended purpose, lawful basis, notices, permissions, retention and customer contracts. In the EU, AI Act Article 5(1)(f) prohibits specified AI uses to infer emotions in workplaces and education institutions, subject to the stated medical or safety exception. This is not a blanket ban on every emotion label or annotation task; assess the actual system, purpose and context.
The gap between benchmark and deployed accuracy
Published emotion-recognition numbers are usually measured on datasets built to make the task tractable: acted or elicited emotion, one speaker at a time, clean audio, a fixed and often small set of classes. Under those conditions, accuracy figures look strong. Deployed audio looks nothing like that. Real calls carry cross-talk, background noise, compressed and narrowband telephony audio, accented speech the training set underweighted, and emotion that is muted or blended rather than performed.
Every one of those conditions degrades a model trained on curated benchmark data, and the drop is not a rounding error, it is often the difference between a feature that is usable and one that is not. Treat any accuracy number quoted without the evaluation conditions attached as a benchmark number, not a deployment promise.
Cross-cultural generalization is weak
This is worth stating plainly rather than hedging: emotion recognition models generalize poorly across cultures and languages, and the field has not solved this. The acoustic and prosodic cues that read as frustration in one language do not map cleanly onto another, annotators from different cultural backgrounds label the same clip differently, and a model trained predominantly on one population's speech will misread another population's baseline tone as something it is not. If a deployment spans multiple markets, a model validated in one language and rolled out to the rest is not validated, it is untested.
What data you need before an affective feature is viable
Before committing to an affective feature, a team needs four things in hand. Natural or spontaneous audio from the actual domain the product will run in, not only acted or elicited material, because that is what deployment conditions look like. Enough labeled volume per class the product cares about, including the muted and blended cases, not just the clean prototypical ones.
Multiple annotators per clip with agreement reported per class, because a single-rater label on a graded signal like emotion tells you almost nothing about how trustworthy it is. And coverage that matches the languages and cultures the product will actually see in production, collected and labeled by people from those populations rather than translated or extrapolated from one. Skipping any of the four does not make the feature cheaper to build, it moves the cost from data collection to production incidents.