Most openly available emotion corpora are performed: actors reading set sentences in a named emotion. Models trained on them learn a clean, exaggerated version of affect that real callers rarely produce, and they degrade the moment a genuinely irritated person speaks in a flat voice.
Labeling spontaneous conversation is harder and the agreement figures are lower, which is exactly why we publish them. What you get in return is affect that occurred rather than affect that was staged.