Words have a ground truth. A transcript is right or it is not, and two good annotators agree almost every time. Affect has no such anchor: one clipped sentence reads as irritation to one listener, fatigue to another, and nothing at all to a third, and none of them is wrong.
That does not make the signal useless, it makes the reporting matter. Every batch is labeled by several annotators and shipped with per-class agreement, so you can weight training on the classes that held up and treat the contested ones with the caution they deserve.