Data Labeling Quality Metrics: IAA, Cohen's Kappa & Fleiss' Kappa Explained
"Our annotators agree 90% of the time" sounds like a quality guarantee. It often isn't. How inter-annotator agreement, Cohen's kappa, and Fleiss' kappa really measure label quality, and why, on subjective African-language tasks, disagreement can be the most important signal in your dataset.