Which NLI labels are least decisive? A per-class and per-genre annotator-agreement tour of SNLI, MultiNLI and ChaosNLI

Automated excerpt

Natural-language-inference benchmarks hand you one gold label per example — entailment, neutral or contradiction. For every validated example we compute two things per gold class: the unanimous rate (all five annotators chose the gold) and the individual-agreement rate (share of the five individual labels equal to the gold — the datasets' own "agreement" statistic). Within it, the original gold flips to a different plurality under 100 votes 28.3% of the time, and 10.0% never reach a strict majority even at 100 votes.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.