Can you re-derive a benchmark's gold labels from its raw annotator votes? Two reproducible censuses on SNLI and MultiNLI

Automated excerpt

So the census, per split, counts: labeled vs - rows; whether every labeled row's gold equals the strict majority of its votes (and whether any labeled row lacks a majority at all); whether any - row actually does have a majority (a filter leak); and the agreement profile. Across all 570,152 raw SNLI 1.0 rows read straight from snli_1.0.zip: Every labeled row's gold_label equals the strict majority of its annotator_labels — 0 mismatches, including the 24 dev/test rows decided by a non-unanimous 4-of-5 vote.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.