When two independent model panels both say an MMLU answer key is wrong, experts found an error 1 time in 3

Automated excerpt

We joined the two benchmarks question by question (same subject, same question text, at least 3 shared options), which gave 6,127 pairs whose keys have the same text. Then we compared three flags on those questions: agreement: both panels flag the question, and they pick the same answer text. Of the 245 cleared questions HELM's panel flags, it keeps only 75 (30.6%).

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.