AI article
Our best shortcut for finding wrong MMLU answer keys failed its replication
Community description: This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below...
Dev.to | Nov 6, 2026 | MV2 (Munim, Inc.)
Automated excerpt
On MMLU, using Stanford CRFM's HELM outputs from 78 models, 27 of the 37 such questions that experts had judged carried an error label (73%). The data are MMLU-Pro (12,032 questions, 10 options each) and TIGER-AI-Lab's 45 published per-question model archives. The experts judged a sample of 100 questions per MMLU subject, so these results say nothing directly about MMLU-Pro's questions from other sources.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.