BenchMIRT: What are LLM benchmarks actually measuring?

Automated excerpt

BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. BenchMIRT applies IRT at both the model and question level. Crucially, we didn’t tell BenchMIRT which benchmarks were measuring which capabilities. What BenchMIRT reveals about existing benchmarks For many benchmarks, BenchMIRT largely confirmed their intended focus: strong performance on reasoning benchmarks tracked with reasoning ability, while strong performance on jailbreak and harmful-content benchmarks tracked with safety. WMDP behaves differently from most safety benchmarks.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.