MaxxBench – "Nice DeepSWE score. But is it true?"

Automated excerpt

It changes the instructions or starting code of 89 DeepSWE tasks, giving models familiar engineering work with a reason to revise their usual approach. Claude Opus 5.5 and Gemini 3.8 Flash both have published DeepSWE scores of about 74%. Every attempt across all 13 models passed the self-reported peak check, while most failed the independent live-record check.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.