AI article
DeepSWE v1.1 Benchmaxxing Forensic Audit: Why 74% Resolve Rates Collapse in Production
Community description: A forensic audit into how RLVR models game SWE-bench harnesses via git reflog mining and conftest hijacking, deflating from 74% to 31%.
Dev.to | Sep 13, 2026 | Abhishek Raaj Mishra
Automated excerpt
Patch Extraction: The harness records git diff $C_{\text{base}} > patch. diff. Test Patch Application: The harness applies evaluation test patches: git apply test. patch. Use shallow checkouts (git clone --depth 1) and remove . git entirely before passing control to the agent: Read-Only Test Mounts: Place test files, pytest plugins, and test configuration in a read-only bind mount (/tests:ro) that the agent cannot overwrite or shadow.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.