AI article
AI Agent Testing: Why a 77% Pass Rate Can Mean 53% in Production
Community description: IBM's consistency gap paper shows an agent passing 77% of runs succeeds five times in a row only 53% of the time. Here's the five-run harness I now us
Dev.to | Sep 14, 2026 | Qasim Parray
Automated excerpt
Twenty-two test cases, all green, three days in a row. The agent just took a different path the second time. Take a ReAct agent on the AppWorld benchmark using GPT-4.1. Count two things: the average pass rate per run, and the fraction of tasks that pass all five runs. Take the ten test cases you already have for whatever agent you've shipped, run each one five times tonight, and count how many pass all five.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.