AI article
Every LLM Eval Library Has the Same Bug: Stochastic Judges Used as Deterministic Oracles
Your eval library calls the judge once per test case and prints a number. The judge flips its verdict on 5-15% of reruns. That number is noise wearing a suit.
Dev.to | Apr 29, 2026 | Gabriel Anhaia