My nightly eval gate went red on prompts nobody touched. The judge had changed its mind

Automated excerpt

The answer was in the judge logs: the same reply to the same ticket had scored 0. 7 the night before and 0. 6 that night, and my pass mark was 0. 7. The judge had scored the same input two different ways. An LLM judge gives you a slightly different score each time you run it, so on a gate that grades thousands of cases, some of the red is the judge disagreeing with itself.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.