AI article
My Eval Passed Because the Model Had Already Seen the Answers
Community description: My first green eval run was worthless: the training data and the test set came from the same sources, so the model was reciting. Fixing that exposed two more traps. Three runs of one unchanged model scored 31, 21 and 31 out of 50, and a later set that looked steady at 27, 28, 27 was hiding 29 of 50 tests changing verdict underneath.
Dev.to | Oct 1, 2026 | Chidozie Uzoegwu
Automated excerpt
A baseline is only valid against the identical battery version. So, precisely: three runs of one model, 50 rows, six samples per row. Only 21 rows gave the same answer all three times.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.