AI article
Your eval set is probably in your training set — here's how to check in ten minutes
Community description: Train/test contamination inflates benchmark scores quietly. Here's what it is, the three levels of overlap worth measuring, and how to check two files without writing a script.
Dev.to | Sep 13, 2026 | Thomas Kalnik
Automated excerpt
Rough orientation for word 5-grams: Short records → higher threshold. Records, not pairs: a training file that duplicates one eval example forty times is one contaminated eval record. near: 27 eval records (threshold 0. 8) 70 of 1,000 eval records (7. 00%) The first two lines are certainties. Clean the training set instead: remove every train record that matched, retrain, re-evaluate.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.