A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

Automated excerpt

Evol-Instruct grows a dataset by repeatedly rewriting seed problems, and some of the seeds were benchmark problems. Practice on HumanEval-style problems helped on HumanEval-style problems generally, including ones that were never in the data. Only 10 MBPP problems matched removed data, and the two original-data runs differ by 11 problems.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.