Benchmark Data Contamination: When Test Sets Leak into Training Data
A benchmark score is meant to estimate how a system will do on inputs it has never seen. That assumption breaks if the test questions, or their answers, were part of the training data. For models trained on large scrapes of the web, this is not a hypothetical risk: popular benchmarks are published online, copied into repositories, discussed on forums and quoted in papers, and all of that text can be collected along with everything else. The problem has a name, data contamination, and it now shapes how NLP results should be read.
What counts as contamination
The clearest case is verbatim: the exact test items, with their gold labels, are in the training corpus. But contamination comes in degrees. A test question may appear without its answer, which still makes the input familiar. It may appear paraphrased, translated, or reformatted as a quiz on a study website. Sometimes only the answer key is online, in a table or a solutions file. Oscar Sainz and colleagues, in a 2023 position paper titled "NLP Evaluation in trouble", distinguish contamination during pretraining from contamination at later stages, such as instruction tuning on datasets that include benchmark training or test splits, and argue that it should be measured and reported benchmark by benchmark rather than assumed away.
There is also a quieter route. Closed models served through an API may receive benchmark items from the researchers who evaluate them, and depending on the provider's data policy those prompts can later be used for training. Simone Balloccu and colleagues ("Leak, Cheat, Repeat", EACL 2024) reviewed published evaluations of ChatGPT and documented how often test data had been sent to the model in this way.
Why it is so hard to rule out
Contamination is not new. The paper introducing GPT-3 (Brown and colleagues, 2020) already searched for 13-gram overlaps between its training data and the test sets it reported on, and acknowledged that a bug had left some overlapping material in the training data; it then reported results on "clean" subsets alongside the full ones. Documentation of the C4 corpus by Jesse Dodge and colleagues (EMNLP 2021) found examples from other NLP benchmark datasets inside that widely used web crawl.
Three things make the problem worse today. Training corpora have grown to trillions of tokens, so exhaustive checks are expensive even for those who hold the data. Many developers do not publish their training data at all, so outside researchers cannot check it, a recurring issue in the debate on what makes a model open. And benchmarks live for years: the longer a test set is public, the more copies of it circulate.
How contamination is detected
Searching the training data
When the corpus is available, the direct method is to look for overlaps: long n-grams shared between test items and training documents, or near-duplicates found with fuzzy matching. The choice of n matters. Short n-grams flag common phrases and produce false alarms; long ones miss paraphrases and translations. Overlap tells you that test text was present, not whether the model learned from it, so a good report also compares scores on the overlapping and non-overlapping parts of the test set. If the two are close, the contamination probably did little harm; if the overlapping part scores much higher, the headline figure is inflated.
Probing the model
Without the training data, researchers look for traces of memorisation in the model's behaviour, using techniques close to those described in our article on memorisation in language models.
- Guided completion. Shahriar Golchin and Mihai Surdeanu ("Time Travel in LLMs", ICLR 2024) give the model the name of a dataset, the split, and the first part of a test instance, and ask it to complete the rest. Exact or near-exact reproduction of the reference is a sign that the instance was seen.
- Token probabilities. The Min-K% Prob method of Weijia Shi and colleagues (ICLR 2024) rests on a simple observation: text a model has never seen usually contains a few surprising words with very low probability, while memorised text does not. Averaging the log-probabilities of the least likely tokens gives a score that can flag likely training examples, provided the model exposes token probabilities.
- Ordering tests. Yonatan Oren and colleagues ("Proving Test Set Contamination in Black Box Language Models", ICLR 2024) use the fact that, in an uncontaminated model, every ordering of a benchmark's examples should be about equally likely. A model that assigns much higher likelihood to the published order than to shuffled versions has probably seen the file. Because the test compares against shuffles, it yields a statistical guarantee rather than a guess.
- Fresh look-alike test sets. A different approach builds a new test set that mirrors an old one. Hugh Zhang and colleagues commissioned GSM1k, a set of grade-school maths problems designed to match the established GSM8k benchmark in difficulty and style (arXiv:2405.00332, 2024). Several model families scored lower on the new problems than on the old ones, with drops in accuracy of up to 8%, while many frontier models showed little sign of overfitting.
Each method has blind spots. Completion probes miss models trained to refuse or paraphrase; probability-based scores need access to log-probabilities and have their own false positives; ordering tests need datasets whose examples are interchangeable. A negative result is therefore weaker evidence than a positive one: failing to detect contamination does not prove its absence.
What benchmark builders can do
Alon Jacovi and colleagues ("Stop Uploading Test Data in Plain Text", EMNLP 2023) set out practical rules for people who publish test sets:
- Encrypt or gate the test split. Distribute test data encrypted with a public key, or behind a simple access step, so that crawlers do not pick it up as ordinary text; add a licence that forbids redistribution of derived copies.
- Use canary strings. The BIG-bench project embeds a unique identifier, a "canary" GUID, in its task files. Corpus builders can filter any document containing it, and anyone can test whether a model reproduces it.
- Avoid items whose solutions are already online. Questions scraped from the web often come with their answers on the same page; if they are used, release the page context so that others can check exposure.
- Refresh. Benchmarks such as LiveBench (Colin White and colleagues, 2024) regularly add questions drawn from recent sources, so that a model can be evaluated on material published after its training data was collected.
How to read a result in the meantime
For anyone comparing models, a few questions go a long way. How old and how widely copied is the benchmark? Did the authors run a contamination check, and on which data? Are results on a clean subset or a newer test set reported next to the headline score? Is the gap between a model and its predecessors suspiciously large on one benchmark and ordinary elsewhere? These questions belong with the broader cautions in our guide to how NLP models are evaluated, and with the reporting habits that make results checkable, discussed in our article on reproducibility in NLP: state which benchmark version was used, when the evaluation was run, and what was done to rule out leakage.
Contamination does not make benchmarks useless. It changes what a score means: a measurement of performance on a particular, public set of questions, whose relationship to performance on new questions has to be argued for rather than assumed.
Sources
- Sainz and colleagues, "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark", arXiv:2310.18018, 2023.
- Oren and colleagues, "Proving Test Set Contamination in Black Box Language Models", ICLR 2024.
- Zhang and colleagues, "A Careful Examination of Large Language Model Performance on Grade School Arithmetic", 2024.
- Brown and colleagues, "Language Models are Few-Shot Learners", NeurIPS 2020; Dodge and colleagues, "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus", EMNLP 2021; Golchin and Surdeanu, "Time Travel in LLMs: Tracing Data Contamination in Large Language Models", ICLR 2024; Shi and colleagues, "Detecting Pretraining Data from Large Language Models", ICLR 2024; Jacovi and colleagues, "Stop Uploading Test Data in Plain Text", EMNLP 2023; Balloccu and colleagues, "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs", EACL 2024; White and colleagues, "LiveBench: A Challenging, Contamination-Limited LLM Benchmark", 2024.
Methods & evaluationHow NLP models are evaluated
Splits, F1, leaderboards, significance tests and contamination: how to read an NLP evaluation, and how to run one you can trust.
Ethics & dataMemorisation in language models
Phone numbers, code and whole paragraphs can come back out of a trained model. How researchers extract and measure memorised text, and what reduces it.
Open researchReproducibility in NLP
Preprocessing, seeds, versions and silent API updates: why NLP results fail to repeat, what studies found, and a routine that makes your work checkable.