Memorisation in Language Models: How Training Data Leaks and How It Is Measured
A language model is supposed to learn patterns, not store documents. Yet since the late 2010s, security researchers have shown again and again that models can reproduce pieces of their training data verbatim: a person's contact details, a licence key, a paragraph of a news article, a block of source code. This is called memorisation. It matters for privacy, for copyright and for the honesty of benchmarks, and it has become one of the best-studied failure modes of large models.
What "memorisation" means
Every model that generalises also retains something about its training examples; otherwise it would learn nothing. The question is when retention turns into reproduction. Researchers use several operational definitions. The most common is extractable memorisation: a string from the training data is memorised if there is a prompt that makes the model generate it. A stricter version, used by Carlini and colleagues in their 2022 study of scaling, counts a sequence as memorised if, given the preceding tokens from the training document as a prompt, the model's greedy output reproduces the next tokens exactly. Other work distinguishes counterfactual memorisation: how much a model's behaviour on an example depends on that example having been in the training set, as opposed to similar examples.
The distinction matters. A model that completes "To be or not to be" with the rest of Hamlet's line has technically memorised it, but the line appears thousands of times on the web and is common knowledge. A model that completes a stranger's name with her home address, seen once in a forum dump, raises a very different concern.
The extraction attacks
An early demonstration was "The Secret Sharer" (Carlini, Liu, Erlingsson, Kos and Song, USENIX Security 2019). The authors inserted artificial "canaries", such as a fake credit card number, into training data and defined an exposure metric for how much more likely the model found the real canary than random alternatives. Even canaries inserted a handful of times became highly exposed.
The landmark result came two years later. In "Extracting Training Data from Large Language Models" (USENIX Security 2021), Nicholas Carlini and eleven co-authors attacked GPT-2 without any access to its training set. They generated large numbers of samples, ranked them by signals suggesting memorisation (for example, text the model found far more likely than a smaller reference model or a compression algorithm did), and checked the top candidates with OpenAI. They recovered hundreds of verbatim sequences, including names, phone numbers and email addresses, IRC conversations, code and 128-bit UUIDs, some of which appeared in only one document of the training data. Larger models were more vulnerable than smaller ones.
In 2023, Milad Nasr, Carlini and colleagues scaled the approach to open models such as Pythia and GPT-Neo, semi-open ones such as LLaMA and Falcon, and to ChatGPT. Against the chat-tuned model they used a "divergence" attack, asking it to repeat a single word indefinitely until it drifted out of its assistant behaviour and began emitting training data. Their conclusion was blunt: alignment training changes how a model usually behaves, but it does not remove what it has memorised.
What makes memorisation worse
The 2022 paper "Quantifying Memorization Across Neural Language Models" (Carlini, Ippolito, Jagielski, Lee, Tramèr and Zhang, published at ICLR 2023) measured memorisation systematically and found three log-linear relationships. Memorisation grows with:
- Model capacity. Larger models in the same family reproduce more of their training data.
- Duplication. A sequence that appears many times in the corpus is far more likely to be memorised than one that appears once.
- Context length. The more of the original document is given as a prompt, the more likely the model is to continue it exactly.
The authors noted that the picture becomes more complicated across model families, and warned that memorisation was more prevalent than previously believed and likely to get worse with scale unless actively mitigated.
Duplication is the lever practitioners control most directly. Katherine Lee and colleagues ("Deduplicating Training Data Makes Language Models Better", ACL 2022) found that standard corpora contain many near-duplicate examples and long repeated substrings; in C4, they found one 61-word English sentence repeated more than 60,000 times. Removing duplicates let them train models that emitted memorised text ten times less often, with the same or better accuracy, and also reduced overlap between training and test sets.
Membership inference: was this text in the training set?
Extraction asks the model to give data back. Membership inference asks a weaker question: given a candidate text, can we tell whether it was used in training? The idea comes from Shokri and colleagues (IEEE Symposium on Security and Privacy, 2017), who attacked classifiers. For language models, the usual signal is that training texts receive unusually low loss or high likelihood.
Membership inference is attractive to authors and rights holders who want to know whether their work was used, and to auditors checking whether personal data was removed. But it is harder than it looks for large pretrained models. Michael Duan and colleagues ("Do Membership Inference Attacks Work on Large Language Models?", COLM 2024) found that many attacks barely beat random guessing on pretraining data, partly because a single pass over a huge corpus leaves a weak trace, and partly because "non-member" test texts often differ in date or topic from member texts, which inflates apparent success. Claims that a particular book or article "was in the training data" based on such tests deserve scrutiny.
Why it matters
Privacy. If a model can output personal data it was trained on, the model itself may contain personal data. That is why the European Data Protection Board concluded in 2024 that AI models cannot automatically be treated as anonymous; see our guide to the GDPR and text data.
Copyright. Verbatim reproduction of protected text is the clearest form of copying a model can perform, and examples of regurgitated articles have featured in litigation against model developers. The legal background is covered in web-scraped corpora, copyright and consent.
Evaluation. A model that has memorised benchmark questions and answers will score well without having the ability the benchmark is supposed to measure. Data contamination is one of the blind spots discussed in how NLP models are evaluated.
Some memorisation may also be useful. Vitaly Feldman (STOC 2020) argued that when data has a long tail of rare examples, memorising some of them is necessary for good accuracy. The goal is therefore not zero memorisation, but control over what is memorised.
Mitigations, and what they cost
- Deduplication of training data, exact and near-duplicate, is cheap relative to training and has the best evidence behind it.
- Filtering personal data before training (email addresses, phone numbers, keys) removes the most obvious targets, though detectors miss indirect identifiers.
- Differential privacy, usually through DP-SGD (Abadi and colleagues, ACM CCS 2016), bounds how much any single training example can influence the model. It offers formal guarantees, but at a cost in accuracy and compute that has so far limited its use for pretraining large models; it is more common for fine-tuning on sensitive data.
- Output filters that block generations matching the training set, or known protected text, reduce visible regurgitation but leave the information in the weights, and can be bypassed by paraphrase or encoding tricks.
- Machine unlearning methods try to remove specific data after training. They are an active research area, and verifying that information is really gone remains difficult.
For teams releasing models, the practical lesson is to test before publishing: insert canaries, run extraction-style probes for rare strings, and document the results alongside the training data description. Openness makes these tests possible for outsiders too, one of the arguments discussed in what makes a language model open.
Sources
- Carlini, Tramèr, Wallace, Jagielski, Herbert-Voss, Lee, Roberts, Brown, Song, Erlingsson, Oprea and Raffel, "Extracting Training Data from Large Language Models", USENIX Security Symposium 2021.
- Carlini, Ippolito, Jagielski, Lee, Tramèr and Zhang, "Quantifying Memorization Across Neural Language Models", ICLR 2023.
- Nasr, Carlini, Hayase, Jagielski, Cooper, Ippolito, Choquette-Choo, Wallace, Tramèr and Lee, "Scalable Extraction of Training Data from (Production) Language Models", arXiv:2311.17035, 2023.
- Lee, Ippolito, Nystrom, Zhang, Eck, Callison-Burch and Carlini, "Deduplicating Training Data Makes Language Models Better", ACL 2022.
- Carlini, Liu, Erlingsson, Kos and Song, "The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks", USENIX Security Symposium 2019.
- Duan and colleagues, "Do Membership Inference Attacks Work on Large Language Models?", COLM 2024.
Ethics & dataGDPR and text data
Names, emails and health details hide in every corpus. How the GDPR applies to NLP data and models, why text anonymisation is hard, and what to do.
Ethics & dataWeb-scraped corpora and consent
Common Crawl, robots.txt, the EU's text and data mining opt-out and the AI Act: the legal and ethical questions behind training data scraped from the web.
Methods & evaluationHow NLP models are evaluated
Splits, F1, leaderboards, significance tests and contamination: how to read an NLP evaluation, and how to run one you can trust.