Reproducibility in NLP: Why Results Fail to Repeat and How to Fix It

Open research · Published 2026-10-26

Two handwritten pages of a nineteenth-century laboratory notebook with a small sketch of apparatus and dated entries — Pages from Alexander Graham Bell's laboratory notebook, March 1876. Public domain.

A result that cannot be repeated cannot be built on. Natural language processing moves fast and publishes much of its work openly, yet reproducing a published number is often surprisingly difficult. This guide explains why, what the research community has learned from systematic reproduction attempts, and what individual researchers can do.

Repeat, reproduce, replicate

The vocabulary is not used consistently across fields. A widely used set of distinctions, adopted by the Association for Computing Machinery for its artifact badges, separates three situations: the same team obtaining the same result with the same setup (repeatability), a different team obtaining it with the original setup and artefacts (reproducibility), and a different team obtaining a consistent result with its own independent setup (replicability). In NLP, "reproducing a paper" usually means the second case: rerunning released code, or reimplementing the method, on the same data.

Why NLP results fail to repeat

Hidden preprocessing

Antske Fokkens and colleagues tried to reproduce two published NLP experiments and described the experience in "Offspring from Reproduction Problems: What Replication Failure Teaches Us" (ACL 2013). Their attempts failed at first, and the causes were mundane: details of preprocessing, versions of resources such as WordNet, tie-breaking rules and other choices that had seemed too minor to report. Each changed the outcome. Tokenization and text normalisation are typical culprits; our article on tokenization explains why.

Randomness

Neural networks depend on random initialisation, data shuffling and non-deterministic operations on graphics processors. Nils Reimers and Iryna Gurevych, in "Reporting Score Distributions Makes a Difference" (EMNLP 2017), trained sequence-tagging models many times with different seeds and showed that the spread of scores was wide enough for one method to appear better or worse than another depending on luck. A single run is an anecdote, not a measurement.

Changing software and data

Libraries change their defaults; web pages used to build a corpus disappear or change; datasets are re-released with corrections. Without exact versions, a reimplementation may differ in ways no one can trace.

Moving targets behind an API

When an experiment calls a commercial model through an online interface, the researcher does not control the model. Lingjiao Chen, Matei Zaharia and James Zou (Stanford University and UC Berkeley) compared versions of the same widely used chat models in March and June 2023 and documented substantial changes in behaviour on several tasks over those few months. Results obtained through such services should be dated and, where possible, complemented with experiments on models whose weights are fixed and available; see our guide to what makes a language model open.

What reproduction studies have found

Anya Belz, Shubham Agarwal, Anastasia Shimorina and Ehud Reiter reviewed the reproduction literature in "A Systematic Review of Reproducibility Research in Natural Language Processing" (EACL 2021). They found that reproduced scores frequently differed from the original ones, in both directions, and that the field lacked shared definitions and methods for deciding whether a reproduction had succeeded. Follow-up work led by the same community organised shared tasks on reproducing results (ReproGen for language generation) and a multi-laboratory project, ReproHum, focused on the reproducibility of human evaluations, which are even harder to repeat than automatic scores because they depend on who the raters are and exactly what they were asked.

Outside NLP, the Machine Learning Reproducibility Challenge has invited students and researchers since 2018 to reproduce papers from major conferences and publish reports on what worked and what did not.

Checklists and incentives

Conferences now ask authors to state what they release. NeurIPS introduced a reproducibility checklist in 2019, analysed by Joelle Pineau and colleagues in the Journal of Machine Learning Research (2021). The ACL Rolling Review system used by the main computational linguistics venues requires a "Responsible NLP Research" checklist covering, among other things, the description of datasets, hyperparameters, computing infrastructure, statistics over multiple runs and licences of artefacts. Funders push in the same direction: Horizon Europe, the EU research framework programme, requires open access to peer-reviewed publications and asks projects to manage research data under the principle "as open as possible, as closed as necessary", with a data management plan.

A practical routine

Freeze the environment. Record library versions in a lock file or container image, and the hardware used.
Version the data. Store checksums of every dataset file, the date of download, and scripts for any filtering or preprocessing. If you cannot share the data, share the script that rebuilds it and identifiers for every document (see dataset licences).
Run several seeds and report the mean, the spread and the number of runs.
Log every hyperparameter, including those left at default values, and the search procedure used to choose them.
Date and version external models. For API-based experiments, record model identifiers, dates, prompts and decoding settings, and keep the raw outputs.
Release code and outputs in an archive with a persistent identifier (for example a DOI from a repository such as Zenodo), not only in a personal code repository that may move.
Document data and models with datasheets and model cards, explained in our article on documentation for datasets and models.

Reproducing someone else's paper

If you set out to reproduce a result, decide in advance what will count as success (an exact match, a score within a stated tolerance, or the same ranking of systems), contact the authors early, and publish the outcome whether or not it matches. Negative reproductions are informative, and several venues now welcome them. Evaluation pitfalls that often explain discrepancies, such as test-set contamination and missing significance tests, are covered in our guide to evaluating NLP models.

Sources

Fokkens, van Erp, Postma, Pedersen, Vossen and Freire, "Offspring from Reproduction Problems: What Replication Failure Teaches Us", ACL 2013.
Reimers and Gurevych, "Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging", EMNLP 2017.
Belz, Agarwal, Shimorina and Reiter, "A Systematic Review of Reproducibility Research in Natural Language Processing", EACL 2021.
Pineau and colleagues, "Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)", JMLR, 2021.
Chen, Zaharia and Zou, "How Is ChatGPT's Behavior Changing over Time?", 2023 (later published in Harvard Data Science Review, 2024).
Association for Computing Machinery, "Artifact Review and Badging", version 1.1.
European Commission, Horizon Europe Programme Guide, open science requirements.