How NLP Models Are Evaluated: Benchmarks, Metrics and Their Blind Spots

Methods & evaluation · Published 2026-10-26

A yellow tape measure marked in feet and centimetres lying on a wooden floor next to two copper coins — A tape measure with two scales. Photo: Stilfehler via Wikimedia Commons, public domain.

Almost every NLP paper ends with a table of numbers that claims one system beats another. Those numbers are only as meaningful as the test data, metrics and procedures behind them. This guide explains how evaluation works, where it goes wrong, and what to check before you trust a leaderboard.

The basic recipe: train, develop, test

Supervised NLP experiments split annotated data into three parts. The training set is used to fit the model. The development (or validation) set is used to choose settings such as learning rate or model size. The test set is kept aside and used once, at the end, to estimate how the system will behave on new data. Every time a researcher looks at test results and adjusts the system, the test set becomes a little less "unseen", and scores become a little more optimistic.

Many datasets come with a standard split so that results are comparable across papers. That convenience has a cost. Kyle Gorman and Steven Bedrick, in "We Need to Talk about Standard Splits" (ACL 2019), re-ran several part-of-speech taggers on randomly generated splits of the same corpus and found that the ranking reported with the standard split could not always be reproduced. A single split can reward systems that happen to suit it.

Choosing a metric

Accuracy, the share of correct predictions, is intuitive but misleading when classes are imbalanced: if 95% of emails are not spam, a filter that never flags anything is 95% accurate. Two measures help:

Precision: of the items the system labelled positive, how many really were.
Recall: of the items that really were positive, how many the system found.

The F1 score is their harmonic mean, which is low whenever either is low. For tasks with many classes, check whether a paper reports a macro average (every class counts equally, so rare classes matter) or a micro average (every item counts equally, so frequent classes dominate). The two can tell very different stories.

Generation tasks such as translation and summarisation need different tools, because many different outputs can be correct. Overlap metrics compare a system's output with one or more human references; learned metrics use a trained model to judge quality. Our guide to machine translation evaluation covers BLEU, chrF, COMET and human evaluation in detail.

Human agreement sets the ceiling

Test labels come from people, and people disagree. Before reading a system's score, look at the inter-annotator agreement reported for the dataset, typically measured with Cohen's kappa (for two annotators) or Krippendorff's alpha (for several), both of which correct for agreement expected by chance. When trained annotators agree only moderately, as often happens with fine-grained word senses (see word sense disambiguation), sentiment or toxicity, a system that "beats humans" on the benchmark may simply be matching one annotator's habits.

Benchmarks and leaderboards

Benchmark suites bundle several tasks to give a broader picture. GLUE (Wang and colleagues, 2018) combined nine English sentence-understanding tasks. Within a short time, top systems exceeded the human baseline its authors had estimated, and the same team released the harder SuperGLUE in 2019. This cycle of saturation has repeated many times: a benchmark designed to be difficult becomes easy, not always because models understand language better, but because they get better at the benchmark.

Researchers have tried several answers. Stanford's Center for Research on Foundation Models launched HELM (Holistic Evaluation of Language Models) in 2022 to evaluate many models on many scenarios with multiple metrics, including calibration, robustness and bias, not only accuracy. Dynabench (Kiela and colleagues, NAACL 2021) collects new test examples adversarially, by asking people to write inputs that fool current models. CheckList (Ribeiro and colleagues, ACL 2020) borrows from software testing: it probes specific capabilities, such as handling negation or changing a name, with templated test cases.

When models learn the wrong thing

A high score does not prove that a model solved the intended task. Suchin Gururangan and colleagues showed in 2018 that, in popular natural language inference datasets, a classifier could often guess the label from the hypothesis sentence alone, without reading the premise, because crowdworkers had used telltale words when writing contradictions. Tom McCoy and colleagues (ACL 2019) built a test set showing that models relied on simple heuristics such as word overlap. Such "annotation artefacts" are now a standard concern: good papers report simple baselines that use only part of the input, to show how much of the task can be solved by shortcuts.

Is the difference real?

Two systems that differ by half a point may not differ at all. Neural training involves random initialisation and data ordering; Nils Reimers and Iryna Gurevych (EMNLP 2017) showed that changing only the random seed could change scores enough to reverse conclusions, and recommended reporting score distributions over several runs. Rotem Dror and colleagues, in "The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing" (ACL 2018), surveyed papers in major venues, found that significance testing was often missing or misapplied, and explained which tests suit which metrics. Bootstrap resampling and approximate randomisation tests are common, assumption-light choices.

Data contamination

Large language models are trained on huge web crawls, and popular benchmarks are published on the web. If test items, or their answers, appear in the training data, the model may have memorised them. Oscar Sainz and colleagues (Findings of EMNLP 2023) argued that contamination should be measured and reported for each benchmark, because without access to the training data, reviewers cannot rule it out. Signs to look for include a model reproducing test items verbatim, scores far higher on older benchmarks than on newly created ones of similar difficulty, and evaluations of closed models whose training data is undisclosed. Documentation of training data, discussed in our article on reproducibility, is the only reliable remedy.

A checklist for reading (or running) an evaluation

Is the test set truly held out, and is there any chance it appeared in pre-training data?
Does the metric fit the task, and is the averaging method (macro or micro) stated?
What is human agreement on the dataset, and how strong are trivial baselines such as the majority class?
Are results averaged over several runs with variance or confidence intervals, and is a significance test reported?
Is preprocessing documented, including tokenization (see how tokenization works) and any filtering of the test data?
Is there an error analysis that looks at what the system gets wrong, not only how often?

Sources

Gorman and Bedrick, "We Need to Talk about Standard Splits", ACL 2019 (ACL Anthology P19-1267).
Dror, Baumer, Shlomov and Reichart, "The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing", ACL 2018 (ACL Anthology P18-1128).
Reimers and Gurevych, "Reporting Score Distributions Makes a Difference", EMNLP 2017.
Wang and colleagues, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding", 2018; "SuperGLUE", NeurIPS 2019.
Liang and colleagues, "Holistic Evaluation of Language Models", Stanford CRFM, 2022.
Gururangan and colleagues, "Annotation Artifacts in Natural Language Inference Data", NAACL 2018; McCoy, Pavlick and Linzen, "Right for the Wrong Reasons", ACL 2019.
Ribeiro, Wu, Guestrin and Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList", ACL 2020.
Sainz and colleagues, "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark", Findings of EMNLP 2023.