Machine Translation Evaluation: BLEU, chrF, COMET and Human Judgement

Methods & evaluation · Published 2026-10-26

Two men wearing headphones sit at a desk with microphones and an amplifier, working as interpreters in a black-and-white photograph — Interpreters at work during the post-war German war crimes trials. US National Archives, public domain.

A translation can be fluent and wrong, or clumsy and exact. Measuring machine translation (MT) quality has occupied researchers for decades, and the methods they settled on shape which systems get built and deployed. This guide explains the main metrics, what they miss, and how to evaluate translation properly.

Why translation is hard to score

Most NLP tasks have one right answer per item; translation has many. The French sentence "Il pleut des cordes" can be rendered "It's pouring", "It's raining heavily" or "It's raining cats and dogs", all acceptable. An automatic metric must therefore judge closeness to a few human references without penalising valid alternatives, and it must weigh meaning errors, which matter most to readers, against surface differences that matter little. General principles of evaluation, such as held-out test sets and significance testing, are covered in our guide to how NLP models are evaluated.

BLEU: the metric everyone quotes

BLEU was introduced by Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu of IBM in "Bleu: a Method for Automatic Evaluation of Machine Translation" (ACL 2002). It measures how many sequences of one to four words (n-grams) in the system output also occur in the reference translations, combines these precisions, and applies a "brevity penalty" so that very short outputs cannot score highly by including only safe words. Scores are usually reported on a 0–100 scale for a whole test set.

BLEU was a breakthrough because it was cheap, fast and language-independent, which allowed systems to be compared and tuned far more often than human evaluation permitted. Its weaknesses are well documented:

It rewards exact word matches, so synonyms and legitimate reordering are penalised.
It treats all words alike: missing "not" costs about as much as missing "the".
It is designed for whole test sets; sentence-level BLEU is noisy.
Its value depends on preprocessing.

The last point is serious. Matt Post, in "A Call for Clarity in Reporting BLEU Scores" (WMT 2018), showed that differences in tokenization and normalisation of the reference could shift BLEU by amounts comparable to the improvements papers were claiming. He released SacreBLEU, a tool that computes the score in a standard way and prints a "signature" recording the settings. Reporting that signature is now good practice. Our article on tokenization explains why segmentation choices matter so much.

Other surface metrics

chrF (Maja Popović, 2015) computes an F-score over character sequences. It is less sensitive to tokenization and gives partial credit for word forms that differ only in endings, which helps for morphologically rich languages.
METEOR (Banerjee and Lavie, 2005) aligns words using stems and synonyms as well as exact matches.
TER (translation edit rate, Snover and colleagues, 2006) counts the edits needed to turn the output into the reference, which roughly mirrors post-editing effort.

Learned metrics

Since about 2020, metrics built on pretrained neural models have become standard in research. COMET (Rei and colleagues, EMNLP 2020) encodes the source sentence, the system output and the reference, and is trained to predict scores given by human raters. BLEURT (Sellam and colleagues, 2020) follows a similar idea using only output and reference. Because they compare meanings rather than strings, these metrics recognise paraphrases.

The annual Conference on Machine Translation (WMT) runs a metrics shared task in which metrics are compared against human judgements. In recent editions, learned metrics have generally correlated better with people than BLEU. A large study by Tom Kocmi and colleagues at Microsoft (WMT 2021), titled "To Ship or Not to Ship", compared metrics on many system pairs judged by humans and recommended against relying on BLEU alone for deployment decisions. Learned metrics bring their own caveats: they are less transparent, their scores are not comparable across metric versions, and they can inherit biases from their training data or reward fluent output that drops content.

Human evaluation

Direct assessment

In direct assessment, raters see a source sentence (or a reference) and a translation and give a score on a 0–100 scale. Scores are standardised per rater to account for strict and lenient people. WMT used crowd-sourced direct assessment for several years because it scales to many systems.

MQM: professional error annotation

The Multidimensional Quality Metrics (MQM) framework asks trained translators to mark each error, classify it (accuracy, fluency, terminology, style and so on) and rate its severity. Errors are then weighted into a score. Markus Freitag and colleagues at Google, in "Experts, Errors, and Context" (TACL 2021), re-annotated WMT outputs with professional MQM ratings and found that they ranked systems differently from crowd ratings: in particular, professional translators clearly preferred human translations that crowd workers had rated similar to strong MT output. MQM is slower and more expensive, but it explains why a translation is worse, which a single number cannot.

Quality estimation: scoring without a reference

In real use there is no reference translation. Quality estimation (QE) models predict the quality of a translation from the source and output alone, so that a system can flag sentences likely to need human review. QE has its own WMT shared task. Its most valuable application is triage, not certification: a low score is a useful warning, but a high score does not guarantee that a critical detail, such as a date, an amount or a negation, survived.

What this means for users

Published scores describe average performance on a particular test set, usually news, parliamentary or general web text. They do not tell you how a system handles legal contracts, medical instructions or the vocabulary of your field, and they say nothing about the rare but serious errors that matter most when you act on a translation. If you rely on MT for practical decisions, check key facts with a second method; our guide to using machine translation to read a lease abroad applies these lessons to one everyday case. Professional standards also exist: ISO 18587:2017 sets requirements for the human post-editing of machine translation output.

Comparing two systems: a short protocol

Use a test set from your own domain, never seen by either system, with at least one professional reference translation.
Report more than one metric: a learned metric such as COMET together with chrF or BLEU computed with SacreBLEU, including the signatures and metric versions.
Test significance with paired bootstrap resampling rather than reading small differences as real.
Run a human evaluation on a sample, ideally MQM-style error annotation by translators of the target language.
Read the errors, paying special attention to numbers, names, negation and terminology.

Sources

Papineni, Roukos, Ward and Zhu, "Bleu: a Method for Automatic Evaluation of Machine Translation", ACL 2002 (ACL Anthology P02-1040).
Post, "A Call for Clarity in Reporting BLEU Scores", WMT 2018 (ACL Anthology W18-6319).
Popović, "chrF: character n-gram F-score for automatic MT evaluation", WMT 2015.
Rei, Stewart, Farinha and Lavie, "COMET: A Neural Framework for MT Evaluation", EMNLP 2020.
Freitag and colleagues, "Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation", TACL 2021.
Kocmi and colleagues, "To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation", WMT 2021.
ISO 18587:2017, Translation services — Post-editing of machine translation output — Requirements.