Language Identification: How NLP Systems Tell Which Language a Text Is In

Multilingual & low-resource · By the Parallel Text editorial team · Published 2026-11-05

Corner of a weathered wooden house with blue street-name plates, one giving the same street name in three lines of Cyrillic text — A street sign in Saransk, Mordovia, in Russian, Erzya and Moksha: three languages, one script. Photo: SarRus, public domain, via Wikimedia Commons.

Before a multilingual corpus can be built, a translation system trained or a search engine tuned for a market, someone has to decide which language each document is written in. At web scale that decision is made by a classifier, usually in milliseconds and without human review. Language identification looks like a solved problem: for long, clean texts in major languages it nearly is. For short messages, closely related languages and the thousands of languages with little digital text, it is one of the main reasons multilingual data is worse than it appears.

What the task is, exactly

Language identification (often shortened to LID, or "language detection" in software libraries) assigns a language label to a span of text. That sounds simple, but every part of the definition hides a choice. Which languages are on the list? A model that knows only 100 languages will label a text in an unlisted language as the closest listed one, with confidence. What counts as a separate language? Serbian, Croatian and Bosnian are mutually intelligible standard varieties; Norwegian has two written standards; Hindi and Urdu share much of their spoken language but use different scripts. And what is the unit: a document, a sentence, or a single word in a message that switches between two languages?

Labels are usually codes from ISO 639 (two-letter codes such as fr, or three-letter ISO 639-3 codes such as myv for Erzya and mdf for Moksha) or BCP 47 tags, which can also record the script, as in sr-Cyrl and sr-Latn for Serbian in Cyrillic and Latin letters. Recording the script matters more than it may seem: the same language in two scripts behaves like two languages for a classifier, and the same script shared by many languages is where most confusions happen.

How it works: from n-gram profiles to neural classifiers

Character n-grams

The classic method was described by William Cavnar and John Trenkle in 1994. For each language, count the most frequent sequences of one to five characters in a sample of text and rank them; this ranked list is the language's profile. To classify a new document, build its profile the same way and choose the language whose profile is closest, measured by how far each n-gram's rank is "out of place". Sequences such as th, ing and the rank high in English, ij and sch in Dutch, ő and sz in Hungarian. No dictionary is needed, the method tolerates spelling errors, and it works for any language with enough sample text.

Most later systems keep the same raw material and improve the statistics. Marco Lui and Timothy Baldwin's langid.py (ACL 2012 demonstrations) selected the n-gram features that distinguish languages well across several text domains and shipped a pretrained model covering 97 languages. Because these models read characters rather than words, they share some of the logic of the subword methods described in our article on tokenization.

Linear and neural classifiers

Two families now dominate in practice. fastText, released by Facebook AI Research and described by Armand Joulin and colleagues in "Bag of Tricks for Efficient Text Classification" (2016), represents a text as the average of embeddings for its words and character n-grams and feeds it to a linear classifier. It trains and predicts very fast, which is why it is the basis of many corpus-building pipelines; the project's public language identification model is named lid.176 after the 176 languages it covers. Google's Compact Language Detector 3 (CLD3) uses a small feed-forward neural network over character n-gram embeddings and is designed to run inside a browser. Large multilingual language models can also identify languages, but they are slow and expensive for filtering billions of documents, so they are rarely used for that first pass.

More recent open models focus on the training data and on coverage rather than on architecture. OpenLID (Laurie Burchell and colleagues, ACL 2023) is a fastText model for 201 languages whose training data was audited by hand and released with the model, so that its errors can be traced. GlotLID (Amir Hossein Kargaran and colleagues, Findings of EMNLP 2023) extends coverage to more than 1,600 languages, with particular attention to labels that are ambiguous or misused in the source data.

Why it fails

Short texts. A product name, a search query or a two-word reply contains too few n-grams to tell Spanish from Portuguese. Accuracy figures reported on full sentences or paragraphs say little about tweets.
Closely related languages. Bosnian, Croatian and Serbian in Latin script, Malay and Indonesian, Danish and Norwegian Bokmål, or Erzya and Moksha on the sign above, share most of their frequent n-grams. Distinguishing them is a research task in its own right, with dedicated shared tasks at the VarDial workshops.
Mixed and noisy input. Code-switching (two languages in one message), transliteration (Hindi or Arabic typed in Latin letters), boilerplate such as cookie banners, and fragments of HTML all push text away from the clean training data.
Out-of-list languages. A text in a language the model does not know will still receive a label, usually that of a related or higher-resource language, and often with high confidence.

The base-rate problem on the web

The most damaging failure is statistical rather than linguistic. Suppose a classifier wrongly labels one English sentence in a million as a small regional language. On a web crawl containing billions of English sentences, that tiny error rate produces thousands of false positives, which may outnumber all the genuine text in the small language. Isaac Caswell and colleagues described exactly this when building language identification for more than a thousand languages ("Language ID in the Wild", COLING 2020): models that looked excellent on held-out test sets produced corpora for many long-tail languages that were dominated by text in other languages or by non-linguistic content.

An audit published in Transactions of the ACL in 2022, "Quality at a Glance" by Julia Kreutzer and colleagues, manually audited samples of 205 language-specific corpora released with five widely used multilingual datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR and mC4). Its authors found systematic problems in lower-resource corpora: at least 15 contained no usable text, and a significant fraction contained less than 50% sentences of acceptable quality. Many were mislabelled or used nonstandard or ambiguous language codes, and the authors stress that these issues are easy to spot, even for people who do not speak the language. The consequences reach far beyond classification: these corpora feed the parallel corpora used to train translation systems and the web-scale datasets discussed in our article on web-scraped corpora. A model trained on a "Twi" corpus that is half English and half boilerplate will not learn much Twi, and its evaluation may not reveal it.

What helps

Confidence thresholds per language. Keep a document only if the top label's probability exceeds a threshold, and set stricter thresholds for languages that attract false positives.
A second, independent filter. Caswell and colleagues proposed checking candidate documents against short lists of frequent, distinctive words for each language; a text labelled Erzya that contains none of the commonest Erzya words is suspect. Script checks catch many errors at almost no cost.
Document-level and sentence-level passes. Labelling a whole page hides embedded passages in other languages; labelling single sentences increases errors on short ones. Many pipelines use both and keep only agreements.
Human inspection. A few minutes spent reading a random sample with a speaker of the language reveals problems that no aggregate metric shows. The audit above is the best argument for doing it.
Documentation. Record the identifier, its version, the thresholds and the label set in the dataset's documentation, so that users know what "Erzya" means in your corpus.

How to evaluate a language identifier

Overall accuracy is close to meaningless when the test set is dominated by a few large languages. Report precision, recall and F1 for each language, and macro-averaged scores across languages so that each counts equally. Test on text that resembles real input: short segments if the system will see search queries, noisy web text if it will filter a crawl. The FLORES-200 benchmark, with the same sentences translated into some 200 language varieties, is a common clean test set, and its labels pair a language code with a script code. Because it contains the same content in every language, it is good for comparing languages but optimistic about real data. Finally, look at the confusion matrix: it shows which pairs of languages the model mixes up, and those pairs usually deserve their own test set. For the wider methodological cautions, see our guide to low-resource languages, where much of this matters most.

Sources

Joulin, Grave, Bojanowski and Mikolov, "Bag of Tricks for Efficient Text Classification", arXiv:1607.01759, 2016 (fastText; the lid.176 model is documented on the fastText website).
Kreutzer, Caswell, Wang and colleagues, "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets", Transactions of the ACL, 2022.
Kargaran, Imani, Yvon and Schütze, "GlotLID: Language Identification for Low-Resource Languages", Findings of EMNLP 2023.
Cavnar and Trenkle, "N-Gram-Based Text Categorization", SDAIR 1994; Lui and Baldwin, "langid.py: An Off-the-shelf Language Identification Tool", ACL 2012 System Demonstrations; Burchell, Birch, Bogoychev and Heafield, "An Open Dataset and Model for Language Identification", ACL 2023; Caswell, Breiner, van Esch and Bapna, "Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus", COLING 2020.