Bias in NLP: Where It Comes From, How It Is Measured and What Helps

Ethics & data · Published 2026-10-26

Two women programmers set up the wiring panels of the room-sized ENIAC computer in a black-and-white photograph — Two programmers rewiring ENIAC, one of the first general-purpose electronic computers, in the 1940s. US Army photo, public domain.

Language technology learns from text written by people, and text carries the assumptions, stereotypes and inequalities of the societies that produce it. The result can be systems that translate "the doctor" as a man and "the nurse" as a woman, or flag some dialects as offensive more often than others. This article summarises what research has found, how bias is measured, and what reduces harm.

What "bias" means here

In statistics, bias means systematic error. In discussions of language technology the word covers several distinct harms, and mixing them up produces confusion. A common distinction, drawn by researchers including Solon Barocas and Kate Crawford, separates representational from allocational harms, and many researchers add a third category:

Representational harms: a system portrays groups in stereotyped, demeaning or invisible ways, for example by completing sentences about a religion with references to violence.
Allocational harms: a system's outputs lead to resources or opportunities being distributed unfairly, for example a CV-screening tool that ranks candidates differently according to words associated with gender.
Quality-of-service gaps: a system simply works worse for some people, such as speech recognition that makes more errors for certain accents, or NLP tools that handle standard varieties of a language better than regional ones.

The US National Institute of Standards and Technology adds a further lens in its Special Publication 1270 (March 2022), which distinguishes systemic, statistical and computational, and human-cognitive biases, and argues that bias in AI cannot be addressed by technical fixes alone.

What research has found

Word embeddings

Tolga Bolukbasi and colleagues (NeurIPS 2016) showed that word vectors trained on a large news corpus completed analogies with occupational stereotypes, as captured in the title of their paper, "Man is to Computer Programmer as Woman is to Homemaker?". Aylin Caliskan, Joanna Bryson and Arvind Narayanan (Science, 2017) developed a test modelled on psychology's Implicit Association Test and found that widely used embeddings reproduced many human-like associations, including ones linking names typical of different groups with pleasant or unpleasant words.

Coreference and translation

Two benchmarks published in 2018, WinoBias and Winogender, showed that coreference systems tended to link pronouns to occupations according to gender stereotypes ("the nurse... she", "the mechanic... he") rather than the sentence's meaning. Gabriel Stanovsky, Noah A. Smith and Luke Zettlemoyer applied this idea to machine translation in "Evaluating Gender Bias in Machine Translation" (ACL 2019): when translating from English into languages that mark gender grammatically, the commercial and academic systems they tested frequently chose stereotypical genders, even when the English sentence made the opposite clear.

Dialects and toxicity detection

Maarten Sap and colleagues (ACL 2019) found that tweets written in African American English were more likely to be labelled offensive, both by annotators in widely used datasets and by models trained on them. Content moderation systems built on such data can silence the communities they are meant to protect.

Large language models

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell (under the pseudonym "Shmargaret Shmitchell"), in "On the Dangers of Stochastic Parrots" (FAccT 2021), argued that ever-larger web-trained models absorb hegemonic viewpoints from their data, that filtering steps can exclude marginalised voices, and that the data is too large to document properly. Later audits of filtered web corpora supported parts of this argument; see our article on web-scraped corpora.

A critical look at the field

Su Lin Blodgett, Solon Barocas, Hal Daumé III and Hanna Wallach reviewed 146 papers on bias in NLP in "Language (Technology) is Power: A Critical Survey of 'Bias' in NLP" (ACL 2020). They found that many papers did not say clearly what harm they were concerned with, why it was harmful, or to whom, and that measurement techniques were often disconnected from those motivations. Their recommendations: ground work in the relationship between language and social hierarchies, state the harms explicitly, and engage with the communities affected.

The limits of technical fixes were shown by Hila Gonen and Yoav Goldberg (NAACL 2019) in "Lipstick on a Pig": after applying popular debiasing methods to embeddings, gender information could still be recovered from the geometry of the vectors. Removing one measurable signal does not remove the underlying pattern.

Measuring bias in practice

Disaggregated evaluation: report performance separately for relevant groups, dialects, names or demographic attributes, as model cards recommend (see datasheets and model cards).
Template and counterfactual tests: change only a name, pronoun or dialect feature and check whether the output changes.
Task-specific benchmarks such as WinoBias or WinoMT, used with awareness of what they do and do not measure.
Qualitative review by people from the affected groups, which catches harms no benchmark anticipated.

Regulation

In the EU, the AI Act requires providers of high-risk AI systems to examine training, validation and testing data for possible biases likely to affect health, safety or fundamental rights or to lead to discrimination, and to take measures to detect, prevent and mitigate them (Article 10). Processing of special categories of personal data for this purpose is allowed only under strict conditions, which links bias work to data protection law, discussed in our article on GDPR and text data.

Steps for a team building an NLP system

Name the harms that matter in your application and who would bear them.
Document your data: whose language is in it, whose is missing, who annotated it.
Test with counterfactuals and disaggregated metrics before deployment, not after complaints.
Involve affected people in evaluation and give users a way to report problems.
Re-test after every model update; behaviour can shift with new data or versions.

Sources

Bolukbasi, Chang, Zou, Saligrama and Kalai, "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings", NeurIPS 2016.
Caliskan, Bryson and Narayanan, "Semantics derived automatically from language corpora contain human-like biases", Science 356, 2017.
Zhao and colleagues, "Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods", NAACL 2018; Rudinger and colleagues, "Gender Bias in Coreference Resolution", NAACL 2018.
Stanovsky, Smith and Zettlemoyer, "Evaluating Gender Bias in Machine Translation", ACL 2019.
Sap, Card, Gabriel, Choi and Smith, "The Risk of Racial Bias in Hate Speech Detection", ACL 2019.
Blodgett, Barocas, Daumé III and Wallach, "Language (Technology) is Power: A Critical Survey of 'Bias' in NLP", ACL 2020.
Gonen and Goldberg, "Lipstick on a Pig", NAACL 2019.
Bender, Gebru, McMillan-Major and Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?", FAccT 2021.
NIST Special Publication 1270, "Towards a Standard for Identifying and Managing Bias in Artificial Intelligence", March 2022.
Regulation (EU) 2024/1689 (AI Act), Article 10.