Ethics & data

Web-Scraped Corpora: Copyright, Consent and Opt-Outs in NLP

Ethics & data · Published 2026-10-26

A server rack packed with stacked machines, blue status displays and bundles of white network cables
Servers in the NERSC data centre, Berkeley, California. Photo: Derrick Coetzee, CC0.

Most large language models learned to write from text copied off the public web. The people who wrote that text were rarely asked. Whether this is lawful, fair and sustainable has become one of the central questions of language technology. This guide explains how web corpora are built and what European law and research ethics say about them.

This article describes the legal framework in general terms as of October 2026; it is not legal advice. Court cases on AI training are under way in several countries and the law is still developing.

How web corpora are made

Common Crawl, a non-profit organisation founded in 2007, has crawled the web regularly since 2008 and publishes its archives free of charge. Most large web-based training sets start from its snapshots and then filter them: removing duplicate pages, non-text and boilerplate, identifying languages, and applying quality and safety filters.

Those filters shape what models learn. Jesse Dodge and colleagues examined C4, a cleaned subset of Common Crawl created for Google's T5 model, in "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus" (EMNLP 2021). They found unexpected sources such as patents and machine-generated text, examples from benchmark test sets, and evidence that the blocklist used to remove "bad" words disproportionately removed text by and about minority groups, including non-offensive content about LGBTQ+ identities and text in African American English. Cleaning, in other words, is not neutral, a point developed in our article on bias in NLP.

Copyright: the EU framework

Most web text is protected by copyright, and copying it to build a corpus is an act of reproduction. The Directive on Copyright in the Digital Single Market (EU) 2019/790 created two exceptions for text and data mining:

How these exceptions apply to generative AI is being tested in court. In a first-instance judgment of 27 September 2024, the Regional Court of Hamburg dismissed a photographer's claim against LAION, a non-profit that had compiled a large image-text dataset from web data, finding that its downloading of an image for dataset creation was covered by the German provision implementing the scientific research exception. On 10 December 2025 the Hamburg Higher Regional Court dismissed the photographer's appeal (case 5 U 104/24). Other cases, in Europe and in the United States, concern commercial developers and different facts; no general rule can be drawn from a single dispute.

Distributing a corpus is a separate question from mining it: the exceptions cover the copies needed for analysis, not the right to republish the texts. That is why many research datasets are shared as lists of URLs or document identifiers, as explained in our guide to dataset licences.

Opting out: robots.txt and beyond

The Robots Exclusion Protocol, the robots.txt file proposed in 1994, lets a website tell crawlers which parts they may visit. It was standardised by the Internet Engineering Task Force as RFC 9309 in September 2022. It is a request, not access control: well-behaved crawlers follow it, others may not. Since 2023 several AI developers have published distinct crawler names (user agents), allowing sites to block AI training crawlers while still allowing search indexing.

Whether a robots.txt entry, a clause in a site's terms, or a statement in natural language counts as a valid machine-readable reservation under Article 4 is part of the legal debate. Researchers from the Data Provenance Initiative, in "Consent in Crisis" (2024), measured a rapid rise in such restrictions among the websites most used in training corpora during 2023–2024, which shows that publishers are using these signals and that the open web available for training is shrinking.

The AI Act's transparency duties

The EU AI Act (Regulation (EU) 2024/1689) requires providers of general-purpose AI models, from 2 August 2025, to put in place a policy to comply with EU copyright law, in particular to identify and respect reservations of rights under Article 4 of the 2019 directive, and to publish a sufficiently detailed summary of the content used for training, following a template the AI Office published in July 2025 (Article 53(1)(c) and (d)). A voluntary General-Purpose AI Code of Practice, published in July 2025, includes a copyright chapter on respecting crawler exclusions. These duties apply to providers placing models on the EU market, regardless of where training took place.

Consent, data protection and fairness

Copyright protects authors' economic and moral rights; it does not answer every ethical question. People who posted on a forum ten years ago did not anticipate their words training a commercial model, even where the law permits it. Web text also contains personal data, and its use is governed by the GDPR regardless of copyright, as explained in our article on GDPR and text data. And communities whose languages are scraped may reasonably ask for a say in how their language data is used, an issue raised by Indigenous data sovereignty movements.

Researchers have proposed responses ranging from documentation (datasheets for every corpus) to opt-in data collections, licensing agreements with publishers, and datasets built only from openly licensed or public-domain text.

Practical steps when building a web corpus

  1. Respect crawler exclusions and record the robots.txt and terms in force at crawl time.
  2. Know which exception you rely on (research or general mining) and whether your organisation qualifies.
  3. Filter thoughtfully, then audit what the filters removed, not only what they kept.
  4. Handle personal data under data protection rules: minimise, pseudonymise, and test models for memorisation.
  5. Share responsibly: prefer identifiers and rebuild scripts to redistributing full texts.
  6. Document sources, dates, filters and opt-out handling in a public datasheet.

Sources

Read next

More in Ethics & data
A rusty metal hasp and padlock fastening a weathered red wooden doorEthics & data

GDPR and text data

Names, emails and health details hide in every corpus. How the GDPR applies to NLP data and models, why text anonymisation is hard, and what to do.

Two women programmers set up the wiring panels of the room-sized ENIAC computer in a black-and-white photographEthics & data

Bias in NLP

Stereotypes in embeddings, gendered mistranslations, dialects flagged as offensive: what research has shown about bias in language technology and what helps.

Rows upon rows of stacked boxes of punched record cards filling a large warehouse floorOpen research

Dataset licences for NLP

CC BY, ShareAlike, NonCommercial, ODbL, LDC agreements and the EU's text and data mining exceptions, explained for people who build and share corpora.