Dataset Licences for NLP: Creative Commons, Open Data Commons and the EU Rules
A corpus is only as reusable as its licence. Many NLP datasets are shared with unclear, inconsistent or missing terms, which can block publication, commercial use or even replication years later. This guide explains the licences you will meet most often, the European rules that sit underneath them, and how to license your own data.
This article explains general principles; it is not legal advice. Licences and laws differ by country, and the application of copyright to machine learning is still being settled in courts and legislatures.
Three layers of rights
A typical NLP dataset bundles several things that may carry different rights:
- The underlying texts: news articles, books, posts, transcripts. Each author or publisher normally holds copyright.
- Your additions: annotations, labels, alignments, translations. These may be protected as your own work.
- The collection as a whole: in the EU, the maker of a database who made a substantial investment in obtaining, verifying or presenting its contents has a special "sui generis" right for 15 years under Directive 96/9/EC, separate from copyright.
A licence you choose for the dataset can only cover the rights you hold. If the texts belong to someone else, you need their licence terms to allow redistribution, or you distribute only your annotations with pointers (such as URLs or document identifiers) to the original texts, which users obtain themselves.
Creative Commons, element by element
| Element | Meaning | Effect on NLP reuse |
|---|---|---|
| BY | Credit the creator | Easy to honour; keep a source list |
| SA (ShareAlike) | Adaptations must use the same licence | Annotated versions must stay under the same terms |
| NC (NonCommercial) | No use primarily intended for commercial advantage | Blocks company use; grey zone for funded research |
| ND (NoDerivatives) | No adapted versions may be shared | Usually prevents sharing annotated or cleaned versions |
Version 4.0 of the licences, published in 2013, explicitly covers EU database rights as well as copyright, which makes it better suited to datasets than earlier versions. CC0 is a separate tool: a waiver by which the rights holder gives up their rights as far as the law permits, placing the work effectively in the public domain. Wikidata, for example, publishes its structured data under CC0, while Wikipedia's article text is under a ShareAlike licence, which is why datasets built from Wikipedia text generally have to be shared under ShareAlike terms too.
The NonCommercial clause causes the most confusion. Whether a university project funded by a company, or a start-up's research prototype, counts as "commercial" is not always clear, and Creative Commons itself says the answer depends on the specific use. If you want your corpus to be used widely, including for products, avoid NC; if you choose it, explain in your documentation what you consider acceptable.
Other licences you will meet
- Open Data Commons licences, created for databases: ODC-BY (attribution) and ODbL (attribution and share-alike, used by OpenStreetMap).
- Software licences such as MIT or Apache 2.0 are sometimes applied to datasets on code-hosting platforms. They were not written for data and may not address database rights; prefer a data licence.
- Consortium agreements. The Linguistic Data Consortium, founded in 1992 and hosted at the University of Pennsylvania, and the European Language Resources Association (ELRA), founded in 1995, distribute many classic corpora under signed agreements that often restrict redistribution and sometimes commercial use. Results based on them are publishable, but the data itself cannot be reposted.
- Terms of service. Text collected from websites or platform APIs may be governed by contractual terms in addition to copyright. Platforms have changed their API terms over time, so record the terms in force when you collected the data.
The EU text and data mining exceptions
The EU Directive on Copyright in the Digital Single Market (2019/790) introduced two exceptions for text and data mining, which member states had to transpose by June 2021:
- Article 3 allows research organisations and cultural heritage institutions to carry out text and data mining for scientific research on works they have lawful access to. Rights holders cannot opt out of it. Copies may be retained for research purposes, including verification of results, with appropriate security.
- Article 4 allows anyone to mine lawfully accessible works for any purpose, unless the rights holder has expressly reserved this right, "in an appropriate manner, such as machine-readable means" for content made publicly available online.
These exceptions cover the act of analysis and the copies it requires. They do not grant a general right to publish or redistribute the mined texts as a corpus. How they apply to training generative models is discussed in our article on web-scraped corpora, copyright and consent.
Licences are often wrong on the label
Licence information tends to degrade as datasets are copied, merged and re-uploaded. The Data Provenance Initiative, a volunteer research collective, audited well over a thousand text datasets used to fine-tune language models and reported in Nature Machine Intelligence (2024) that licences on popular hosting platforms were frequently missing, or listed as more permissive than the terms set by the original authors. Anyone building on aggregated collections should trace each component back to its original source.
Licensing your own dataset: a checklist
- List every source with its licence and the date you obtained it.
- Find the most restrictive term among your sources: it constrains what you can offer.
- Choose a standard licence (CC BY 4.0, CC0 or ODC-BY for maximum reuse); avoid writing your own.
- If you cannot redistribute the texts, release annotations with identifiers and scripts that rebuild the corpus.
- Check personal data separately: a licence does not make personal data free to use; data protection rules still apply.
- Write it down in a datasheet or data statement (see datasheets, data statements and model cards), and include a licence file in every copy.
Clear licensing is part of making research reusable and checkable, as discussed in our guides to open language models and reproducibility.
Sources
- Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market, Articles 2–4 and 26.
- Directive 96/9/EC on the legal protection of databases, Articles 7 and 10.
- Creative Commons, licence texts version 4.0 and CC0 1.0 Universal (creativecommons.org).
- Open Data Commons, ODC-BY 1.0 and ODbL 1.0 (opendatacommons.org).
- Longpre and colleagues, "A large-scale audit of dataset licensing and attribution in AI", Nature Machine Intelligence, 2024.
- Linguistic Data Consortium, University of Pennsylvania; European Language Resources Association.
Open researchWhat makes a language model open?
Open weights are not open source. The parts of a model release, the OSI definition, restrictive licences and what the EU AI Act says.
Open researchDatasheets, data statements and model cards
The three documentation formats every NLP researcher should know, what each asks, and how they connect to EU transparency rules.
Ethics & dataWeb-scraped corpora and consent
Common Crawl, robots.txt, the EU's text and data mining opt-out and the AI Act: the legal and ethical questions behind training data scraped from the web.