The Pile's documentation made its contested data visible

An 825-gigabyte corpus with a public ingredient list

EleutherAI posted 'The Pile: An 800GB Dataset of Diverse Text for Language Modeling' to arXiv on 31 December 2020. The paper describes an English text corpus of 825 gibibytes assembled from 22 named component datasets, spanning books, web text, code, academic papers and other sources, released under a Creative Commons Attribution 4.0 licence. The project's own site, pile.eleuther.ai, lists the same 22 components with their relative sizes and describes the whole as intended to give models trained on it broader cross-domain generalisation than a single web-crawl source would.

What documentation was meant to do, and did

The paper states the authors carried out 'an in-depth exploratory analysis' of the corpus and documented 'potentially concerning aspects of the data for prospective users', treating disclosure itself as a contribution rather than a side effect. That distinguishes the Pile from web-scraped corpora with no published component list: a researcher can point to a specific named dataset, such as its PubMed Central or FreeLaw components, rather than describing training data only in aggregate. The paper also reports that models trained on the Pile improved over models trained on undocumented web-crawl baselines on downstream evaluations, a comparison against those specific baselines under the paper's own test setup, not against every possible training corpus.

Why documentation made a later problem visible instead of hiding it

One named component, Books3, was a large collection of digitised books that made up roughly 12 percent of the corpus by weight in the original release. The replication code repository on GitHub still lists Books3 in its component table but marks it, without further explanation in that document, as 'Download temporarily unavailable'. Because the Pile's components were named and documented rather than left undisclosed, a change to one component's availability is a visible, checkable fact rather than something a downstream user would have to infer from model behaviour alone; the repository does not state why the component became unavailable, and that reason should not be assumed from outside sources.

Questions to carry into your own evaluation

• Is a claim about a model's training data based on the corpus's own documentation, or inferred indirectly from the model's outputs?
• If a named component is now unavailable or altered, does that change what can be said about a model trained on an earlier version of the corpus?
• Does an 'open' or 'documented' dataset claim mean the components are licensed for reuse, or only that they are named and described?

The Pile's contribution was disclosure: a named, described list of components that let later researchers examine what a model was trained on, component by component. That same disclosure is why a change to one component, such as Books3 becoming unavailable, is a documented fact rather than a rumour, which is closer to the exception for training data generally than to the norm.