The Human Record: What We Give the Machine

The most popular picture of artificial intelligence right now is a monster wearing a mask. The Shoggoth with a smiley face began as a joke posted on Twitter at the end of 2022 and spread because it says something true in one glance: a friendly conversational surface sits on top of a system nobody fully understands. I find the image potent. I also find it incomplete. It tells you to be uneasy and stops there. It has nothing to say about where the system came from, what it is made of, or who is responsible for it.

I want to tell a different creation story. It needs to hold wonder, because the thing is wonderful. It needs to hold inheritance, because a model is downstream of everything people bothered to write down. It needs to hold physical reality, because none of this happens without mineral, electricity and machines. And it needs to hold responsibility, because every step is a choice someone made. The working metaphor for the series is making sand think: silicon refined from quartz becomes chips, chips run arithmetic, and the arithmetic learns a structure from human expression. That is a metaphor, and I will keep calling it one. It is not a claim that silicon is conscious or that anyone has settled what thinking is. It is a way of putting the pieces in order so they can be looked at honestly.

This series is seven essays, each paired with a painting in progress. The paintings are large horizontal panels in the tradition of didactic history painting, where composition and gesture carry an argument. The essays carry the same argument in prose. This first one is about the material that comes before any machine: the record.

What a model is given

A large language model learns from text. That sentence hides almost everything that matters. Which text? Chosen by whom? In what form?

The raw inheritance is immense. Books digitized over two decades. Scientific papers and their preprints. Code published in public repositories. Encyclopedias written by volunteers. News archives, forums, product reviews, transcribed speech, government filings, song lyrics, recipes, the comment sections under all of it. For a multimodal model, add photographs and their captions, diagrams, screenshots, video frames. Most of this exists because someone, somewhere, decided it was worth recording and then worth putting online.

What reaches a model is a fraction of that, and the fraction is not chosen at random. A training corpus is assembled. Web crawls are the usual backbone, and the largest public one, Common Crawl, has gathered petabytes of pages since 2008. From there the material passes through a sequence of decisions that teams of people make and encode in software:

• Language filtering. Pages are classified by language and often weighted toward English, which shapes who the model speaks well for.
• Quality filtering. Classifiers trained to recognize “good” text, by some definition, score every document and drop the low end. The definition is a human judgment with a model standing in for it.
• Deduplication. Repeated passages are removed so the model does not memorize boilerplate. This also decides which of several near-identical copies survives.
• Content filtering. Pages judged toxic, adult or spam are removed, with all the false positives and blind spots that implies.
• Mixing. Different sources are weighted. A page from a reference work might be shown more often than a page from a random forum, or less.

Open datasets document these steps in detail. The Pile, published in 2020, listed its twenty-two sources and their weights. Dolma, released in 2023 and documented in a 2024 paper, published its full filtering pipeline. For the models most people use, the equivalent decisions are mostly private. A model’s maker may describe the categories of data it used without saying what was in or out. So the honest statement is this: a model inherits a selected, processed, partly undocumented portion of the human record, and the selection says as much about its makers as the record says about us.

Access is not permission

Much of what is on the web can be read by a program. That is a technical fact. Whether it may be used to train a commercial model is a legal and ethical question, and it is currently being argued in courts in several countries. Authors, newspapers and artists have sued over training on their work. Some data is licensed, some is in the public domain, some is scraped under a claim of fair use, and the lines are not settled. I am not going to pretend otherwise, and the essays that follow will not treat the question as closed.

This matters to the creation story because it changes who is in the picture. The record is not a free natural resource like sunlight. It is the work of people, most of whom were never asked. A story that leaves them out is not complete enough to be truthful.

Following one sentence

To make the mechanism concrete, follow a single sentence from a regional newspaper printed in 1921.

The page was microfilmed in the 1970s, scanned by a library in the 2000s, and run through optical character recognition that misread a few letters. The text landed on a public archive site. A crawler fetched the page and stored it. During dataset construction, a language classifier marked it English. A quality classifier gave it a middling score, high enough to keep. Deduplication found that a genealogy site had copied the same article and dropped one copy. The surviving text was broken into tokens, which the next essay explains, and during training the model saw that sentence once, perhaps twice.

What did the sentence leave behind? No file. No entry. Each time the model saw it, the training process nudged a very large number of parameters by very small amounts in the direction that would have made the sentence slightly less surprising. The same parameters were nudged by every other sentence in the corpus. The 1921 sentence survives, if it survives at all, as a faint statistical pressure spread across the whole system, blended with everything else.

This is the single most important thing to understand about the record and the model. The corpus is not a library the model consults when you ask it something. After training, the corpus is gone. What remains is a learned structure, numbers that encode regularities in how people have written. When a chatbot does quote a document accurately, one of two things is happening: either the passage appeared often enough to be memorized, which is a known and studied phenomenon, or the application retrieved the document from a separate store and placed it in front of the model as part of the prompt. That second mechanism, retrieval, is real and useful and is a different thing from training. Keeping them apart will matter again when we reach inference.

What the record leaves out

Every archive has edges, and a training corpus has sharper ones than most. Languages with little online text are thin. Oral traditions, which hold a great deal of human knowledge, are nearly absent unless someone transcribed them. Work behind paywalls, inside institutions, or on paper that nobody scanned is missing. The recent past is overrepresented because the internet is recent. The voices that posted most are the voices the model learned most from.

None of this makes the model useless. It makes it specific. A model is a portrait of a particular slice of humanity at a particular moment, filtered through a particular set of choices. When it speaks with confidence about the world, it is speaking from that slice. That is also true of any one person, and it is a reason for humility rather than alarm.

The painting

The first panel in the series shows the act this essay is about: people deciding what to keep. Not a heap of books and not an invented ancient library, but the ordinary, consequential work of selection. Someone scanning a page. Someone writing a rule. The light of the Bay behind them, because so much of this work has been done in and around San Francisco. Historical figures appear where they belong in the record, in period dress, and are named in the key beside the painting. The gathering is allegorical. No such meeting took place.

A creation story directs attention. The Shoggoth directs it toward fear of the thing. This story begins by directing it toward the people who made the record and the people who chose from it. The next essay follows the chosen material into the machine: how sand becomes silicon, and how silicon, running arithmetic for months, arrives at a set of numbers that can continue a sentence.