Models: A Structure of Learned Possibilities

Ask a language model about Adventures of Huckleberry Finn and it will discuss the raft, the river, Jim, the ending that Hemingway called cheating. It may quote the opening line. The natural question, and the one I kept asking as I learned how these systems work, is where the novel is. Not the text, which the first essay explained is gone after training. The understanding. When the model discusses the book, what is it consulting?

The short answer is that it is consulting nothing. There is no place in the model where the novel lives. The longer answer is this essay, and it begins with what a model actually is when you hold one.

Three things in a file

When I download a model to run at home, I receive a file, or a set of files, and they contain three things.

The first is the architecture: the shape of the arithmetic, the transformer the third essay described, specified as a handful of settings. How many layers, how wide each one is, how many attention heads, what vocabulary. This takes a few kilobytes.

The second is the parameters: the final value of every dial that training turned, stored as a long sequence of numbers in the order the architecture expects. This is nearly all of the file.

The third is the configuration that surrounds them: the tokenizer that turns text into the model’s vocabulary, and the settings for how generation should run. Without the tokenizer the parameters cannot be addressed; without the architecture they are a meaningless list.

Together these are the model. Copy the files and you have copied the model, exactly, with no loss. This is what makes the sixth essay’s subject, proliferation, possible: a trained system is an artifact, and artifacts can be duplicated at the cost of storage.

Counting the dials

A model described as seventy billion parameters holds roughly seventy billion learned numbers, under the counting convention its maker uses. The convention matters more than it sounds.

Most of those numbers are weights: the entries of the large matrices in each layer’s attention and feed-forward blocks, the matrices that multiply the representations as they pass through. But parameters also include the embedding table from the encoder essay, the small vectors that shift and scale each layer’s output, and in some designs other learned quantities. “Weights” is often used as a synonym for “parameters,” and in loose conversation it does no harm. In a precise sentence, weights are a kind of parameter.

A second convention has become important recently. Some models route each token through only a portion of their parameters, a design called mixture of experts. Such a model may hold several hundred billion parameters in total while using a few tens of billions for any one token. Both numbers are honest, and they answer different questions: the first says how much the model knows, the second how much work each token costs. Comparing a dense model’s total with a mixture model’s total, or either with the other’s active count, is the most common way to be misled by a parameter figure.

Where the novel is

Now the question. The model has seen Huckleberry Finn, probably, since the book has been in the public domain for a century and sits in every free library on the web, though how many copies it saw and whether it saw the whole text are not things anyone outside the lab can check. What did the training leave behind?

Not a copy. The second essay showed that each passage nudges every parameter a little, and the nudges from the novel are blended with the nudges from everything else. Interpretability researchers, who study trained models from the inside, have found that this blending is not merely a figure of speech. Individual components inside a model respond to many unrelated things at once, and a single idea is spread across many components, a condition they call superposition. There is no Huckleberry Finn layer, no neuron for Jim, no shelf.

What there is instead is a direction. In 2024, researchers at Anthropic reported finding, inside a production model, Claude 3 Sonnet, a pattern of activation that fired when the text concerned the Golden Gate Bridge, in several languages and in images of the bridge. When they artificially amplified it, the model began working the bridge into everything, including its own description of itself. The “concept” of the bridge was real and locatable, and it was a direction across many numbers rather than a place in one. Something similar is presumably true of Twain’s novel: a set of directions the model’s activations move in when the river and the raft come up, shaped by every discussion of the book in the corpus as well as the book itself.

That is where the understanding lives, if “lives” is the word. In the parameters, as a disposition. In the activations, as a pattern that appears when the right context arrives. The parameters are fixed. The pattern is computed each time, which is why the model’s account of the novel can be faithful in one conversation and wrong in the next, and why it can quote the famous first line and invent a scene that is not in the book with the same confidence.

Making the file smaller

Here is the part that lets a model like this run on a desk. A seventy-billion-parameter model stored at sixteen bits per number is about a hundred and forty gigabytes, more than the memory of any machine I own. Quantization is the fix, and it is widely misunderstood.

Quantizing a model means storing its numbers at lower precision. A common approach takes each small group of weights, finds a scale that fits them, and stores each weight as a small integer, perhaps four bits, plus the shared scale. On loading, the integers are multiplied back out. Each weight is now approximate, off by a small amount, and the approximations are everywhere. The file is about a quarter of its size.

What is lost is precision, spread evenly across the whole structure, like a slight blur over a photograph. What is not lost is a quarter of the knowledge. There is no sense in which quantizing to four bits removes three quarters of what the model learned; the structure of directions survives with a little noise added. Measured on standard tasks, good four-bit quantization of a large model costs a small fraction of accuracy, and the tradeoff depends on the method and the hardware. Some quantized models run faster, because memory bandwidth is the bottleneck and there is less to move; some run slower if the hardware has to do extra work unpacking the integers.

All of this concerns the weights. During a conversation, the model also produces activations and keeps a cache of past computation, and those can be quantized too, by different methods with their own tradeoffs. The next essay covers what that cache is. The point here is only that “a quantized model” is a statement about the stored parameters, and nothing else.

A cast, not a brain and not a library

What, then, is a model? The popular images all miss. It is not a brain, because nothing in it grows, remembers a conversation or changes with use; the dials stay put. It is not a database, because there is no row for Huckleberry Finn to look up. It is not a quantum computer or a mystery box.

The image I have settled on is a cast. A sculptor presses clay against a form and the clay takes its shape, every curve of the original recorded as a curve in the clay, none of the original’s substance transferred. Training presses the model against the human record and the parameters take its shape: the regularities, the likelihoods, the turns a sentence tends to take. The result is a structure of learned possibilities, holding not the record but the shape of what the record made likely. It is a thin thing next to a library and an extraordinary thing next to any previous machine. And like a cast it can be copied, which is where the story turns from one artifact to millions.

The painting

The fifth panel shows the artifact as an object. At its centre, in a glass case, a structure of learned relationships: a lattice, crystalline, lit from the window, large enough to read as a thing of consequence and abstract enough to make no claim about architecture. Around it, the copies: trays of identical drives, being handled, shelved, carried out. A figure at the edge reads a page of the structure’s numbers in a bound volume, which is what a parameter file looks like if you print it, and the volume is enormous and says nothing to the eye. The key names the historical witness, John Hopfield, whose 1982 work showed how a network could hold memories as patterns distributed across the whole, the idea this essay has been circling.

The next essay takes a copy off the shelf and asks it a question.