---
title: "Language Models from the Ground Up"
book: language-models-from-the-ground-up
edition: 2026.10.3
verified: 2026-10-25
edition_code: 1W0R-Q8X3
canonical: https://rollingedition.com/language-models-from-the-ground-up
license: CC BY 4.0
attribution: "Language Models from the Ground Up, Edition 2026.10.3, examples passed Oct 25, 2026. By Rolling Edition, https://rollingedition.com/language-models-from-the-ground-up. Text CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.10.3. Examples passed Oct 25, 2026. 86 of 86 examples pass. Written with AI. Verified by people.
To cite a section, name Edition 2026.10.3 and link to the section's permanent link: https://rollingedition.com/language-models-from-the-ground-up/s/{section_id}
# Predicting the next word
By the end of this chapter you can build a complete language model from counts, use it to suggest the next word as a shopper types, generate text with it, and measure with one number how well it predicts text it has never seen. Every later chapter replaces one part of this model with a better one, and from [chapter 3](#counting-further-back) on, measures each change the same way.
A **language model** is a program that assigns a probability to every word that could come next, given the words so far. That one ability covers a lot. Suggesting how to finish a search query means picking a likely next word. Writing a sentence means picking a next word, adding it, and picking again. And a model that predicts well has learned something about the text it was trained on, so how well it predicts is how you measure it. The large language models behind chat assistants do the same thing at a vastly larger scale: given the text so far, they assign a probability to each possible next piece of text. This book builds that machinery one part at a time, small enough to run on a laptop, so you can see what each part does.
This book follows one case study throughout: an online furniture and home-goods store, which we call the store. The store's search box needs autocomplete, so that a shopper who types `coffee` sees `table` offered as the next word, and the store needs short product descriptions written and checked. Both are language-model problems, and the store has the text to learn them from: the names and descriptions of every product in its catalog. In this chapter you count which word follows which in the product names, turn the counts into probabilities, and put them to work. The model you build here has a serious flaw, and the chapter ends by measuring it.
## Load the store's text {#load-the-store-text}
Before a model can learn which words follow which, it needs text to learn from, and a way to tell whether what it learned carries over to text it hasn't seen. The second need matters as much as the first. A model can predict the text it learned from perfectly by memorizing it, which says nothing about how it will do on the next product the store adds. So the text is split once, before anything learns from it, into three parts: a **training split** that models learn from, a **validation split** for comparing models while you build them, and a **test split** kept for final reports. Each product belongs to exactly one split, so no name or description a model is scored on appears in what it learned from.
The store's text, which every example in this book learns from, is real. It comes from WANDS, a public product-search dataset that Wayfair published with a paper at ECIR 2022 ([Chen et al., 2022](https://easychair.org/publications/preprint/wm4R)), under the MIT licence ([licence](https://github.com/wayfair/WANDS/blob/main/LICENSE)). The script `prep/wands_text.py` prepares it, and its data card, `examples/data/wands-text/README.md`, says how: the text is already lowercased, with spaces around punctuation, descriptions are cut at a sentence end to keep the file small, and products whose text names the store itself are left out, so no model in this book learns to write the store's name. The examples read the text through `lib.store`, one of the book's shared modules: short files in `examples/lib/` that hold setup code many examples need, which you can open and read.
Example `store-text` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.store import load_products, ordinary_words, words
products = load_products()
print("products:", f"{len(products):,}")
for split in ["train", "validation", "test"]:
print(f"{split} products:", f"{(products['split'] == split).sum():,}")
train = products[products["split"] == "train"]
print("words in training names:", f"{sum(len(words(name)) for name in train['name']):,}")
print("different words in training names:", f"{len({w for name in train['name'] for w in words(name)}):,}")
print()
common = ordinary_words(300) # words in 300 descriptions or more: plainly not names
for kind in ["coffee table", "bar stool", "area rug", "bathroom vanity"]: # one name of each kind, as examples
print(next(name for name in train["name"] if kind in name and all(w in common or not w.isalpha() for w in words(name))))
```
Output:
```text
products: 42,815
train products: 34,252
validation products: 4,282
test products: 4,281
words in training names: 231,048
different words in training names: 25,907
solid wood coffee table
25 '' bar stool
hand-knotted wool/silk teal/plum area rug
36 '' single bathroom vanity
```
The store's text covers 42,815 products: 34,252 in the training split, 4,282 in the validation split and 4,281 in the test split. This chapter learns from product names only, which are short and repetitive enough for counting to work. The training names hold 231,048 words, of 25,907 different kinds. That second number is large for a catalog of furniture: many names start with a product line's name, a word that appears in a handful of products and nowhere else. This book doesn't print those names: where an example's output would show one, it prints `[name]`, which marks any word used in fewer than 30 of the training descriptions, most often a product line's name. Those rare words will matter by the end of the chapter.
**Working rule:** Split your text into training, validation and test parts before any model sees it, and split by document (here, by product), so near-duplicates of one item never straddle two parts. Compare models on validation text while you work, and touch the test text only for the final report. Learn from the shortest, most regular text you have first; this chapter starts with names, and descriptions come later in the book.
## Count which word follows which {#count-word-pairs}
With the training names loaded, a simple model of "what comes next" is a table of how often each word followed each other word. A **bigram** is a pair of adjacent words, such as `coffee table`. Count every bigram in the training names, and you know, for any word, which words came after it and how often. Claude Shannon used exactly this idea in the paper that founded information theory, approximating English by choosing each word according to how often it follows the one before ([Shannon, 1948](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x)).
Two details make the counts complete. A name's first word follows nothing, and its last word is followed by nothing, but both are worth predicting: the first word decides what kind of product a shopper is typing, and knowing where a name ends lets a model stop. So each name is padded with two markers, `` at the start and `` at the end, which count as words. The pair ` coffee` then records that a name started with `coffee`, and `table ` records that a name ended with `table`.
For the store, start with one word whose next word matters to the search box: `coffee`. The example counts every pair in the training names, then looks at what followed `coffee`:
Example `count-next-words` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib.store import load_products, words
names = load_products("train")["name"]
pairs = Counter()
for name in names:
tokens = [""] + words(name) + [""] # mark where a name starts and ends
pairs.update(zip(tokens, tokens[1:])) # each word with the word that follows it
print("word pairs:", f"{sum(pairs.values()):,}")
print("different pairs:", f"{len(pairs):,}")
after_coffee = {following: n for (word, following), n in pairs.items() if word == "coffee"}
print("words after coffee:", f"{sum(after_coffee.values()):,}")
for following, n in sorted(after_coffee.items(), key=lambda item: -item[1])[:4]:
print(f" coffee {following:<6} {n:>6,}")
```
Output:
```text
word pairs: 265,300
different pairs: 84,529
words after coffee: 1,260
coffee table 1,129
coffee mug 45
coffee , 28
coffee jar 13
```
The training names hold 265,300 word pairs, of 84,529 different kinds, and `coffee` is followed by a word 1,260 times. One word dominates: almost every `coffee` in the store's names is a `coffee table`. A comma comes third, because the store's text treats punctuation as words of its own. That's a choice about tokens, the units a model counts, and [chapter 2](#tokens) shows how much it matters.
The table of counts is the whole model. It has no rules about grammar and no knowledge of furniture; it has seen `coffee table` many times and `coffee lamp` never, and that is all it knows. The shared module `lib.bigrams` holds this same counting code, as the class `Bigrams`, so later examples can build the model in one line.
**Working rule:** Count pairs in the training split only, and pad every text with start and end markers so the model learns how texts begin and end. Keep the raw counts, not only the probabilities you'll derive from them: the counts tell you how much evidence each probability rests on, and the next sections need them.
## Turn counts into probabilities {#counts-to-probabilities}
Counts say what happened; a language model needs probabilities, which say how likely each next word is. The step from one to the other is division. The probability of a word given the word before it is the number of times the pair occurred, divided by the number of times the first word occurred with anything after it. In symbols, writing `w` for the previous word and `v` for the next one:
P(v | w) = count(w v) / count(w)
Read it as "the probability of `v`, given `w`". Estimating probabilities this way, from the fraction of times something happened, is called **maximum likelihood estimation**: of all the probability tables you could choose, it's the one that makes the training text likeliest ([Jurafsky and Martin, chapter 3](https://web.stanford.edu/~jurafsky/slp3/3.pdf)).
For the store, apply the formula to the word `coffee` and four candidates for the next word:
Example `next-word-probabilities` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.bigrams import Bigrams
from lib.store import load_products
model = Bigrams(load_products("train")["name"])
after_coffee, total = model.counts["coffee"], model.totals["coffee"]
for following in ["table", "mug", "jar", "lamp"]:
print(f"P({following} | coffee): {after_coffee[following] / total:.3f} ({after_coffee[following]} of {total})")
print("sum over every word after coffee:", round(sum(n / total for n in after_coffee.values()), 6))
```
Output:
```text
P(table | coffee): 0.896 (1129 of 1260)
P(mug | coffee): 0.036 (45 of 1260)
P(jar | coffee): 0.010 (13 of 1260)
P(lamp | coffee): 0.000 (0 of 1260)
sum over every word after coffee: 1.0
```
The probability of `table` after `coffee` is 0.896 (1129 of 1260), of `mug` 0.036 (45 of 1260) and of `jar` 0.010 (13 of 1260). The probabilities of every word that followed `coffee` add up to 1.0, as a probability distribution must: whatever comes next, it's one of these words.
The fourth line is the one to remember. `coffee lamp` never occurs in the training names, so its probability is 0.000 (0 of 1260). The model doesn't say that a coffee lamp is unlikely; it says a coffee lamp is impossible. For a furniture store that's a reasonable bet, but the same zero applies to every pair the model hasn't seen, including perfectly ordinary ones. That's the flaw this chapter ends on.
**Working rule:** Derive probabilities from counts by division, and check that each word's probabilities sum to one; a sum that doesn't is a bug in the counting. Treat a probability as only as good as the count under it: 0.9 from 1,000 pairs is evidence, while 1.0 from a single pair is an anecdote.
## Suggest the next word as a shopper types {#suggest-next-words}
A probability table answers the store's first question directly. Autocomplete offers the likeliest next words, so a shopper who has typed `coffee` sees `table` first. With the bigram model, the suggestions for any typed text are the words with the highest probability after its last word. The bigram model's class, `Bigrams`, from [count which word follows which](#count-word-pairs), has a method, `next_words`, that returns them, likeliest first.
For the store, type five prefixes a shopper might enter, and ask for three suggestions each:
Example `autocomplete` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.bigrams import Bigrams
from lib.store import load_products
model = Bigrams(load_products("train")["name"])
for typed in ["coffee", "bar", "kitchen", "outdoor", "patio bar"]:
last_word = typed.split()[-1] # a bigram model sees only the last word
suggestions = model.next_words(last_word, k=3)
print(f"{typed:<10} -> " + ", ".join(f"'{word}' {p:.2f}" for word, p in suggestions))
names = load_products("train")["name"]
print("training names containing 'outdoor door':", sum("outdoor door" in n for n in names))
print("of them, followed by 'mat':", sum("outdoor door mat" in n for n in names))
```
Output:
```text
coffee -> 'table' 0.90, 'mug' 0.04, ',' 0.02
bar -> 'stool' 0.32, 'pull' 0.18, '&' 0.15
kitchen -> 'sink' 0.17, 'island' 0.16, 'canister' 0.12
outdoor -> 'door' 0.22, 'patio' 0.07, 'area' 0.06
patio bar -> 'stool' 0.32, 'pull' 0.18, '&' 0.15
training names containing 'outdoor door': 215
of them, followed by 'mat': 214
```
The suggestions after `coffee`, `bar` and `kitchen` are what a shopper would want: tables, stools, sinks and islands. The ones after `outdoor` are weaker. The top suggestion, `door`, is the start of a real product type: of the 215 training names containing `outdoor door`, 214 continue with `mat`. But a one-word suggestion can't offer a two-word product type, and after `door` the shares are small, because `outdoor` is followed by many different words. A comma and an ampersand also show up as suggestions, which no search box should offer. Filtering punctuation out of suggestions is a reasonable patch here, and [chapter 2](#tokens) treats the cause.
The last line shows the model's real limit. A shopper who types `patio bar` gets exactly the suggestions for `bar`, because a bigram model looks at one word of context and nothing before it. `patio` is lost, so the model offers `pull` and `&`, which belong to cabinet hardware and to names such as `bar & counter stools`. Using more context is the subject of [chapter 3](#counting-further-back), and a model that weighs every earlier word is where the book is heading.
**Working rule:** For autocomplete, offer the top few next words by probability, and drop tokens that aren't words, such as punctuation. Expect a bigram model's suggestions to be good after words that are almost always followed by the same thing, such as `coffee`, and poor after words with many possible continuations, such as `outdoor`. When the earlier words matter, as they do in `patio bar`, a bigram model can't help, however much text it learns from.
## Generate names one word at a time {#generate-one-word-at-a-time}
The same probability table can write text as well as complete it. To **generate**, start at ``, draw the next word at random in proportion to its probability, append it, and repeat from the word you drew, until the model draws ``. Drawing at random, rather than always taking the likeliest word, is what lets the model produce varied names instead of the same one every time. The large language models this book builds toward write text the same way: one token at a time, each chosen from probabilities computed from the text so far.
For the store, generate six product names. The random generator uses a fixed seed, so the example prints the same names on every run, and the names in this book match what the code produces:
Example `generate-names` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.bigrams import END, START, Bigrams
from lib.store import load_products, masked
model = Bigrams(load_products("train")["name"])
rng = np.random.default_rng(2026) # a fixed seed, so the same names come out every run
def generate(longest=12):
word, name = START, []
while len(name) < longest:
candidates = list(model.counts[word])
chances = [model.probability(word, c) for c in candidates]
word = candidates[rng.choice(len(candidates), p=chances)] # draw the next word by its probability
if word == END:
break
name.append(word)
return " ".join(name)
generated = [generate() for _ in range(6)]
print("\n".join(masked(name) for name in generated)) # [name]: a rare word, most often a product line's name
print("generated words never seen in training:", sum(w not in model.counts for name in generated for w in name.split()))
```
Output:
```text
waterproof table
long bistro set , dimmable led , smart dryer
[name] 3 '' wide genuine leather lounge chair
[name] kitchen cart
[name] pressure balanced tub door , french [name] panel post cap
[name] 68.75 '' wide armchair
generated words never seen in training: 0
```
Read the names closely and two things stand out. Each adjacent pair of words is plausible, because each pair occurred in a real name: `genuine leather`, `kitchen cart`, `lounge chair`. And the names as a whole often aren't, because nothing keeps a name on one product. One name runs from a bistro set to a dimmable light to a smart dryer, joined by commas, since after a comma almost anything can follow. The model has no memory beyond the last word, so it can't know it was describing a bistro set. Others attach a product line's name, printed as `[name]`, to a product of a different kind, such as a leather lounge chair, because names that start with a product line are common in the training text. Nothing is invented: a bigram model can only write words it counted, and the example confirms it, with 0 generated words that the training names don't contain.
Generated names make a model's limits visible in a way a number can't, which makes them worth reading. They don't measure anything, though: six names are too few to compare two models, and a reader's sense of "plausible" changes from day to day. The next section gives the number.
**Working rule:** Generate samples whenever you change a model, and read them, because they show failures that a score hides, such as text drifting from one product to another. Use a fixed seed so the samples are reproducible. Never compare models by their samples; compare them by perplexity on validation text.
## Score a model with perplexity {#score-with-perplexity}
A language model is good when it gives high probability to the text that actually occurs. That suggests a score: run the model over real text, and see how much probability it gave to each word that came next. The standard way to summarize this is **perplexity**, and this book uses it to compare every model it builds.
Start from one pair. If the model gave the actual next word probability p, its **surprise** at that word is −log p: zero when it was certain (p = 1) and larger the less it expected the word. The **cross-entropy** of a text is the average surprise over every word in it, written with N as the number of predicted words:
cross-entropy = −(1/N) Σ log P(next word | previous word)
**Perplexity** is that average turned back into a count: perplexity = exp(cross-entropy) ([Jurafsky and Martin, chapter 3](https://web.stanford.edu/~jurafsky/slp3/3.pdf)). It reads as the number of equally likely words the model was choosing between, on average, at each step. A perplexity of 10 means the model was, on average, as uncertain as if it had to pick among 10 equally likely words. Lower is better, and a model that knew every next word would score 1. This book uses natural logarithms throughout, which changes cross-entropy's units but not perplexity.
For the store, score one training name, `sled coffee table`, pair by pair, and then every training name:
Example `perplexity-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
from lib.bigrams import Bigrams, padded, perplexity
from lib.store import load_products
train = load_products("train")["name"]
model = Bigrams(train)
name = "sled coffee table" # a name from the training split
tokens = padded(name)
surprise = 0.0
for word, following in zip(tokens, tokens[1:]):
p = model.probability(word, following)
surprise += -math.log(p)
print(f"P({following} | {word}) = {p:.4f} -log P = {-math.log(p):.2f}")
print("average -log P:", round(surprise / (len(tokens) - 1), 3))
print("perplexity of the name:", round(math.exp(surprise / (len(tokens) - 1)), 1))
print("perplexity of every training name:", round(perplexity(model, train), 1))
```
Output:
```text
P(sled | ) = 0.0001 -log P = 8.83
P(coffee | sled) = 0.7846 -log P = 0.24
P(table | coffee) = 0.8960 -log P = 0.11
P( | table) = 0.6903 -log P = 0.37
average -log P: 2.389
perplexity of the name: 10.9
perplexity of every training name: 16.9
```
The surprise is almost all at the start. Few names begin with `sled`, so the model was surprised by it, and after that each word was close to expected: `coffee` follows `sled` in most training names that contain `sled`, and `table` follows `coffee`. The name's average surprise is 2.389, and its perplexity 10.9. Over every training name, the model's perplexity is 16.9: on the text it learned from, it was choosing among about that many likely words at each step.
That number flatters the model, because it's measured on the text the counts came from. Every pair in the training names has a count, so no pair surprises the model infinitely. The real test is text it hasn't seen.
**Working rule:** Score every language model by its perplexity on validation text, computed exactly as above, with natural logarithms and every predicted word counted, the end marker included. Compare perplexities only between models that use the same tokens and the same validation text, since splitting text differently changes N and what counts as a word. Report training perplexity only beside validation perplexity, to show how much the model memorized.
## Unseen word pairs break the model {#unseen-pairs-break-the-model}
A model trained by counting knows only the pairs it has counted, and the validation names were kept apart from the training names for exactly this reason: they show what happens when the model meets text it never saw. Before running the next example, predict its first line. The model scored 16.9 on the training names, in [score a model with perplexity](#score-with-perplexity). What will its perplexity be on the 4,282 validation names?
Example `perplexity-on-new-names` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.bigrams import Bigrams, padded, perplexity
from lib.store import load_products, masked
model = Bigrams(load_products("train")["name"])
validation = load_products("validation")["name"]
print("validation perplexity:", perplexity(model, validation))
pairs = [(w, f) for name in validation for w, f in zip(padded(name), padded(name)[1:])]
unseen = [pair for pair in pairs if model.probability(*pair) == 0]
print("validation word pairs:", f"{len(pairs):,}")
print("pairs never seen in training:", f"{len(unseen):,}")
print("share never seen in training:", f"{len(unseen) / len(pairs):.0%}")
names_hit = sum(any(model.probability(*p) == 0 for p in zip(padded(n), padded(n)[1:])) for n in validation)
print("names with an unseen pair:", f"{names_hit:,}")
print("validation names:", f"{len(validation):,}")
print("first unseen pairs:", ", ".join(masked(f"{w} {f}") for w, f in unseen[:3])) # [name]: a rare word
```
Output:
```text
validation perplexity: inf
validation word pairs: 33,336
pairs never seen in training: 8,278
share never seen in training: 25%
names with an unseen pair: 3,724
validation names: 4,282
first unseen pairs: [name] [name], [name] 4, wood half-circle
```
The validation perplexity is infinite: the example prints inf. One pair the model gave probability zero is enough, because its surprise, −log 0, is infinite, and so is any average that includes it. And there isn't only one. Of the 33,336 word pairs in the validation names, 8,278 never occur in the training names, 25% of them, and 3,724 of the 4,282 validation names contain at least one. The first few show why: a validation name that starts with a product line the training names never mention, printed as `[name]`, makes both the first pair and the second unseen.
This is the flaw the [probabilities section](#counts-to-probabilities) warned about. Maximum likelihood gives every unseen pair probability zero, and in real text most new documents contain something unseen: a new product line, an unusual size, a familiar word in an unfamiliar place, such as `half-circle` after `wood`. More training text helps a little, but it can't solve the problem. Language keeps producing new combinations, so a model that calls every unseen combination impossible will always fail on new text.
The fix has to give unseen pairs some probability, which means taking a little away from the pairs the model has seen. That's called **smoothing**.
**Working rule:** Never trust a language model that gives zero probability to anything; check for infinite or undefined perplexity on validation text before you compare models. When you see one, count the unseen pairs, as above, to know how much of the text the model can't handle.
## Give unseen pairs a little probability {#smooth-unseen-pairs}
The model in [unseen word pairs break the model](#unseen-pairs-break-the-model) fails because it gives unseen pairs probability zero, and smoothing gives them some. The simplest smoothing method, **add-one smoothing**, pretends that every possible pair occurred once more than it did. Every count goes up by one, including the zeros, and the totals grow to match:
P(v | w) = (count(w v) + 1) / (count(w) + V)
where V is the size of the **vocabulary**, every word the model can predict. A word the training text doesn't contain at all is counted as one more vocabulary entry, ``, for unknown, so the probabilities still sum to one. No pair now has probability zero, so perplexity on new text is finite. The question is what it costs.
For the store, compare the counted and add-one models on the probability of `table` after `coffee`, and on perplexity:
Example `add-one-smoothing` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.bigrams import Bigrams, perplexity
from lib.store import load_products
model = Bigrams(load_products("train")["name"])
train, validation = load_products("train")["name"], load_products("validation")["name"]
print("vocabulary:", f"{len(model.vocabulary):,}")
print("P(table | coffee) counted:", round(model.probability("coffee", "table"), 3))
print("P(table | coffee) add-one:", round(model.probability("coffee", "table", add_one=True), 3))
print("training perplexity, counted:", round(perplexity(model, train), 1))
print("training perplexity, add-one:", f"{perplexity(model, train, add_one=True):,.1f}")
print("validation perplexity, add-one:", f"{perplexity(model, validation, add_one=True):,.1f}")
```
Output:
```text
vocabulary: 25,909
P(table | coffee) counted: 0.896
P(table | coffee) add-one: 0.042
training perplexity, counted: 16.9
training perplexity, add-one: 1,232.8
validation perplexity, add-one: 1,575.1
```
The vocabulary holds 25,909 words, the end marker and `` included, and that's the problem. Add-one smoothing adds one imagined occurrence for every one of them after every word, so after `coffee` it imagines tens of thousands of extra pairs against about a thousand real ones. The probability of `table` after `coffee` falls from 0.896 to 0.042. The model now believes far less in what it has seen, and the perplexities show it: on the training names, perplexity rises from 16.9 to 1,232.8, and on the validation names it's 1,575.1. The validation perplexity is finite, which is progress, but the model gives so much probability to word pairs that never happen that its predictions are nearly useless.
Add-one smoothing fixes the infinity and breaks almost everything else, because it adds one imagined pair for every word in the vocabulary, however much evidence the real counts hold, and spreads them evenly. Better smoothing methods take far less probability away from seen pairs, and give it to unseen pairs in proportion to how often their words occur elsewhere. They are the subject of [chapter 3](#counting-further-back), and the counted model of this chapter, looking further back and with a better smoothing method, becomes the baseline that every later model in the book is measured against.
**Working rule:** Never ship a counted model without smoothing, and never ship one with add-one smoothing: measure validation perplexity and expect add-one to lose to every method in [chapter 3](#counting-further-back), which compares them on the same tokens. Use add-one only as this chapter does, to show that zero probabilities are fixable and that the fix needs care.
## What goes wrong in production {#next-word-in-production}
A bigram autocomplete is cheap and fast, and it's still a fair baseline for a search box. These are the problems that come with it:
- **The catalog moves on.** A model counted from last month's names doesn't know this month's product lines. A shopper who types a new product line gets no suggestions at all, because the model has no counts after it, until the counts are rebuilt, so rebuild them on a schedule and whenever the catalog changes.
- **Suggestions can embarrass.** The model suggests whatever followed a word in its training text, punctuation included. Filter suggestions through a list of allowed tokens, and review the most frequent suggestions by hand before they go live.
- **Popular products crowd out the rest.** A word followed by one product type most of the time, such as `coffee`, will always suggest that type. Shoppers looking for a coffee mug have to type more. Offer several suggestions, not one.
- **One word of context is too little.** As `patio bar` showed, the model ignores everything but the last word. If shoppers type long queries, measure how often the suggestions change once you condition on more of them, which [chapter 3](#counting-further-back) does.
- **The table grows with the text.** A bigram table has an entry for every pair ever seen; with longer contexts, the number of entries explodes. Count only what you need, and prune pairs seen once if memory is tight.
## What this chapter decided {#next-word-decisions}
- **The store's text is split once, by product, into training, validation and test splits, and models learn from the training split only.** A score on text a model learned from measures memory, not prediction, and splitting by product keeps near-duplicate names of one product from landing on both sides.
- **Every text is padded with `` and ``.** Models then learn how texts start and end, which autocomplete needs for a shopper's first word and generation needs to stop.
- **Models are compared by perplexity on validation text,** with natural logarithms and every predicted word counted, the end marker included. Generated samples are read for what they show, but never used to choose between models.
- **The bigram model of the store's product names, built from counts, is the book's first model.** It suggests good next words after words with one common continuation, such as `coffee`, and poor ones after words with many, such as `outdoor`. It sees one word of context, so `patio bar` and `bar` get the same suggestions.
- **Unsmoothed, its validation perplexity is infinite; with add-one smoothing it's finite but poor.** [Chapter 3](#counting-further-back) gives it a better smoothing method, makes it the baseline, and keeps the same splits and the same measure, so every later model is compared with it on equal terms.
# Tokens
By the end of this chapter you can build a byte-pair encoding tokenizer from scratch, choose its vocabulary size by measuring what each size costs, and read the tokenizer of an open language model well enough to apply its merges by hand. Every model in the rest of the book reads the store's text through the tokenizer you build here.
A **token** is the unit a language model reads and predicts: whatever the text is cut into before the model sees it. [Chapter 1](#predicting-the-next-word) cut the store's product names into words, because the store's text, which this book's examples learn from, already has spaces between words and around punctuation. That choice was quick and it fell short in two ways. The bigram model couldn't give any probability to a word it never saw in training, so a word it hadn't met in training, such as `barstools`, broke it. And it treated `chair` and `chairs`, or `48` and `49`, as unrelated words, with nothing learned about one carried over to the other. The first problem comes from the choice of tokens, so this chapter fixes the tokens. The second needs more than tokens can give, and [chapter 4](#words-as-vectors) takes it up.
The store is the online furniture and home-goods store this book follows throughout, and its text is the product names and descriptions of its catalog. This chapter measures three ways to cut that text into tokens: words, characters, and the middle ground that most open generative models use, byte-pair encoding. Then it opens one of those models' tokenizers and takes it apart.
## Words leave gaps {#words-leave-gaps}
Cutting text into words gives a model tokens that carry meaning, which is why [chapter 1](#predicting-the-next-word) started there. The cost is the vocabulary. A model can only predict tokens in its **vocabulary**, the set of tokens it learned from training text, so any word that first appears after training is out of reach: an **out-of-vocabulary** word. In text with names, numbers and new products, there are always some.
For the store, count how many words in the validation names never appear in the training names. The validation names are the 10% of the store's products that no model learns from, kept aside to show how a model handles text it hasn't seen:
Example `unseen-words` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib.store import is_ordinary, load_products, words
train_words = Counter(w for name in load_products("train")["name"] for w in words(name))
validation = [w for name in load_products("validation")["name"] for w in words(name)]
unseen = [w for w in validation if w not in train_words]
print("different words in training names:", f"{len(train_words):,}")
print("words seen once in training:", f"{sum(1 for n in train_words.values() if n == 1):,}")
print("validation words:", f"{len(validation):,}")
print("validation words never seen in training:", f"{len(unseen):,}")
print("share never seen in training:", f"{len(unseen) / len(validation):.1%}")
print("share of them that are rare words, most often product lines' names:",
f"{sum(not is_ordinary(w) for w in unseen) / len(unseen):.0%}")
print("most frequent of the ordinary ones:", ", ".join(w for w, _ in Counter(w for w in unseen if is_ordinary(w)).most_common(6)))
```
Output:
```text
different words in training names: 25,907
words seen once in training: 17,242
validation words: 29,054
validation words never seen in training: 2,109
share never seen in training: 7.3%
share of them that are rare words, most often product lines' names: 92%
most frequent of the ordinary ones: neutral, noise-free, details, spacious, 52.4, houses
```
The training names hold 25,907 different words, and 17,242 of them occur only once, which is common for product lines and model numbers. Of the 29,054 words in the validation names, 2,109 never occur in training, 7.3% of all validation words. Most of them, 92%, are rare words, most often product lines' names, which [chapter 1](#load-the-store-text) prints as `[name]`; the rest are ordinary words that happen to miss the training names, such as `neutral` and `noise-free`. A word-level model can't predict any of them, can't score text that contains them without special handling, and can't relate `noise-free` to `noise` and `free`, which it knows well.
A larger training set would shrink the gap but never close it, because the catalog keeps adding product lines, and a word seen once gives a model almost nothing to learn from anyway.
**Working rule:** Measure the out-of-vocabulary rate of your text on held-out data before choosing word tokens. As a rule of thumb, if more than a percent or two of held-out words are unseen, as in the store's names, don't use words as tokens. Word tokens remain reasonable for text with a closed vocabulary, such as a fixed set of commands.
## Characters cover everything {#characters-cover-everything}
Going to the other extreme fixes the gap. If every character is a token, the vocabulary is tiny and almost nothing is out of it: a new product line is spelled with the same letters as an old one. The price is length. Each word becomes several tokens, so the model has more steps to predict, and the context it needs, such as the start of a product's name, is many more tokens back.
For the store, compare the training names' characters with the validation names', and count tokens both ways:
Example `character-tokens` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import unicodedata
from lib.store import load_products, words
train_names, validation_names = load_products("train")["name"], load_products("validation")["name"]
characters = {c for name in train_names for c in name}
unseen = {c for name in validation_names for c in name} - characters
word_tokens = sum(len(words(name)) for name in validation_names)
character_tokens = sum(len(name) for name in validation_names)
print("different characters in training names:", len(characters))
print("validation characters never seen in training:", len(unseen))
print("they are:", ", ".join(unicodedata.name(c).lower() for c in sorted(unseen)))
print("validation tokens as words:", f"{word_tokens:,}")
print("validation tokens as characters:", f"{character_tokens:,}")
print("characters per word:", round(character_tokens / word_tokens, 1))
```
Output:
```text
different characters in training names: 86
validation characters never seen in training: 2
they are: soft hyphen, latin small letter t with cedilla
validation tokens as words: 29,054
validation tokens as characters: 174,142
characters per word: 6.0
```
The training names use 86 different characters, and only 2 characters in the validation names never appear in them. As words, the validation names are 29,054 tokens; as characters, 174,142, or 6.0 characters for each word, counting the spaces. A model reading characters has to predict `c`, `o`, `f`, `f`, `e`, `e` before it gets to the point where a word model starts: knowing that `table` is likely next.
Characters also leave a last, small gap. The 2 unseen characters are a soft hyphen and a `t` with a cedilla, and a character tokenizer can't spell a word that contains one. Open models close that gap by working with **bytes** instead: in UTF-8, the usual text encoding, every character in every language is stored as one to four bytes, and there are only 256 possible byte values, so a vocabulary that starts from all 256 can represent any text at all ([Radford et al., 2019](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), section 2.2). The tokenizer read later in this chapter does exactly that.
Longer sequences cost in two ways. Every step a model takes is computation, so 6.0 times the steps is at least 6.0 times the work, and more for the models of later chapters, whose cost grows faster than the length. And every model in this book looks back over a limited window of tokens; in characters, that window covers that many times fewer words.
**Working rule:** Use characters as tokens only for small experiments, or as a fallback inside a larger scheme, as the next sections do. Check how many characters per word your text has, since that factor multiplies every sequence length and every training step.
## Merge the most frequent pair, again and again {#merge-frequent-pairs}
Words are too coarse and characters too fine, so the method most open generative models use, SmolLM2 among them, sits between them. **Byte-pair encoding**, or BPE, starts from characters and builds bigger tokens out of the pairs that occur most often. It began as a data-compression method, which Sennrich, Haddow and Birch adapted to cut words into pieces for translation models ([Sennrich et al., 2016](https://aclanthology.org/P16-1162/)). The algorithm is short:
1. Spell every word in the training text as characters, and mark its end with ``, so a merge never joins one word to the next.
2. Count every adjacent pair of symbols, weighting each word by how often it occurs.
3. Merge the most frequent pair into one new symbol everywhere, and add it to the vocabulary.
4. Repeat from step 2, as many times as the vocabulary size you want.
The list of merges, in the order they were learned, is the tokenizer. To encode a word, spell it as characters and apply the merges in the same order.
For the store, run the algorithm by hand on the training names for eight merges, written out in full so every step is visible:
Example `merge-pairs-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import bpe
from lib.store import load_products, words
word_counts = Counter(w for name in load_products("train")["name"] for w in words(name))
spelled = {w: list(w) + [""] for w in word_counts} # every word as characters, plus an end marker
learned = []
for step in range(1, 9):
pairs = Counter()
for w, syms in spelled.items(): # count every adjacent pair of symbols
for pair in zip(syms, syms[1:]):
pairs[pair] += word_counts[w]
best = min(pairs, key=lambda p: (-pairs[p], p)) # the most frequent pair; ties go to the first in order
for w, syms in spelled.items(): # replace it everywhere with one new symbol
merged, i = [], 0
while i < len(syms):
if syms[i:i + 2] == list(best):
merged.append(syms[i] + syms[i + 1])
i += 2
else:
merged.append(syms[i])
i += 1
spelled[w] = merged
learned.append(best)
print(f"merge {step}: {best[0]!r} + {best[1]!r} -> {best[0] + best[1]!r} ({pairs[best]:,} times)")
print("table spelled now:", spelled["table"])
print("same merges as lib.bpe.train:", learned == bpe.train(word_counts, 8))
```
Output:
```text
merge 1: 'e' + '' -> 'e' (35,498 times)
merge 2: 'r' + '' -> 'r' (24,635 times)
merge 3: 'i' + 'n' -> 'in' (21,312 times)
merge 4: 'd' + '' -> 'd' (20,216 times)
merge 5: 't' + '' -> 't' (18,628 times)
merge 6: 'a' + 'n' -> 'an' (15,697 times)
merge 7: 'e' + 'r' -> 'er' (12,981 times)
merge 8: 's' + 't' -> 'st' (12,781 times)
table spelled now: ['t', 'a', 'b', 'l', 'e']
same merges as lib.bpe.train: True
```
The first merges join common letters to the end marker, such as `e` with ``, because so many of the store's words end in `e`, `r` or `d`. Then common pieces inside words appear, such as `in`, `an` and `st`. After eight merges, `table` is spelled `t a b l e`: still mostly characters. With thousands of merges, common words such as `table` become single tokens, while rare words stay split into pieces that other words share.
That's the idea that solves [chapter 1](#predicting-the-next-word)'s first problem. Frequent words get their own tokens, so sequences stay short. Rare and new words are spelled from pieces the model has seen, so any word spelled with characters the training text contains is in reach. BPE doesn't solve the second problem: as the [next section](#train-the-store-tokenizer) shows, a frequent word and its plural can each become a whole token with nothing in common.
**Working rule:** Learn merges from the training split only, as you would any other model, and keep the order of the merges, since encoding depends on it. Break ties between equally frequent pairs by a fixed rule, as the example does, so the same text always gives the same tokenizer.
## Train the store's tokenizer {#train-the-store-tokenizer}
The algorithm in the [previous section](#merge-frequent-pairs) recounts every pair after every merge, which is fine for eight merges and too slow for thousands. The book's shared module `lib.bpe` runs the same algorithm, but counts pairs once and, after each merge, updates only the words the merge changed, keeping the counts in a heap so the most frequent pair is always on top. It learns thousands of merges in seconds, and gives the same merges the plain version does: the hand-written example's last line checks its eight merges against `lib.bpe.train`.
For the store, learn 4,000 merges from the training split's names and descriptions together, since the book's models will learn from both, and encode three names:
Example `store-tokenizer` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import bpe
from lib.store import load_products, words
train = load_products("train")
word_counts = Counter(w for text in train["name"] + " " + train["description"] for w in words(text))
tokenizer = bpe.Tokenizer(bpe.train(word_counts, merges=4000), bpe.alphabet(word_counts)) # the training split's text
print("different words in training text:", f"{len(word_counts):,}")
print("characters and the end marker:", len(bpe.alphabet(word_counts)))
print("tokens in the vocabulary:", f"{len(tokenizer.vocabulary):,}")
for name in ["set of 2 barstools", "sled coffee table", "chair chairs", "48 49", "noise noise-free"]:
print(name, "->", " ".join(tokenizer.encode(name)))
```
Output:
```text
different words in training text: 49,315
characters and the end marker: 137
tokens in the vocabulary: 4,138
set of 2 barstools -> set of 2 bar stools
sled coffee table -> sled coffee table
chair chairs -> chair chairs
48 49 -> 48 4 9
noise noise-free -> noise no i se -free
```
The training text holds 49,315 different words, and the tokenizer's vocabulary has 4,138 tokens: the 137 characters of the training text and the end marker, one new token for each of the 4,000 merges, and ``, which stands for any character the training text never contained. Common words come out whole: `sled coffee table` is three tokens, one per word. The word `barstools`, which [chapter 1's](#unseen-pairs-break-the-model) bigram model couldn't handle because the training names never contain it, comes out as two pieces the tokenizer knows well, `bar` and `stools`. The last three lines show what BPE doesn't do. `chair` and `chairs` are both common, so each became a whole token, and the two tokens share nothing. `48` is one token and `49` two, `4` and `9`. And `noise-free` splits into pieces that share nothing with `noise`. Tokens give every word a spelling, not a meaning; relating them is the job of [chapter 4](#words-as-vectors).
A model reading these tokens can spell any word made of characters the training text contains. It still has to learn what a new product line is, but it can at least predict one, and give it a probability.
**Working rule:** Train the tokenizer on the same text your model will learn from, here names and descriptions together, so frequent words in that text become single tokens. Build it once and use it for every model you compare; the book's models all read the store's text through `lib.bpe.store_tokenizer()`.
## Choose how many merges {#choose-how-many-merges}
The number of merges sets the vocabulary size, and it's the one real choice in a BPE tokenizer. More merges make sequences shorter, which saves computation. But each new token covers rarer text, and a token the training text shows only a few times gives a model too little to learn from: the same problem as rare words, in a new place.
For the store, learn 16,000 merges once, and use the first 0, 1,000, 4,000 and 16,000 of them as four tokenizers; the first k merges of a longer run are exactly the merges a k-merge run would learn. For each, count how many different tokens the training text uses, how many tokens each validation word becomes, and what share of the tokens appear fewer than 20 times in the training text:
Example `vocabulary-size` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import bpe
from lib.store import load_products, words
training, validation = load_products("train"), load_products("validation")
counts = Counter(w for text in training["name"] + " " + training["description"] for w in words(text))
merges = bpe.train(counts, merges=16000) # the first k merges of a longer run are a k-merge tokenizer
validation_words = [w for text in validation["name"] + " " + validation["description"] for w in words(text)]
results = {}
print(f"{'merges':>6} {'tokens used':>11} {'tokens per word':>15} {'rare types':>10}")
for k in [0, 1000, 2000, 4000, 8000, 16000]:
tokenizer = bpe.Tokenizer(merges[:k], bpe.alphabet(counts)) if k else None
encode = tokenizer.encode_word if k else bpe.symbols
seen = Counter(t for w, n in counts.items() for t in encode(w) for _ in range(n))
per_word = sum(len(encode(w)) for w in validation_words) / len(validation_words)
rare = sum(1 for n in seen.values() if n < 20) / len(seen) # share of different tokens seen fewer than 20 times
print(f"{k:>6,} {len(seen):>11,} {per_word:>15.2f} {rare:>10.0%}")
results[k] = per_word, rare
print("tokens per word at 4,000 merges:", f"{results[4000][0]:.2f}")
print("rare token types with characters only:", f"{results[0][1]:.0%}")
print("rare token types at 4,000 merges:", f"{results[4000][1]:.0%}")
print("rare token types at 16,000 merges:", f"{results[16000][1]:.0%}")
```
Output:
```text
merges tokens used tokens per word rare types
0 137 5.61 49%
1,000 1,130 1.71 7%
2,000 2,123 1.42 5%
4,000 4,099 1.22 6%
8,000 8,007 1.10 13%
16,000 15,517 1.05 53%
tokens per word at 4,000 merges: 1.22
rare token types with characters only: 49%
rare token types at 4,000 merges: 6%
rare token types at 16,000 merges: 53%
```
Before you read the table, consider what each end of it should show. With no merges, every token is a character, so tokens are frequent but words are long. With many merges, words are short, but many tokens are rare. The table bears that out. The last column is the share of the different tokens the training text uses, its token types, that it shows fewer than 20 times. With characters only, a word takes more than five tokens, and 49% of the token types are rare, because the descriptions contain unusual characters. At 4,000 merges, a validation word takes 1.22 tokens on average, and 6% of the token types are rare, about as few as at 2,000. From 8,000 merges the rare share climbs, and at 16,000 it's 53%: most of the extra merges cover words the training text barely contains.
The vocabulary size also sets part of the model's size. A neural language model, from [chapter 5](#a-neural-language-model) on, stores a vector of numbers for every token in its vocabulary, and scores every token in the vocabulary at every step it predicts. A vocabulary four times as large means four times as many of those vectors to learn, and four times as many scores to compute at every step, whatever the text. For the small models this book trains on a laptop, that cost matters as much as sequence length.
So the store's tokenizer uses 4,000 merges: past that, each merge buys little length and adds a token that the book's small models can't learn well. Large open models use vocabularies of tens of thousands of tokens or more, SmolLM2's has 49,152, because they learn from far more text, so even their rarer tokens occur often enough.
**Working rule:** Choose the number of merges by measuring, on your training text, the share of rare token types, and on held-out text, the tokens per word; stop where tokens per word stops falling much and rare types start climbing. Scale the vocabulary with the training text; as a rule of thumb, a small model on a few million words wants thousands of merges, not tens of thousands.
## Read an open model's tokenizer {#read-an-open-tokenizer}
Open language models ship their tokenizer as a file beside their weights, and reading one shows the same ideas at a larger scale, plus a few choices the store's tokenizer didn't have to make. This book uses SmolLM2, an open language model by Hugging Face whose weights and tokenizer anyone can download, under the Apache 2.0 licence ([model card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M)). The book keeps a copy of its tokenizer file in `examples/data/smollm2-tokenizer/`, fetched at a pinned revision by `prep/smollm2_tokenizer.py`, so the examples run offline. The examples read it with the tokenizers library, 0.23.3, the library the file was made for.
For the store, encode three product names with SmolLM2's tokenizer and with the store's, and count tokens per validation name each way:
Example `smollm2-tokens` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · tokenizers 0.23.3), passing in Edition 2026.10.3:
```python
from tokenizers import Tokenizer
from lib import bpe
from lib.store import load_products
smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")
store = bpe.store_tokenizer()
print("SmolLM2 vocabulary:", f"{smollm2.get_vocab_size():,}")
for name in ["set of 2 barstools", "6.5 qt . slow cooker", "48 '' wide genuine leather wingback chair"]:
print(name)
print(" SmolLM2:", " ".join(smollm2.encode(name).tokens))
print(" store: ", " ".join(store.encode(name)))
names = load_products("validation")["name"]
print("tokens per validation name, SmolLM2:", round(sum(len(smollm2.encode(n).ids) for n in names) / len(names), 2))
print("tokens per validation name, store:", round(sum(len(store.encode(n)) for n in names) / len(names), 2))
print("tokens per validation name, words:", round(sum(len(n.split()) for n in names) / len(names), 2))
print("sled at the start of a text:", " ".join(smollm2.encode("sled coffee table").tokens))
print("sled after a space:", " ".join(smollm2.encode(" sled coffee table").tokens))
words_total = sum(len(n.split()) for n in names)
print("SmolLM2 tokens per word:", round(sum(len(smollm2.encode(n).ids) for n in names) / words_total, 2))
```
Output:
```text
SmolLM2 vocabulary: 49,152
set of 2 barstools
SmolLM2: set Ġof Ġ 2 Ġbar st ools
store: set of 2 bar stools
6.5 qt . slow cooker
SmolLM2: 6 . 5 Ġq t Ġ. Ġslow Ġcooker
store: 6.5 qt . slow cooker
48 '' wide genuine leather wingback chair
SmolLM2: 4 8 Ġ'' Ġwide Ġgenuine Ġleather Ġwing back Ġchair
store: 48 '' wide genuine leather wingback chair
tokens per validation name, SmolLM2: 10.53
tokens per validation name, store: 9.91
tokens per validation name, words: 6.79
sled at the start of a text: s led Ġcoffee Ġtable
sled after a space: Ġsled Ġcoffee Ġtable
SmolLM2 tokens per word: 1.55
```
SmolLM2's vocabulary has 49,152 tokens, against the store tokenizer's 4,138, because it was trained on far more text than furniture names. Three differences in its output are worth naming:
- **Spaces are part of tokens.** `Ġ` stands for a space: `Ġtable` is "table" with the space before it. Instead of marking where a word ends, as the store's `` does, this tokenizer keeps each word's leading space, so the text can be rebuilt exactly, spaces included.
- **Every digit is its own token.** `48` becomes `4` and `8`, and `6.5` becomes `6`, `.` and `5`. Splitting digits stops the vocabulary from filling with numbers, each seen a few times, and treats all numbers the same way.
- **A word at the start of a text has no space before it, so it's a different token.** SmolLM2's vocabulary has `Ġsled` as one token, but not `sled` without the space, so a name that starts with `sled` gets `s` and `led`, as the last lines of the output show. And `barstools`, which SmolLM2's vocabulary lacks as a whole word, splits into `bar`, `st` and `ools`, as the first lines show.
On the store's validation names, SmolLM2 uses 10.53 tokens per name and the store's tokenizer 9.91, against 6.79 words. A tokenizer trained on your own text is slightly more efficient on it, partly because it doesn't split numbers into digits, and partly because SmolLM2 spends a separate `Ġ` token before every number and splits each name's first word, which has no space before it.
**Working rule:** When you use an open model, use its tokenizer exactly as shipped; a model only understands the tokens it was trained on. Count tokens with that tokenizer, not words, whenever you budget for context length or cost. When you train your own model on narrow text, train your own tokenizer on that text.
## Apply SmolLM2's merges by hand {#apply-merges-by-hand}
SmolLM2's tokenizer file looks like a black box until you see that it holds the same thing the store's tokenizer does: a list of merges, in order. The file, `tokenizer.json`, has three parts that matter here. A pre-tokenizer splits text into pieces before any merging: its steps, which the example prints, split digits apart one at a time, then split words with their leading spaces and map every byte to a printable character, which is where `Ġ` comes from. That mapping is GPT-2's `bytes_to_unicode` ([GPT-2's encoder](https://github.com/openai/gpt-2/blob/master/src/encoder.py)). Then a list of merges, ranked by when they were learned. And the vocabulary, which gives each token its number.
For the store, take the pre-tokenizer's pieces from the library, apply the merges by hand, earliest-learned first, as in the [store's tokenizer](#train-the-store-tokenizer), and compare the result with the library's:
Example `merges-by-hand` (Python 3.12.15 · tokenizers 0.23.3), passing in Edition 2026.10.3:
```python
import json
from tokenizers import Tokenizer
smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")
spec = json.load(open("data/smollm2-tokenizer/tokenizer.json", encoding="utf-8"))
rank = {tuple(m.split(" ")): i for i, m in enumerate(spec["model"]["merges"])} # lower rank: learned earlier
def merge(piece): # apply the learned merges, earliest first, until none applies
syms = list(piece)
while len(syms) > 1:
pairs = [(rank.get(p, len(rank)), i) for i, p in enumerate(zip(syms, syms[1:]))]
best, i = min(pairs)
if best == len(rank):
break
syms[i:i + 2] = [syms[i] + syms[i + 1]]
return syms
for name in ["set of 2 barstools", "sled coffee table", "6.5 qt . slow cooker"]:
pieces = [p for p, _ in smollm2.pre_tokenizer.pre_tokenize_str(name)] # split into words, digits and bytes
by_hand = [token for piece in pieces for token in merge(piece)]
print(name, "->", " ".join(by_hand), "| same as the library:", by_hand == smollm2.encode(name).tokens)
steps = [step["type"] for step in spec["pre_tokenizer"]["pretokenizers"]]
print("pre-tokenizer steps:", ", ".join(steps), "| digits one at a time:", spec["pre_tokenizer"]["pretokenizers"][0]["individual_digits"])
```
Output:
```text
set of 2 barstools -> set Ġof Ġ 2 Ġbar st ools | same as the library: True
sled coffee table -> s led Ġcoffee Ġtable | same as the library: True
6.5 qt . slow cooker -> 6 . 5 Ġq t Ġ. Ġslow Ġcooker | same as the library: True
pre-tokenizer steps: Digits, ByteLevel | digits one at a time: True
```
The tokens built by hand match the library's for every name: the line ends with `same as the library: True` each time. That's all SmolLM2's tokenizer does to these names; it also defines special tokens, matched before any of these steps. It differs from the store's in its pre-tokenizer and its training text, but it encodes text by repeatedly merging the earliest-learned pair, as the store's does.
The match is worth checking because subtle differences in how text is split matter downstream. A model trained on one tokenizer and fed another's tokens reads nonsense without raising an error, since every token number is valid. The same thing happens, and is harder to spot, when one version of a tokenizer differs from another in how it handles spaces or digits.
**Working rule:** Treat a tokenizer as part of a model: version it with the model, load it from the model's own files, and test it on your text, as this section does, whenever you upgrade either one. When two tokenizers disagree on even one token for the same text, find out why before you trust anything a model built on them says.
## What goes wrong in production {#tokens-in-production}
Tokens decide what a model can see, and most tokenizer problems show up far from the tokenizer:
- **A mismatched tokenizer fails silently.** A model given another tokenizer's token numbers reads valid numbers that mean the wrong things, and produces confident nonsense. Ship the tokenizer file with the model, and test them together.
- **Costs are counted in tokens.** Context limits and per-token prices apply to tokens, not words. The store's names take 1.55 tokens per word in SmolLM2's tokenizer; text full of numbers, codes or another language takes more. Measure your own text.
- **Numbers are hard.** A tokenizer that splits digits apart makes a model reason about `48` as two tokens; one that doesn't gives `48` and `49` unrelated tokens. Either way, expect models to handle sizes and quantities worse than words, and check them separately.
- **Special tokens can be typed.** Open models mark the start or end of a text, or a turn in a conversation, with special tokens written as strings such as `<|endoftext|>`. If text a user typed contains one of those strings and the tokenizer turns it into the special token, the model reads a control signal the user shouldn't be able to send. Check how your tokenizer handles special-token strings in untrusted text.
- **New text drifts from the training text.** A tokenizer trained on last year's catalog splits this year's product lines into more pieces. Measure tokens per word on new text from time to time; a steady rise means the tokenizer is falling behind its text.
## What this chapter decided {#tokens-decisions}
- **The book's models read the store's text as byte-pair encoding tokens.** Words left 7.3% of validation words out of the vocabulary, and characters made sequences 6.0 times as long as words. BPE avoids both: any word spelled with characters the training text contains can be encoded, and common words are single tokens. It doesn't relate words to each other, such as `chair` and `chairs`; that's [chapter 4](#words-as-vectors)'s job.
- **The store's tokenizer learns 4,000 merges from the training split's names and descriptions,** in `lib.bpe.store_tokenizer()`. At that size, a word averages 1.22 tokens, and few token types are too rare to learn. More merges shorten sequences a little and add many rare tokens.
- **Each word ends with ``, so merges never cross words.** Open models such as SmolLM2 attach the space to the next word instead; both keep word boundaries visible to the model.
- **Open models are always used with their own tokenizers.** A model only understands the token numbers it was trained on, so [chapter 10](#inside-an-open-model) reads SmolLM2's model with the tokenizer this chapter took apart, and every recording of an open model in this book names the tokenizer it used.
- **Perplexities from different tokenizers aren't comparable.** A model that predicts more, smaller tokens takes more steps and has a different perplexity on the same text, so every comparison of models in this book uses the store's tokenizer, and [chapter 3](#counting-further-back) starts that comparison over on its tokens.
# Counting further back
By the end of this chapter you can build an n-gram language model that looks several tokens back, smooth it with Kneser-Ney so it handles text it hasn't seen (smoothing gives some probability to what the counts never saw, as chapter 1 showed), and measure the baseline that every neural model in this book is measured against. You'll also see why counting, however clever, runs out of road.
[Chapter 1](#predicting-the-next-word) built a model that predicted the next word from the one before it, and two things went wrong. It looked back only one word, so a shopper who typed `patio bar` got the suggestions for `bar`. And it gave probability zero to every pair of words it hadn't counted, which made its perplexity on new text infinite. [Chapter 2](#tokens) then gave the book better units to count: the store's tokenizer, which splits every word into pieces it knows, so almost every word is spelled from tokens it has seen. This chapter goes back to counting with those tokens, looks further back, and handles the unseen properly.
The store, the online furniture and home-goods store this book follows, is where every method here is tried. From this chapter on, every model in the book is scored the same way: by its [perplexity](#score-with-perplexity) per token on the store's validation names, tokenized with the store's tokenizer. The validation names are the product names of the 10% of products no model learns from. Product names are what the search box's autocomplete has to predict, and they're short enough for every example to run in seconds. One fixed text, one tokenizer and one measure make every model in the book comparable with every other on names; later chapters add descriptions, from [chapter 8](#training-a-small-model), and per-word scores, in [chapter 10](#inside-an-open-model), where a model's tokenizer differs.
## Longer contexts fit the training text better {#longer-contexts-fit-training}
An **n-gram model** predicts each token from the n − 1 tokens before it: a **bigram** model (n = 2) from one, a trigram model (n = 3) from two, and so on. The tokens before the one being predicted are its **context**. More context should mean better predictions: after `patio bar`, the next token is far more predictable than after `bar` alone. Counting works exactly as in [chapter 1](#predicting-the-next-word), except that the counts are now of a context of n − 1 tokens followed by a token, and each text is padded with n − 1 start markers, so even its first token has a full context.
For the store, count n-grams of every order from 1 to 5 in the training names, and score each model, without any smoothing, on the same training names it counted:
Example `ngram-memorizes` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
from collections import Counter
from lib.names import name_tokens
from lib.ngrams import ngrams, padded
train = name_tokens("train")
for n in [1, 2, 3, 4, 5]:
pairs = Counter(pair for text in train for pair in ngrams(padded(text, n), n))
contexts = Counter()
for (context, _), count in pairs.items():
contexts[context] += count
surprise = -sum(count * math.log(count / contexts[context]) for (context, _), count in pairs.items())
print(f"{n}-gram training perplexity: {math.exp(surprise / sum(pairs.values())):.1f}")
print(f"{n}-grams seen once: {sum(1 for c in pairs.values() if c == 1) / len(pairs):.0%}")
```
Output:
```text
1-gram training perplexity: 676.4
1-grams seen once: 8%
2-gram training perplexity: 23.9
2-grams seen once: 60%
3-gram training perplexity: 4.9
3-grams seen once: 76%
4-gram training perplexity: 3.2
4-grams seen once: 82%
5-gram training perplexity: 2.9
5-grams seen once: 85%
```
The perplexity falls fast as the context grows: 676.4 with no context at all, 23.9 with one token, and 2.9 with four. At four tokens of context, the model is choosing among about three tokens at each step, because most n-grams that long occur only once: 82% of the 4-grams in the training names are seen a single time, and the model has memorized what followed each of them.
That's the trap. On the text it counted, a longer context always looks better, because with enough context every text is unique and the model can recite it. Scoring a model on its own training text measures memory, which is why [chapter 1](#predicting-the-next-word) set aside validation text in the first place.
**Working rule:** Never judge an n-gram model, or any language model, by its perplexity on its own training text: it falls toward 1 as the model memorizes. Choose the order by perplexity on held-out text, as the rest of this chapter does.
## Longer contexts run out of evidence {#longer-contexts-run-out}
Memorizing the training names says nothing about new ones. A model can only use a context it has counted, and the number of possible contexts grows exponentially with their length: with four thousand tokens, there are sixteen million possible pairs and sixty-four billion possible triples. No training text comes close to covering them. So the longer the context, the more often a new text uses one the model has never seen.
For the store, measure how much of the validation names the training names cover, order by order. Before you run it, predict: for 4-grams, what share of the validation names' 4-grams never occur in the training names?
Example `unseen-ngrams` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.names import name_tokens
from lib.ngrams import ngrams, padded
train, validation = name_tokens("train"), name_tokens("validation")
print("training tokens:", f"{sum(len(t) for t in train):,}")
print("validation tokens:", f"{sum(len(t) for t in validation):,}")
for n in [1, 2, 3, 4, 5]:
seen = {pair for text in train for pair in ngrams(padded(text, n), n)}
held_out = [pair for text in validation for pair in ngrams(padded(text, n), n)]
unseen = sum(pair not in seen for pair in held_out)
print(f"{n}-grams in validation never seen in training: {unseen / len(held_out):.1%}")
```
Output:
```text
training tokens: 337,071
validation tokens: 42,422
1-grams in validation never seen in training: 0.1%
2-grams in validation never seen in training: 12.7%
3-grams in validation never seen in training: 31.1%
4-grams in validation never seen in training: 43.9%
5-grams in validation never seen in training: 53.4%
```
The training names hold 337,071 tokens, and the validation names 42,422. Almost every validation token occurs somewhere in training: only 0.1% don't, which is the store's tokenizer doing its job. But 12.7% of the validation bigrams are unseen, 43.9% of the 4-grams, and 53.4% of the 5-grams. With four tokens of context, the model has never seen more than half of what it needs to predict.
More training text helps less than you might hope. In language, a few n-grams occur very often and most occur rarely, so each doubling of the text adds mostly n-grams seen once or twice, which barely improve the counts behind any one prediction. The store's catalog shows the same pattern in miniature: a common phrase such as `coffee table` is counted more than a thousand times, while 82% of the training names' 4-grams are counted once. Our judgement, not a measurement, is that a catalog ten times as large would still leave a large share of new 4-grams unseen, because new product lines and new combinations of sizes and materials keep appearing.
The two examples together are the core problem of counting. Longer contexts predict better when they've been seen, and are seen less and less often. Every method in the rest of this chapter is a way to use long contexts when the counts support them, and fall back to shorter ones when they don't.
**Working rule:** Before choosing an n-gram order, measure the share of held-out n-grams your training text never contains, as above. As a rule of thumb, where it nears a third, as it does here for trigrams, a model of that order depends mostly on how it handles the unseen, so its smoothing matters more than its order.
## Mix short and long contexts {#mix-short-and-long-contexts}
The obvious fix for unseen long contexts is to fall back on shorter ones. **Interpolation** does it smoothly: the probability of a token is a weighted mix of what the trigram, bigram and unigram counts say, so a long context the model has seen contributes its sharp prediction, and the shorter ones fill in what it hasn't. With weights λ₃, λ₂ and λ₁ that sum to one:
P(token | a b) = λ₃ · P₃(token | a b) + λ₂ · P₂(token | b) + λ₁ · P₁(token)
where each P is a maximum-likelihood estimate from counts, as in [chapter 1's probabilities](#counts-to-probabilities). Chen and Goodman's study of smoothing methods covers interpolation and its variants in detail ([Chen and Goodman, 1996](https://aclanthology.org/P96-1041/)).
For the store, build an interpolated trigram model by hand, with weights of 0.6, 0.3 and 0.09 for the three orders, and the last 0.01 spread evenly over every token, so no token ever gets probability zero. When a context was never seen, its weight passes to the orders below, so the probabilities still sum to one:
Example `interpolate-orders` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
from collections import Counter
from lib.names import name_tokens
from lib.ngrams import ngrams, padded, perplexity
train, validation = name_tokens("train"), name_tokens("validation")
counts = {k: Counter() for k in (1, 2, 3)} # counts[k][(the k-1 tokens before, token)]
for text in train:
for context, token in ngrams(padded(text, 3), 3):
for k in (1, 2, 3):
counts[k][(context[3 - k:], token)] += 1
totals = {k: Counter() for k in (1, 2, 3)}
for k in (1, 2, 3):
for (context, _), count in counts[k].items():
totals[k][context] += count
vocabulary = len({token for (_, token) in counts[1]}) + 1
weights = {3: 0.6, 2: 0.3, 1: 0.09} # and 0.01 spread evenly over every token
class Interpolated:
n = 3
def probability(self, context, token):
p, carried = 0.0, 0.0
for k in (3, 2, 1):
weight, total = weights[k] + carried, totals[k][context[3 - k:]]
if total: # an unseen context passes its weight down
p, carried = p + weight * counts[k][(context[3 - k:], token)] / total, 0.0
else:
carried = weight
return p + (0.01 + carried) / vocabulary
print("interpolated 3-gram validation perplexity:", round(perplexity(Interpolated(), validation), 1))
bigram_pairs = Counter((context[-1:], token) for text in train for context, token in ngrams(padded(text, 2), 2))
bigram_totals = Counter()
for (context, _), count in bigram_pairs.items():
bigram_totals[context] += count
class AddOne:
n = 2
def probability(self, context, token):
return (bigram_pairs[(context, token)] + 1) / (bigram_totals[context] + vocabulary)
print("add-one 2-gram validation perplexity:", round(perplexity(AddOne(), validation), 1))
```
Output:
```text
interpolated 3-gram validation perplexity: 31.4
add-one 2-gram validation perplexity: 218.6
```
The interpolated trigram model's validation perplexity is 31.4. For a fair comparison with [chapter 1's add-one smoothing](#smooth-unseen-pairs), the example also runs add-one on a bigram model of the same tokens, since perplexities over different tokens can't be compared: it scores 218.6. Add-one spreads most of its probability over pairs that never happen; mixing orders keeps what each order knows.
It has two weaknesses. The weights are fixed, and these were picked by hand, not tuned: the same 0.6 goes to the trigram whether its context was seen once or a thousand times, though a context seen a thousand times deserves more trust. And the unigram part gives each token probability in proportion to how often it occurs, which, as the next section shows, is the wrong question for a fallback.
**Working rule:** Never use a single n-gram order alone; always combine it with lower orders, so every token keeps some probability. Prefer methods that set each context's weight from its counts, such as Kneser-Ney below, over fixed weights. If you do use fixed weights, choose them on held-out text, not the text you counted.
## Count the contexts a token follows {#count-contexts-a-token-follows}
When a model backs off to a shorter context, it's asking a specific question: how likely is this token to show up in a context we haven't seen? Raw frequency answers a different one. A token can be very common and still occur only after one or two particular tokens, and such a token is a poor guess for a new context. The better measure is the number of different tokens it follows, its **continuation count**: a token that has followed many different tokens is likely to follow a new one too.
For the store, take the 300 most frequent tokens in the training names, and list those that follow the fewest different tokens for how often they occur, beside two ordinary product words:
Example `continuation-counts` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter, defaultdict
from lib.names import name_tokens
occurrences, before = Counter(), defaultdict(set)
for text in name_tokens("train"):
for previous, token in zip([""] + text, text):
occurrences[token] += 1
before[token].add(previous) # the different tokens this one follows
common = [t for t, _ in occurrences.most_common(300)]
narrow = sorted(common, key=lambda t: len(before[t]) / occurrences[t])[:3]
for token in narrow + ["table", "chair"]:
print(f"{token:<14} occurs {occurrences[token]:>5,} times, after {len(before[token]):>4,} different tokens")
```
Output:
```text
tvs occurs 323 times, after 1 different tokens
wide occurs 1,985 times, after 14 different tokens
'' occurs 10,335 times, after 85 different tokens
table occurs 3,487 times, after 154 different tokens
chair occurs 2,791 times, after 138 different tokens
```
The token `tvs` occurs hundreds of times, after only one other token: in the store's names it's always part of the same phrase, in products made for TVs. A model that backed off to raw frequency would think `tvs` a reasonable guess anywhere, and it almost never is. `wide` follows only a handful of tokens, mostly numbers' inch marks, and the inch mark itself, `''`, follows mostly digits. Meanwhile `table` and `chair` each follow well over a hundred different tokens: `coffee table`, `side table`, `outdoor table`. Their continuation counts say what their frequency doesn't: they turn up after well over a hundred different tokens.
Continuation counts are cheap to compute: for each token, keep the set of tokens seen before it, and count the set. They need no extra text and no tuning, only a different question asked of the same counts, and the next section builds on them.
This is the insight behind Kneser-Ney smoothing ([Kneser and Ney, 1995](https://doi.org/10.1109/ICASSP.1995.479394)): when the model falls back to a shorter context, it uses continuation counts instead of raw counts, so tokens that appear in many contexts get the probability, and tokens that appear in one fixed phrase don't.
**Working rule:** When you back off from a long context to a short one, weight tokens by how many different contexts they follow, not by how often they occur. Use raw counts only at the highest order, where the context has actually been seen.
## Smooth with Kneser-Ney {#smooth-with-kneser-ney}
Kneser-Ney smoothing combines [interpolation of orders](#mix-short-and-long-contexts) with [continuation counts](#count-contexts-a-token-follows). At each order, it subtracts a fixed **discount** d from every count it has seen, and gives the probability it took to the next order down, so a context seen often keeps nearly all its own prediction, and a context seen rarely leans on the shorter context. The same formula applies at every order; below the highest, the counts are continuation counts, except for n-grams that start a text, which keep their raw counts, as the KenLM toolkit does. In words: trust the counts you have, minus a little, and spend that little on tokens that turn up in many contexts. For a context h and a token w, with c(h w) the count of the pair, c(h) the count of the context, and N(h) the number of different tokens seen after h:
P(w | h) = max(c(h w) − d, 0) / c(h) + d · N(h) / c(h) · P(w | shorter h)
The book's shared module `lib.ngrams` implements interpolated Kneser-Ney as Chen and Goodman describe it ([Chen and Goodman, 1999](https://doi.org/10.1006/csla.1999.0128)), with a discount of 0.75, a common default. Its lowest order mixes in a uniform share over the vocabulary, so even a token never seen in training keeps a small probability.
Before trusting it, compute one probability by hand. For the store, take a name that starts `patio bar`, and work out the probability of `stool` next, order by order, with the formula above, beside the module's answer and the sum of the module's probabilities over every token:
Example `kneser-ney-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib import bpe
from lib.names import name_tokens
from lib.ngrams import START, KneserNey
model = KneserNey(name_tokens("train"), 4)
context = tuple([START] + bpe.store_tokenizer().encode("patio bar")) # a name that starts "patio bar"
token, d = "stool", model.d
def by_hand(context, k): # the chapter's formula, one order at a time
if k == 0:
return 1 / (len(model.vocabulary) + 1)
context = context[len(context) - (k - 1):] if k > 1 else ()
following, total = model.counts[k][context], model.totals[k].get(context, 0)
if total == 0:
return by_hand(context, k - 1)
kept = max(following[token] - d, 0) / total # the discounted count
passed_down = d * len(following) / total # what the discount frees for the order below
print(f"order {k}: kept {kept:.4f}, passed down {passed_down:.4f}")
return kept + passed_down * by_hand(context, k - 1)
print("by hand:", f"{by_hand(context, 4):.4f}")
print("lib.ngrams:", f"{model.probability(context, token):.4f}")
total = sum(model.probability(context, t) for t in model.vocabulary)
print("probabilities after the context, summed:", f"{total:.4f}")
```
Output:
```text
order 4: kept 0.2500, passed down 0.7500
order 3: kept 0.8056, passed down 0.1667
order 2: kept 0.0630, passed down 0.1275
order 1: kept 0.0007, passed down 0.0318
by hand: 0.8621
lib.ngrams: 0.8621
probabilities after the context, summed: 1.0000
```
At the 4-gram order the model keeps a share for `stool` and passes the rest down, and each lower order does the same, until the shares add up to 0.8621, the same as the module's 0.8621. Over every token in the vocabulary, the module's probabilities after this context sum to 1.0000, as a probability distribution must.
For the store, build Kneser-Ney models of every order from 1 to 5 on the training names, and score each on the validation names:
Example `kneser-ney-by-order` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.names import name_tokens
from lib.ngrams import KneserNey, perplexity
train, validation = name_tokens("train"), name_tokens("validation")
for n in [1, 2, 3, 4, 5]:
model = KneserNey(train, n)
print(f"{n}-gram validation perplexity: {perplexity(model, validation):.1f}")
model = KneserNey(train, 4)
top = [c for following in model.counts[4].values() for c in following.values()]
once, twice = top.count(1), top.count(2)
print("4-grams seen once:", f"{once:,}")
print("4-grams seen twice:", f"{twice:,}")
print("discount estimated from them:", f"{once / (once + 2 * twice):.2f}")
```
Output:
```text
1-gram validation perplexity: 687.2
2-gram validation perplexity: 46.9
3-gram validation perplexity: 24.9
4-gram validation perplexity: 23.1
5-gram validation perplexity: 23.5
4-grams seen once: 164,374
4-grams seen twice: 18,359
discount estimated from them: 0.82
```
The bigram model's validation perplexity is 46.9, the trigram's 24.9, the 4-gram's 23.1 and the 5-gram's 23.5. Kneser-Ney beats the hand-weighted interpolated trigram from the [interpolation section](#mix-short-and-long-contexts) at the same order, with no weights to choose, and longer contexts now help on new text instead of only on the old, up to a point. The step from trigram to 4-gram is small, and the 5-gram is worse than the 4-gram, because by then most contexts are unseen and the extra order adds noise. The discount, too, can be estimated from the counts: Chen and Goodman give D = n₁ / (n₁ + 2 n₂), where n₁ and n₂ are the numbers of n-grams seen once and twice, and on the store's 4-grams it's 0.82, close to the default.
**Working rule:** Smooth n-gram models with interpolated Kneser-Ney and a discount of 0.75 unless held-out text says otherwise. Raise the order only while validation perplexity keeps falling noticeably, and stop when a step barely helps or hurts, as from 4-gram to 5-gram here. A variant, modified Kneser-Ney, uses three discounts instead of one and is the one toolkits such as KenLM build; it would likely lower the baseline a little, and this book keeps the single discount because it's easier to follow by hand.
## Autocomplete with three tokens of context {#autocomplete-with-more-context}
A longer context should fix the search box's worst suggestion from [chapter 1](#predicting-the-next-word), where `patio bar` got the same suggestions as `bar`. The 4-gram Kneser-Ney model sees the last three tokens a shopper typed, so `patio` stays in view.
For the store, ask the 4-gram model for its three likeliest next tokens after three prefixes. Each prefix is treated as the start of a name, as a shopper's typing is. Each suggestion shows the token without its end-of-word marker:
Example `ngram-autocomplete` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib import bpe
from lib.names import name_tokens
from lib.ngrams import START, KneserNey
model = KneserNey(name_tokens("train"), 4)
tokenizer = bpe.store_tokenizer()
candidates = sorted(model.vocabulary)
for typed in ["bar", "patio bar", "outdoor bar"]:
context = tuple([START] * 3 + tokenizer.encode(typed))[-3:] # the last three tokens typed
ranked = sorted(candidates, key=lambda t: -model.probability(context, t))[:3]
print(f"{typed:<12} -> " + ", ".join(f"{t.replace('', '')} {model.probability(context, t):.2f}" for t in ranked))
names = [" ".join(t) for t in name_tokens("train")]
print("training names containing 'outdoor bar':", sum("outdoor bar" in n for n in names))
```
Output:
```text
bar -> & 0.23, unit 0.13, with 0.11
patio bar -> stool 0.86, & 0.03, with 0.02
outdoor bar -> & 0.23, with 0.19, stool 0.06
training names containing 'outdoor bar': 0
```
After `bar` alone, the model suggests continuations such as `&` and `with`, which belong to names that start like `bar & counter stools`; [chapter 1](#predicting-the-next-word)'s bigram, which ignored where the word sat, suggested `stool` first. After `patio bar`, it suggests `stool` with high probability: the context changed the prediction, which a bigram model can't do. After `outdoor bar`, though, the suggestions drift back toward `&` and `with`, because no training name contains `outdoor bar`, as the example's last line shows, and the model backs off to what follows `bar`.
Suggestions such as `&` and `with` are real continuations of the store's names, but a shopper rarely wants them offered. A search box would filter them out, as [chapter 1](#predicting-the-next-word) suggested for punctuation, or complete to whole phrases instead of single tokens. That's a choice about presentation; the model's probabilities are the same either way.
The `outdoor bar` line is the limit of counting in miniature. The model has seen `patio bar stool` and `outdoor patio` and `bar stool` many times, so it has the pieces to guess that an outdoor bar probably comes with stools. But a count of `outdoor bar stool` is a separate entry in a table, and nothing connects it to `patio bar stool`. A model needs some notion that `outdoor` and `patio` are alike, which counting doesn't have. [Chapter 4](#words-as-vectors) gives the store's tokens that notion.
**Working rule:** Use the longest context your model can support for suggestions, and check suggestions for contexts that are rare in training, such as `outdoor bar`, separately from common ones: that's where a counted model falls back to generic guesses.
## The baseline every later model is measured against {#the-baseline}
Every chapter from here on builds a model that learns, not one that counts, and each claims to predict the store's text better. A claim like that needs a fixed yardstick: one model, one text, one tokenizer and one measure that every later model is compared against. That's a **baseline**, and it should be the strongest simple model available, so that beating it means something.
The baseline for this book is the 4-gram Kneser-Ney model of this chapter, scored on the store's validation names with the store's tokenizer. Its validation perplexity is 23.1 per token, from the [Kneser-Ney example](#smooth-with-kneser-ney). The 5-gram model is worse, at 23.5, and holds more counts, so the 4-gram is the baseline: the best of this chapter's counted models on the store's names. A stronger counted baseline exists, modified Kneser-Ney, and the book doesn't use it, for the reason the [Kneser-Ney section](#smooth-with-kneser-ney) gives.
The baseline also sets expectations. A perplexity near twenty means the model is, on average, choosing among about twenty likely tokens at each step of a product name. Some steps are nearly certain, such as `table` after `coffee`, and some are wide open, such as the first token of a product line's name. A neural model that beats the baseline should do so by predicting the open steps better, because it can relate one context to another, not by memorizing more.
Two things about the baseline will stay fixed for the rest of the book. Every perplexity is per token of the store's tokenizer, on the validation names, so the numbers in every chapter can be set side by side. And the test names stay untouched until the book's final comparison, so nothing in the book is tuned on them.
**Working rule:** Before building a learned model, build the strongest counted model you can and record its held-out perplexity as the baseline. Report every later model against it, on the same text and with the same tokenizer, and treat a learned model that doesn't beat it as a result to explain, never as a success. Later chapters find exactly that, and earn the learned models their place another way.
## What goes wrong in production {#ngrams-in-production}
N-gram models are still useful where speed and transparency matter more than the last point of perplexity, such as fast autocomplete. These are the problems they bring:
- **The tables grow.** Every order adds another table of counts, and the number of distinct n-grams grows almost as fast as the text. Prune n-grams seen once, and measure validation perplexity after pruning, since some rare n-grams carry real information.
- **Rare contexts get generic answers.** As `outdoor bar` showed, a context the model rarely saw falls back to what the shorter context suggests. Monitor the suggestions for rare prefixes, not only the common ones.
- **The discount is a choice.** A discount of 0.75 is a sound default, not a law. If your text differs a lot from the store's, tune it on held-out text.
- **New text changes the counts.** Product lines come and go, and the counts behind every suggestion are only as current as the last rebuild. Rebuild on a schedule, and compare validation perplexity on recent text to the baseline to see when the model has gone stale.
- **The probabilities must sum to one.** Hand-written backoff schemes often lose or double-count probability, which makes perplexity look better or worse than it is. Check, on a few contexts, that each one's probabilities over the vocabulary sum to one.
## What this chapter decided {#ngrams-decisions}
- **Every model in the book is scored by perplexity per token on the store's validation names, through the store's tokenizer.** One text, one tokenizer and one measure make every chapter's numbers comparable on names, and the test names are kept for the book's final comparison, in [chapter 11](#look-back-at-the-yardstick).
- **N-gram models are smoothed with interpolated Kneser-Ney, discount 0.75.** It beat hand-weighted interpolation at the same order, with no weights to tune, because it trusts each context in proportion to its counts and backs off to tokens that follow many contexts.
- **The baseline is the 4-gram Kneser-Ney model,** at 23.1 per token on the validation names. The 5-gram model scores worse and holds more counts, so the 4-gram is the yardstick.
- **Autocomplete uses the 4-gram model's suggestions, with three tokens of context.** It fixes `patio bar`, and still falls back to generic suggestions for rare contexts such as `outdoor bar`, which is where a learned model has to do better.
- **Longer contexts stop helping because they're unseen, not because they don't matter.** More than half of the validation names' 5-grams never occur in training, and the model can't relate an unseen context to similar seen ones. [Chapter 4](#words-as-vectors) starts on that problem, by giving tokens vectors that say which ones are alike.
# Words as vectors
By the end of this chapter you can turn every token of the store's text into a short vector of numbers, so that tokens used in similar ways get similar vectors, by two methods: compressing counts of which tokens appear together, and training a small model to predict each token's neighbours. You'll measure what those vectors know, where they fail, and how they let a model relate text it has never seen to text it has.
[Chapter 3](#counting-further-back) ended on a limit that counting can't get past. Its 4-gram model had seen `patio bar stool` many times, and suggested `stool` after `patio bar`. After `outdoor bar`, which no training name contains, it fell back to generic suggestions, because a count of one context tells it nothing about another. A shopper, or any reader, knows that `outdoor` and `patio` mean nearly the same thing to a furniture store, and a model needs some way to know it too. This chapter gives every token a vector, a list of numbers, and arranges the vectors so that tokens used alike sit close together.
The store, the online furniture and home-goods store this book follows, has the text to learn this from: the names and descriptions of its catalog. Its training split, the 80% of products that models learn from, holds 2,744,320 tokens of it. Every vector in this chapter comes from that text alone, through the store's tokenizer from [chapter 2](#tokens).
## One-hot vectors know nothing {#one-hot-vectors-know-nothing}
Every model so far has treated tokens as names: `patio` is one entry in a table and `outdoor` another, with nothing linking them. Written as vectors, that's a **one-hot** encoding: each token gets a vector as long as the vocabulary, with a 1 in its own position and 0 everywhere else. To measure how alike two vectors are, this book uses **cosine similarity**: the cosine of the angle between them, 1 when they point the same way, 0 when they're at right angles. It ignores how long the vectors are and compares only their directions.
For the store, give the 2,000 most frequent tokens of its training text one-hot vectors, and compare three pairs. Keeping only the most frequent tokens, about half of the store tokenizer's vocabulary, holds every matrix in this chapter small enough to compute in seconds:
Example `one-hot-vectors` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.vectors import BREAK, training_ids
ids, vocabulary = training_ids()
index = {token: i for i, token in enumerate(vocabulary)}
def one_hot(token):
vector = np.zeros(len(vocabulary))
vector[index[token]] = 1.0 # one column per token: 1 for this token, 0 everywhere else
return vector
def cosine(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print("numbers in each vector:", len(vocabulary))
print("cosine(patio, outdoor):", cosine(one_hot("patio"), one_hot("outdoor")))
print("cosine(patio, sofa):", cosine(one_hot("patio"), one_hot("sofa")))
print("cosine(patio, patio):", cosine(one_hot("patio"), one_hot("patio")))
print("share of training tokens with a vector:", f"{(ids >= 0).sum() / (ids != BREAK).sum():.0%}")
```
Output:
```text
numbers in each vector: 2000
cosine(patio, outdoor): 0.0
cosine(patio, sofa): 0.0
cosine(patio, patio): 1.0
share of training tokens with a vector: 91%
```
Each vector has 2000 numbers. The cosine similarity of `patio` and `outdoor` is 0.0, and so is that of `patio` and `sofa`: a one-hot encoding says the two outdoor words are exactly as unrelated as a patio and a sofa. Only a token compared with itself scores 1.0. The 2,000 tokens that get vectors make up 91% of the training text; the rest are rarer, and this chapter leaves them out. Every pair of different tokens sits at right angles to every other, so nothing learned about one can carry over to another.
That's the same flaw as [chapter 3](#counting-further-back)'s counts, in another form. A model that reads one-hot vectors has to learn about every token separately, from the contexts that token itself appears in. Rare tokens get little, and unseen combinations get nothing.
**Working rule:** Don't feed one-hot vectors to a model that needs to relate tokens to each other. Use them only as an index, as the next sections do, to look up a learned vector for each token.
## Describe a token by the tokens around it {#describe-by-neighbours}
The way out is to describe a word by the company it keeps. Two words that appear among the same neighbours, such as `patio` and `outdoor`, which both turn up near `furniture`, `chair` and `umbrella`, are likely to mean similar things, even if they rarely appear together. So describe each token by its neighbours: count, for every pair of tokens, how often one appears within a small **window** of the other, here two tokens either side, and never across the gap between two products' texts. Each token's row of counts is then a vector of what it appears near.
For the store, count co-occurrences in the training split's names and descriptions, and list what appears most often near `patio`:
Example `cooccurrence-counts` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.vectors import BREAK, cooccurrence, training_ids
ids, vocabulary = training_ids()
counts = cooccurrence(ids, len(vocabulary)) # how often each token appears within 2 tokens of another
patio = vocabulary.index("patio")
print("training tokens:", f"{int((ids != BREAK).sum()):,}")
print("vocabulary kept:", f"{len(vocabulary):,}")
print("tokens near patio, counting repeats:", f"{int(counts[patio].sum()):,}")
for i in np.argsort(-counts[patio], kind="stable")[:6]:
print(f" near patio: {vocabulary[i]:<12} {int(counts[patio, i]):>6,}")
```
Output:
```text
training tokens: 2,744,320
vocabulary kept: 2,000
tokens near patio, counting repeats: 8,790
near patio: , 646
near patio: or 448
near patio: with 433
near patio: your 381
near patio: . 340
near patio: furniture 249
```
Within two places of `patio`, the training text holds 8,790 tokens, counting a token again each time it appears there. The most frequent of them are disappointing: a comma, `or`, `with`, `your` and a full stop, before the first word with anything to do with patios, `furniture`. Those tokens appear near almost everything, because they're everywhere. A row of raw counts mostly says how common each neighbour is, not what's special about the token whose row it is.
So raw counts make poor vectors. Two tokens that both appear often will look alike because they share the same frequent neighbours, commas and `with`, whatever they mean. What matters is which neighbours a token has more often than you'd expect from how common those neighbours are.
**Working rule:** Count co-occurrence within a small window, two to five tokens, in text the model will learn from, and never across document boundaries. Don't use the raw counts as vectors: reweight them, as the next section does, so frequent neighbours stop dominating.
## Weigh co-occurrence by surprise {#weigh-by-surprise}
The fix compares each count with what chance would predict. If `patio` and `furniture` appeared independently of each other, they'd co-occur about as often as their separate frequencies imply. **Pointwise mutual information**, or PMI, measures how far the real count departs from that: the log of the ratio between how often two tokens co-occur and how often they would by chance ([Church and Hanks, 1990](https://aclanthology.org/J90-1003/)). In symbols, for tokens a and b:
PMI(a, b) = log [ P(a, b) / (P(a) · P(b)) ]
where P(a, b) is the share of all co-occurrence counts that are of a and b together, and P(a) and P(b) are each token's share. PMI is positive when two tokens appear together more than chance predicts, and negative when less. Negative values are unreliable, since "less than chance" usually means "rarely seen at all", so **positive PMI**, or PPMI, replaces them with zero. The book's shared module `lib.vectors` computes it from the co-occurrence counts of the [previous section](#describe-by-neighbours), in three lines: the count chance would predict for each pair, the log of the real count over it, and zero wherever that's negative or the pair never occurs:
```python
expected = counts.sum(1, keepdims=True) * counts.sum(0, keepdims=True) / total
pmi = np.log(counts / expected)
return np.where(counts > 0, np.maximum(pmi, 0), 0.0)
```
For the store, reweight the counts with PPMI and list `patio`'s strongest neighbours again:
Example `ppmi-neighbours` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.vectors import cooccurrence, ppmi, training_ids
ids, vocabulary = training_ids()
weights = ppmi(cooccurrence(ids, len(vocabulary)))
patio = vocabulary.index("patio")
print("share of PPMI entries that are zero:", f"{(weights == 0).mean():.0%}")
for i in np.argsort(-weights[patio], kind="stable")[:6]:
print(f" PPMI with patio: {vocabulary[i]:<12} {weights[patio, i]:.2f}")
```
Output:
```text
share of PPMI entries that are zero: 86%
PPMI with patio: porch 4.51
PPMI with patio: deck 4.19
PPMI with patio: backyard 3.97
PPMI with patio: sectional 3.75
PPMI with patio: furniture 3.59
PPMI with patio: garden 3.53
```
The commas are gone. `patio`'s strongest neighbours by PPMI are `porch`, `deck`, `backyard`, `sectional`, `furniture` and `garden`: words a person would associate with a patio, ranked by how much more often than chance they appear near it. And 86% of the PPMI matrix is zero, because most pairs of tokens appear together no more often than chance.
PPMI vectors are already useful: two tokens with similar PPMI rows have similar distinctive neighbours. But each vector is still 2,000 numbers long, mostly zeros, and two tokens with related but not identical neighbours, say `deck` and `porch`, can share few nonzero entries.
**Working rule:** Reweight co-occurrence counts with positive PMI before using them as vectors. Expect PMI to overrate pairs of rare tokens, whose few co-occurrences look far above chance; that's one reason to keep only frequent tokens, as this chapter does, or to require a minimum count.
## Compress counts into dense vectors {#compress-into-dense-vectors}
A [PPMI vector](#weigh-by-surprise) of 2,000 numbers, mostly zeros, has a lot of room and little information. Compressing it into a short, dense vector keeps the strongest patterns of co-occurrence and drops the noise, and in the process puts tokens with related neighbours close together even when they share few exact neighbours. The tool is the **singular value decomposition**, SVD, which finds the 64 directions that together reconstruct the PPMI rows best, and weights each by how much of the rows it explains. A token's 64 weighted coordinates along those directions are its new vector: an **embedding**, a dense vector that stands for a token. The method `lib.vectors.svd_vectors` computes them, scales each to unit length, and `lib.vectors.nearest` finds the tokens with the highest cosine similarity to any vector.
For the store, compute the 64-number embeddings and list each of six tokens' nearest neighbours:
Example `svd-neighbours` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib.vectors import DIMENSIONS, nearest, svd_vectors
vectors, vocabulary = svd_vectors()
index = {token: i for i, token in enumerate(vocabulary)}
print("dimensions per token:", DIMENSIONS)
for token in ["patio", "outdoor", "sofa", "oak", "blue", "48"]:
neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,))
print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))
print("cosine(patio, outdoor):", f"{float(vectors[index['patio']] @ vectors[index['outdoor']]):.2f}")
```
Output:
```text
dimensions per token: 64
patio -> outdoor 0.84, bistro 0.79, porch 0.79, lounge 0.77
outdoor -> patio 0.84, porch 0.82, indoor 0.80, furniture 0.74
sofa -> sectional 0.93, chaise 0.90, loveseat 0.89, lounge 0.87
oak -> walnut 0.96, veneers 0.90, espresso 0.89, weathered 0.88
blue -> green 0.86, beige 0.86, pur 0.80, abstract 0.79
48 -> 36 0.94, 32 0.93, 24 0.92, 30 0.89
cosine(patio, outdoor): 0.84
```
Every token now has 64 numbers, and the neighbours read like a thesaurus built for a furniture store. `patio` and `outdoor` are each other's nearest neighbour, with a cosine similarity of 0.84, where their one-hot vectors scored zero. `sofa` sits beside `sectional`, `chaise` and `loveseat`; `oak` beside `walnut` and `espresso`, a wood finish; `blue` beside other colours; and `48` beside other sizes in inches.
None of that was taught. The method never saw a definition of `sofa`, a list of colours or a hint that `48` is a number. It found that these tokens appear among the same kinds of neighbours in the store's text, and the SVD put them close together because of it. That's the property [chapter 3](#counting-further-back)'s counts lacked: whatever a model learns about `patio bar stool` can now carry over, through the embeddings, to `outdoor bar stool`.
**Working rule:** Compress PPMI with SVD to between 50 and 300 dimensions, and check the result the way this section does: list a handful of tokens' nearest neighbours and read them. If the neighbours are mostly frequent function words, the reweighting went wrong; if they're unrelated, the window or the vocabulary is too small.
## Learn vectors by predicting neighbours {#learn-by-predicting-neighbours}
The other way to get embeddings is to learn them by prediction, which is how every neural language model in this book gets its own. **Skip-gram with negative sampling**, the method behind word2vec, gives each token two vectors, one for when it's the centre token and one for when it's a neighbour, and trains them on one task: given a token, tell its real neighbours apart from random tokens ([Mikolov et al., 2013a](https://arxiv.org/abs/1301.3781); [Mikolov et al., 2013b](https://arxiv.org/abs/1310.4546)). For a token and one of its real neighbours, the model scores the pair by the dot product of their vectors, turns the score into a probability with the logistic function, and nudges the vectors so the probability rises. For a few random **noise** tokens drawn from the vocabulary, it nudges the probability down. Repeated over millions of pairs, tokens with the same neighbours end up with similar vectors. That's no coincidence: Levy and Goldberg showed that skip-gram with negative sampling, trained this way, in effect factorizes a PMI matrix, shifted down by the log of the number of noise tokens, much as the [SVD](#compress-into-dense-vectors) factorizes PPMI ([Levy and Goldberg, 2014](https://papers.nips.cc/paper_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html)).
Write v for the centre token's vector, u for the real neighbour's, and uₙ for each noise token's, and σ for the logistic function, σ(s) = 1 / (1 + e⁻ˢ), which turns a score into a probability. The loss for one pair is the surprise at the real neighbour plus the surprise at each noise token's not being one:
L = −log σ(v·u) − Σₙ log σ(−v·uₙ)
and `lib.skipgram.step` lowers it with three lines of gradient descent, where `real` is σ(v·u), `fake` holds σ(v·uₙ) for each noise token, and `rate` is the step size:
```python
np.add.at(inputs, centre, -rate * grad_v) # move v: towards u, away from each uₙ
np.add.at(outputs, context, -rate * (real - 1)[:, None] * v) # move u towards v
np.add.at(outputs, noise, -rate * fake[:, :, None] * v[:, None, :]) # move each uₙ away from v
```
For the store, take one real pair, `patio` and `umbrella`, and two noise tokens, `sofa` and `48`. Start the centre vectors at small random values and the neighbour vectors at zero, as `lib.skipgram.train` does, and apply three updates:
Example `skipgram-step` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.skipgram import sigmoid, step
from lib.vectors import training_ids
_, vocabulary = training_ids()
rng = np.random.default_rng(2026)
inputs = (rng.random((len(vocabulary), 8)) - 0.5) / 8 # each token's vector as a centre, small and random
outputs = np.zeros((len(vocabulary), 8)) # and as a neighbour: zero, as training starts
centre, context = [vocabulary.index("patio")], [vocabulary.index("umbrella")]
noise = np.array([[vocabulary.index("sofa"), vocabulary.index("48")]])
def chances():
real = sigmoid(inputs[centre[0]] @ outputs[context[0]])
fake = sigmoid(outputs[noise[0]] @ inputs[centre[0]])
return real, fake
before = chances()
for _ in range(3): # three updates on the same pair
step(np.array(centre), np.array(context), noise, inputs, outputs, rate=1.0)
after = chances()
print("P(umbrella is a real neighbour of patio), before:", f"{before[0]:.3f}")
print("P(umbrella is a real neighbour of patio), after 3 steps:", f"{after[0]:.3f}")
print("P(sofa, 48 are real neighbours), before:", ", ".join(f"{p:.3f}" for p in before[1]))
print("P(sofa, 48 are real neighbours), after 3 steps:", ", ".join(f"{p:.3f}" for p in after[1]))
```
Output:
```text
P(umbrella is a real neighbour of patio), before: 0.500
P(umbrella is a real neighbour of patio), after 3 steps: 0.525
P(sofa, 48 are real neighbours), before: 0.500, 0.500
P(sofa, 48 are real neighbours), after 3 steps: 0.475, 0.475
```
Before training, the model gives every pair a probability of 0.500 of being real: the neighbour vectors are zero, so every score is zero, and σ(0) is one half. The model knows nothing. After three updates, `umbrella` scores 0.525 as a neighbour of `patio`, and `sofa` and `48` have dropped to 0.475, 0.475. That's all training does, millions of times over: pull real neighbours together and push random tokens apart.
**Working rule:** When you train embeddings, start the second set of vectors at zero and the first at small random values, as `lib.skipgram.train` does, and draw noise tokens in proportion to their frequency raised to the power 0.75, which Mikolov et al. found works better than plain frequency. Check one update by hand, as here, before running millions: a probability moving the wrong way means the gradient's sign is wrong.
## Compare learned vectors with counted ones {#compare-learned-and-counted}
Training skip-gram on all the store's training text runs millions of updates, and the exact result depends on the machine's floating-point arithmetic, which varies slightly between processors. So it runs once, in `recordings/skipgram.py`, with the book's own code and a fixed seed, and the examples read its saved vectors, as the book does with every result that depends on the machine. Its manifest records the settings: a window of two tokens, 64 dimensions, five noise tokens per pair and two passes over the text. One setting is fixed in `lib.skipgram` rather than recorded: pairs centred on very frequent tokens, such as commas, are mostly skipped, with a threshold of 10⁻⁴, which keeps them from dominating training ([Mikolov et al., 2013b](https://arxiv.org/abs/1310.4546), who suggest around 10⁻⁵ for larger text).
For the store, read the recorded vectors and list the same six tokens' neighbours as the [SVD's](#compress-into-dense-vectors):
Example `skipgram-neighbours` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import json
import numpy as np
from lib.vectors import nearest
recorded = "data/recorded/skipgram/"
vectors = np.load(recorded + "vectors.npy")
vocabulary = json.load(open(recorded + "vocabulary.json", encoding="utf-8"))
losses = np.load(recorded + "losses.npy")
index = {token: i for i, token in enumerate(vocabulary)}
print("training loss, first batch:", f"{losses[0]:.2f}")
print("training loss, last 100 batches:", f"{losses[-100:].mean():.2f}")
for token in ["patio", "outdoor", "sofa", "oak", "blue", "48"]:
neighbours = nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,))
print(f"{token:<12} -> " + ", ".join(f"{t} {s:.2f}" for t, s in neighbours))
```
Output:
```text
training loss, first batch: 4.16
training loss, last 100 batches: 2.05
patio -> porch 0.77, bistro 0.69, garden 0.69, couch 0.67
outdoor -> indoor 0.75, patio 0.66, garden 0.65, living 0.61
sofa -> couch 0.85, sleeper 0.84, chaise 0.80, sectional 0.78
oak -> walnut 0.91, espresso 0.84, weathered 0.81, glossy 0.80
blue -> beige 0.88, gray 0.87, green 0.85, brown 0.80
48 -> 36 0.94, 32 0.92, 1.5 0.89, 24 0.89
```
The training loss fell from 4.16 on the first batch to 2.05 on the last hundred: the model learned to tell real neighbours from noise. Its neighbours agree with the SVD's on most tokens, `sofa` with `sectional` and `chaise`, `oak` with `walnut` and `espresso`, colours with colours, sizes with sizes, and differ in emphasis on others: skip-gram puts `outdoor` closest to `indoor`, because the two words appear in the same positions in descriptions such as `indoor or outdoor use`.
That difference is worth noticing. Vectors built from neighbours put together words used in the same places, and opposites often are: `indoor` and `outdoor` here, and, in the [next section](#analogies-and-where-they-fail), `small`, whose nearest neighbours include `larger` as well as `smaller`. Similar vectors mean similar use, not similar meaning.
**Working rule:** Treat counted embeddings (PPMI and SVD) and learned ones (skip-gram) as close alternatives: on a few million tokens they give similar neighbours, and counted ones are deterministic and fast. Prefer learned embeddings when they're part of a larger model you're training anyway, as in every later chapter of this book. Never read a high similarity as synonymy without checking for opposites.
## Analogies, and where they fail {#analogies-and-where-they-fail}
Embeddings famously support analogies by arithmetic: if the difference between `chair` and `chairs` is a direction that means "plural", then adding it to `table` should land near `tables`. In vectors, the answer to "a is to b as c is to what?" is the token nearest to b − a + c, leaving out the three tokens in the question ([Mikolov et al., 2013a](https://arxiv.org/abs/1301.3781)). It's a neat test of whether the vectors have learned consistent relationships, and a good place to see their limits.
For the store, try three analogies with the [skip-gram vectors](#compare-learned-and-counted), and list the nearest neighbours of three tokens they raise:
Example `analogies` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import json
import numpy as np
from lib.vectors import nearest
vectors = np.load("data/recorded/skipgram/vectors.npy")
vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))
index = {token: i for i, token in enumerate(vocabulary)}
for a, b, c in [("chair", "chairs", "table"), ("chair", "chairs", "lamp"), ("black", "white", "dark")]:
a, b, c = (word + "" for word in (a, b, c))
query = vectors[index[b]] - vectors[index[a]] + vectors[index[c]] # b - a + c: "c, changed the way a became b"
answers = nearest(vectors, vocabulary, query, k=3, skip=(a, b, c))
print(f"{a} : {b} :: {c} : ? -> " + ", ".join(t for t, _ in answers))
print("lamps in the vocabulary:", "lamps" in index)
print("light in the vocabulary:", "light" in index)
for token in ["light", "dark", "small"]:
print(f"{token:<12} -> " + ", ".join(t for t, _ in nearest(vectors, vocabulary, vectors[index[token]], k=4, skip=(token,))))
```
Output:
```text
chair : chairs :: table : ? -> tables, cup, stools
chair : chairs :: lamp : ? -> tops, coffee, tray
black : white :: dark : ? -> brown, blue, glossy
lamps in the vocabulary: False
light in the vocabulary: True
light -> lights, bulb, dimmable, candle
dark -> brown, walnut, gray, espresso
small -> smaller, larger, displaying, storing
```
The first works: `chair` is to `chairs` as `table` is to `tables`, the top answer. The second fails, and the reason is in the last lines of the output: `lamps` isn't among the 2,000 tokens that have vectors, so the right answer can't be found at all, and the arithmetic returns whatever is nearest. The third fails differently. `light` is in the vocabulary, but `black` is to `white` as `dark` is to `brown`, by the vectors' reckoning. The neighbour lists show why: `light` sits with `lights`, `bulb`, `dimmable` and `candle`, because in the store's text it's mostly a lamp, as in `ceiling light`, not a shade, while `dark` sits with `brown`, `walnut` and `espresso`, the finishes it describes. And `small` sits with `larger` as well as `smaller`: size words appear in the same places, whichever size they name.
Analogies are a demonstration, not a measure. They work for relationships the text expresses consistently and often, such as plurals of common words, and fail for rare words, for words with several senses, and for relationships the text rarely spells out. A model doesn't use analogies; it uses the vectors' positions, and those carry the same strengths and blind spots.
**Working rule:** Don't judge embeddings by analogies; judge them by how much they help the task they're for, as the next section and every later chapter do. When an analogy fails, check whether the answer is in the vocabulary and whether the words mean what you assume in your text.
## Find similar names with vectors {#find-similar-names}
Embeddings for tokens give, with one more step, embeddings for whole texts: average the vectors of a text's tokens, scale the result to unit length, and similar texts get similar vectors. That's a rough version of what search engines call semantic retrieval, finding texts by meaning rather than by matching words, which this book's companion, *A Primer on Search Relevance*, builds properly with a trained model ([semantic retrieval](https://rollingedition.com/primer-on-search-relevance/semantic-retrieval)). Here it tests the one thing this chapter set out to fix: [chapter 3's](#autocomplete-with-more-context) counted model can't relate `outdoor bar` to `patio bar`.
For the store, average the skip-gram vectors of every training name, and find the names closest to two queries, one of them that problem prefix, `outdoor bar`:
Example `name-vectors` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import json
import numpy as np
from lib import bpe
from lib.store import load_products, masked
vectors = np.load("data/recorded/skipgram/vectors.npy")
vocabulary = json.load(open("data/recorded/skipgram/vocabulary.json", encoding="utf-8"))
index, tokenizer = {t: i for i, t in enumerate(vocabulary)}, bpe.store_tokenizer()
def name_vector(text): # the average of the text's token vectors, at unit length
found = [index[t] for t in tokenizer.encode(text) if t in index]
average = vectors[found].mean(0) if found else np.zeros(vectors.shape[1])
return average / (np.linalg.norm(average) or 1.0)
names = load_products("train")["name"].tolist()
matrix = np.array([name_vector(n) for n in names])
for query in ["outdoor bar", "patio umbrella"]:
best = np.argsort(-(matrix @ name_vector(query)), kind="stable")[:4]
print(f"{query}: " + " | ".join(masked(names[i]) for i in best)) # [name]: a rare word
```
Output:
```text
outdoor bar: outdoor modern industrial 5 piece bar height dining set | [name] bar & bar stool | 29.5 '' patio bar stool | 28.5 '' patio bar stool
patio umbrella: premium tight weave outdoor patio umbrella cover | patio lantern | garden patio 88 [name] rattan deck box | outdoor patio chair
```
The example prints a rare word, most often a product line's name, as `[name]`, as [chapter 1](#load-the-store-text) explained. For `outdoor bar`, the closest names include two `patio bar stool` names, neither of which contains the word `outdoor`. [Chapter 3](#counting-further-back)'s 4-gram model couldn't connect the two, because it had counts for `patio bar stool` and none for `outdoor bar`. The averaged vectors connect them through `outdoor` and `patio`, whose vectors are close. For `patio umbrella`, the closest names are patio and outdoor products, and the top one is an umbrella cover, but none of them is an umbrella itself: averaging lets `patio` outweigh `umbrella`.
Averaging is crude: it forgets word order, so `bar stool` and `stool bar` get the same vector, and it lets frequent tokens drown out the one that matters. But it shows the property every later model in this book is built on: a model that reads embeddings instead of one-hot tokens can carry what it learned about one phrase over to another.
**Working rule:** Average token embeddings when you need a quick, order-free representation of short texts, and expect it to find related items but rank them loosely. For real semantic retrieval, use a model trained for it, as the Primer does. For language modeling, give the model the embeddings themselves and let it learn how to combine them, which [chapter 5](#a-neural-language-model) starts doing.
## What goes wrong in production {#vectors-in-production}
Embeddings carry their training text's habits into everything that uses them:
- **Similar use isn't similar meaning.** `indoor` and `outdoor` sit close together. A system that treats close vectors as interchangeable will happily swap opposites; check the nearest neighbours of the words your product depends on.
- **Rare and new tokens get poor vectors, or none.** This chapter kept 2,000 tokens. A token outside that list has no vector, and a token seen a handful of times has a noisy one. The store's tokenizer has more than twice that many tokens, and a rare word that the tokenizer splits into pieces gets a vector only from those of its pieces that are among the 2,000; a model that learns its own embeddings, as every later chapter's does, covers the whole vocabulary.
- **Vectors reflect the text they came from.** The store's text describes `light` as a lamp, so its vector sits with lighting. Embeddings trained on one domain carry its senses, and its biases, into another.
- **Results depend on the machine.** Training embeddings by gradient descent gives slightly different vectors on different processors, which is why this book records them. Store the vectors you ship, with the code and settings that made them.
- **Embeddings go stale.** New product lines and new words appear in the catalog; retrain or extend embeddings when the vocabulary drifts, and check neighbour lists for tokens that matter after each retrain.
## What this chapter decided {#vectors-decisions}
- **Tokens are compared by the cosine similarity of their vectors.** It ignores length and compares direction, and it's what `lib.vectors.nearest` returns. Every vector is scaled to unit length first, so the cosine is a plain dot product and frequent tokens, whose raw vectors are longer, don't win by size.
- **Counted embeddings come from positive PMI compressed with SVD,** over a window of two tokens and the 2,000 most frequent tokens, in `lib.vectors`. On the store's text they put `patio` next to `outdoor`, which one-hot vectors couldn't.
- **Learned embeddings come from skip-gram with negative sampling,** trained once in `recordings/skipgram.py` with the book's own code and a fixed seed, because their exact values depend on the machine. They agree with the counted ones on most tokens.
- **Similar vectors mean similar use.** Opposites such as `indoor` and `outdoor` get close vectors, and analogies work only for relationships the text expresses often, so the book judges embeddings by how much they help a task, never by analogies.
- **Every later model learns its own embeddings,** as part of predicting the next token, starting with [chapter 5](#a-neural-language-model)'s neural language model. This chapter's vectors are the evidence that embeddings carry knowledge from one context to another; the models in the rest of the book put that to work.
# A neural language model
By the end of this chapter you can build a neural language model that predicts the next token from a window of earlier tokens, compute its gradients by hand and check them, train it with PyTorch, and measure it fairly against the counted baseline. You'll also see that a learned model doesn't win by being learned: this one loses to counting on its own, and helps when the two are mixed.
[Chapter 3](#counting-further-back) built the book's baseline, a 4-gram Kneser-Ney model that predicts each token of the store's product names from the three tokens before it, by counting. Its limit was that it can't relate one context to another: it has counts for `patio bar` and none for `outdoor bar`, and nothing links the two. [Chapter 4](#words-as-vectors) showed that vectors can relate tokens, since `patio` and `outdoor` get close vectors when they're learned from the store's text. This chapter puts vectors inside a model. A **neural language model** reads the previous tokens as vectors, combines them with learned arithmetic, and produces a probability for every token in the vocabulary. It learns its vectors and its arithmetic together, by being trained to predict the next token. The design is the one Bengio and colleagues introduced ([Bengio et al., 2003](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)).
The store, the online furniture and home-goods store this book follows, supplies the text, and the test is the one every model since [chapter 3](#counting-further-back) faces: [perplexity](#score-with-perplexity) per token on the store's validation names, through the store's tokenizer. Lower is better, and the baseline scores 23.1.
## Predict from a window of embeddings {#predict-from-a-window}
The model looks at a fixed **window** of the previous tokens, four here, the same context a 5-gram counts. Each token is looked up in an **embedding table**, a matrix with one row of numbers per token in the vocabulary, as in [chapter 4](#compress-into-dense-vectors), except that this table is learned for predicting the next token rather than computed from counts. The four rows are joined side by side into one long vector, x. A **hidden layer** computes h = tanh(xW + b): it multiplies x by a matrix of weights W, adds a vector of biases b, and passes each result through tanh, a function that squashes any number into the range from −1 to 1, so the layer can combine its inputs in ways sums alone can't. An **output layer** then computes z = hU + c, one score per token in the vocabulary, called the **logits**.
The last step turns scores into probabilities. **Softmax** exponentiates every logit and divides by the sum, so the results are positive and sum to one, and a higher score means a higher probability:
P(token i) = exp(zᵢ) / Σⱼ exp(zⱼ)
For the store, take the model the book trains in this chapter, load its weights, and run one prediction by hand: the next token after a name that starts `patio bar`. Each line of the example is one of the formulas above:
Example `window-model-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import bpe, neural
from lib.ngrams import START
weights, vocabulary, manifest = neural.load()
index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]
context = ([START] * width + bpe.store_tokenizer().encode("patio bar"))[-width:] # the last 4 tokens a shopper typed
ids = np.array([[index[t] for t in context]])
vectors = weights["embedding"][ids] # one 64-number vector per token
joined = vectors.reshape(1, -1) # the four vectors side by side
hidden = np.tanh(joined @ weights["hidden_w"] + weights["hidden_b"]) # one layer, squashed into -1 to 1
logits = hidden @ weights["output_w"] + weights["output_b"] # one score for every token
probabilities = np.exp(logits - logits.max())
probabilities /= probabilities.sum() # softmax: scores into probabilities
print("context:", " ".join(context))
for name, array in [("embeddings", vectors), ("joined", joined), ("hidden", hidden), ("logits", logits)]:
print(f"{name} shape:", array.shape)
print("vocabulary size:", f"{len(vocabulary):,}")
print("parameters:", f"{sum(w.size for w in weights.values()):,}")
for part in ["embedding", "hidden_w", "output_w"]:
print(f"parameters in {part}:", f"{weights[part].size:,}")
for i in np.argsort(-probabilities[0], kind="stable")[:3]:
print(f"P({vocabulary[i]} | patio bar): {probabilities[0, i]:.2f}")
```
Output:
```text
context: patio bar
embeddings shape: (1, 4, 64)
joined shape: (1, 256)
hidden shape: (1, 256)
logits shape: (1, 4140)
vocabulary size: 4,140
parameters: 1,394,732
parameters in embedding: 264,960
parameters in hidden_w: 65,536
parameters in output_w: 1,059,840
P(stool | patio bar): 0.63
P(with | patio bar): 0.23
P(& | patio bar): 0.05
```
The window is padded with start markers, because the name has only two tokens so far. Each of the four tokens becomes a vector of 64 numbers; joined, they make one vector of 256; the hidden layer turns that into 256 more; and the output layer gives one score for each of the 4,140 tokens the model knows, the store tokenizer's vocabulary plus the start and end markers. The model has 1,394,732 parameters, the numbers it learns. About three quarters of them are in the output layer's weights, 1,059,840, because it needs a column for every token, and most of the rest are in the embedding table. Its prediction is sensible: `stool` with probability 0.63, then `with` and `&`, which continue names such as `patio bar with stools`.
**Working rule:** Build the forward pass of any model you train by hand at least once, and print the shape of every intermediate array, as this example does: a wrong shape is a common bug in neural code, and the shapes say exactly what the model computes. Expect the output layer to hold most of the parameters whenever the vocabulary is large.
## Compute gradients by backpropagation {#compute-gradients}
A model with random weights predicts badly; training adjusts the weights to predict well. To measure how badly, use the **loss**: L = −log P(next token), the negative log probability the model gave to the token that actually came next, the same surprise [chapter 1](#score-with-perplexity) averaged into perplexity. Training lowers the average loss, and lower loss means lower perplexity. To lower it, each weight needs a **gradient**: how much the loss would change if that weight changed a little, written ∂L/∂w.
**Backpropagation** computes all the gradients at once, by applying the chain rule of calculus backwards through the forward pass: from the loss to the logits, from the logits through the output layer to the hidden layer, through tanh, and back to the embeddings that were used ([Rumelhart et al., 1986](https://doi.org/10.1038/323533a0)). Two of its steps are worth knowing by heart. The gradient of the loss with respect to the logits is the softmax probabilities minus 1 at the true token. And the gradient through tanh is multiplied by 1 − h², which is small where h is near ±1. The book's shared module `lib.tinynet` writes the whole of it out, line by line, for a tiny version of the store's model, with 5 tokens, embeddings of 2 numbers, a window of 2 and 3 hidden units.
Backpropagation code often goes subtly wrong, so check it against a slower method that can't: nudge one weight up and down by a tiny amount, measure how the loss moves, and compare. For the store's model that would mean more than a million nudges, so check every parameter of the tiny one:
Example `gradients-by-hand` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.tinynet import backprop, forward, parameters
p = parameters() # the tiny model: 5 tokens, a window of 2, 3 hidden units
context, target = np.array([1, 3]), 4
worst, step = 0.0, 1e-6
gradients = backprop(p, context, target)
for name, value in p.items(): # nudge each parameter both ways, and see how the loss moves
for i in np.ndindex(value.shape):
value[i] += step
up = forward(p, context, target)[3]
value[i] -= 2 * step
down = forward(p, context, target)[3]
value[i] += step
worst = max(worst, abs((up - down) / (2 * step) - gradients[name][i]))
print("loss:", round(float(forward(p, context, target)[3]), 4))
print("parameters checked:", sum(v.size for v in p.values()))
print("backprop agrees with nudging to 6 decimal places:", worst < 1e-6)
```
Output:
```text
loss: 1.5293
parameters checked: 45
backprop agrees with nudging to 6 decimal places: True
```
All 45 gradients from backpropagation agree with the nudged estimates to six decimal places: the last line reads True. The nudging method costs two forward passes per parameter, which is why no one trains with it, but on a tiny model it's the reference that tells you backpropagation is right.
**Working rule:** Check hand-written gradients against nudged estimates on a tiny model before trusting them, as here. When you use a library that computes gradients for you, as the [PyTorch section](#train-with-pytorch) does, check the library's forward pass against your own instead.
## Take gradient descent steps {#take-gradient-descent-steps}
With gradients in hand, training is a loop. **Gradient descent** moves every weight a small step against its gradient, w ← w − η · ∂L/∂w, where η, the **step size**, also called the **learning rate**, sets how far each step goes. Each step makes the loss a little smaller for the examples it was computed on, and repeated over many examples, the weights settle where the model predicts well.
For the store's tiny model from the [previous section](#compute-gradients), take three (window, next token) examples, compute the average loss, and take twenty steps of gradient descent with a step size of 0.5, using `lib.tinynet`'s gradients:
Example `gradient-descent-by-hand` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.tinynet import backprop, forward, parameters
p = parameters()
examples = [(np.array([1, 3]), 4), (np.array([0, 1]), 3), (np.array([3, 4]), 2)] # three (window, next token) pairs
rate = 0.5 # the step size
def average_loss():
return np.mean([forward(p, c, t)[3] for c, t in examples])
print("average loss before:", f"{average_loss():.3f}")
for step in range(1, 21):
for c, t in examples:
for name, gradient in backprop(p, c, t).items():
p[name] -= rate * gradient / len(examples) # w <- w - rate * dL/dw, averaged over the examples
if step in (1, 5, 20):
print(f"average loss after {step} steps:", f"{average_loss():.3f}")
```
Output:
```text
average loss before: 1.568
average loss after 1 steps: 1.346
average loss after 5 steps: 0.646
average loss after 20 steps: 0.109
```
The average loss falls from 1.568 before training to 1.346 after one step, 0.646 after five, and 0.109 after twenty. With only three examples to learn, the tiny model can drive its loss close to zero, which is memorizing; the store's model learns from hundreds of thousands of examples, where the same loop has to find patterns that hold across them.
Real training changes two things about this loop. It computes each step on a **batch** of examples, a few hundred at a time, instead of all of them, so each step is cheap and the steps vary a little. And it uses a refined update instead of a fixed step size, as the next section describes.
**Working rule:** Start any new model by running a few steps on a handful of examples and checking that the loss falls, as here: a loss that won't fall on three examples means a bug, not a hard problem. Choose the step size by trying a few: too small and the loss crawls, too large and it jumps around or grows.
## Train with PyTorch {#train-with-pytorch}
Writing backpropagation by hand for every model would be slow and error-prone. PyTorch, a widely used library for training neural networks, computes gradients automatically from the forward pass, and runs the arithmetic fast. The book trains its model with it, in `recordings/neural_lm.py`, which builds the same window model as the [first section](#predict-from-a-window): an embedding table, one tanh layer and an output layer, trained on the store's training names. The heart of the script is the loop of the [previous section](#take-gradient-descent-steps), on batches of 512 predictions:
```python
loss = loss_of(model(train_x[pick]), train_y[pick]) # forward pass and average loss on one batch
optimizer.zero_grad() # clear the previous batch's gradients
loss.backward() # backpropagation: a gradient for every weight
optimizer.step() # move every weight against its gradient
```
The optimizer is **Adam**, a refinement of gradient descent ([Kingma and Ba, 2015](https://arxiv.org/abs/1412.6980)). It keeps, for every weight, a running average of its gradients and a running average of their squares, and steps by the first divided by the square root of the second. A weight whose gradients are consistently large gets steps of about the same size as one whose gradients are small, so one step size works for all of them.
Training depends on the machine's floating-point arithmetic, which varies slightly between processors, so it runs once and the book's examples read the weights it saved, as they read every result that depends on the machine. The examples never import PyTorch: `lib.neural` recomputes the model's forward pass in NumPy, from those weights. That only works if the NumPy version computes what PyTorch computed, so the training script saved PyTorch's own logits for 64 validation contexts, and the next example compares:
Example `same-as-pytorch` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import neural
weights, _, manifest = neural.load()
check = np.load("data/recorded/neural-lm/check.npz") # contexts, and PyTorch's logits for them
ours = neural.forward(weights, check["contexts"])
print("PyTorch version:", manifest["torch"])
print("contexts compared:", len(check["contexts"]))
print("logits compared:", f"{ours.size:,}")
print("same as PyTorch to 4 decimal places:", bool(np.abs(ours - check["logits"]).max() < 1e-4))
```
Output:
```text
PyTorch version: 2.14.1+cpu
contexts compared: 64
logits compared: 264,960
same as PyTorch to 4 decimal places: True
```
The NumPy forward pass reproduces all 264,960 of PyTorch's logits to four decimal places, for the model trained with PyTorch 2.14.1+cpu. Every number this chapter reports about the trained model comes from that NumPy code, except the training curve, which the script recorded as it trained.
**Working rule:** Train with a library that computes gradients for you, and keep a short, independent forward pass of your own to check its outputs, as `lib.neural` does here. Record the library's outputs for a few inputs whenever you save weights, so the check can run anywhere, long after training.
## Stop when validation loss turns up {#stop-at-the-best-epoch}
A model with 1.4 million parameters, trained on a few hundred thousand tokens, can memorize its training text, as [chapter 3's 5-gram](#longer-contexts-fit-training) did. Training perplexity keeps falling as it does. Validation perplexity falls at first, while the model learns patterns that hold for new names, then rises, once it starts fitting quirks of the training names. That turn is **overfitting**, and the simplest defence is **early stopping**: measure validation loss after every **epoch**, one full pass over the training text, and keep the weights from the best epoch.
For the store, read the training script's record of both perplexities after each of its twelve epochs:
Example `training-curve` (Python 3.12.15), passing in Edition 2026.10.3:
```python
import json
import math
curves = json.load(open("data/recorded/neural-lm/curves.json", encoding="utf-8"))
manifest = json.load(open("data/recorded/neural-lm/manifest.json", encoding="utf-8"))
print(f"{'epoch':>5} {'training perplexity':>19} {'validation perplexity':>21}")
for c in curves:
print(f"{c['epoch']:>5} {math.exp(c['train_loss']):>19.1f} {math.exp(c['validation_loss']):>21.1f}")
best = min(curves, key=lambda c: c["validation_loss"])
print("epoch kept:", manifest["kept_epoch"])
print("validation perplexity at the kept epoch:", f"{math.exp(best['validation_loss']):.1f}")
print("validation perplexity at the last epoch:", f"{math.exp(curves[-1]['validation_loss']):.1f}")
```
Output:
```text
epoch training perplexity validation perplexity
1 109.3 51.4
2 37.6 39.9
3 27.3 37.3
4 22.4 36.6
5 19.3 36.4
6 17.2 36.9
7 15.6 37.8
8 14.4 38.6
9 13.5 39.7
10 12.7 40.9
11 12.0 42.1
12 11.5 43.7
epoch kept: 5
validation perplexity at the kept epoch: 36.4
validation perplexity at the last epoch: 43.7
```
In epoch 1, training perplexity is higher than validation, because the training figure averages over the whole epoch, including its first batches, when the model was still untrained, while validation is measured at the epoch's end. After that, training perplexity falls at every epoch, while validation perplexity bottoms out at 36.4 in epoch 5 and climbs from there, to 43.7 by the last epoch. The training script kept the weights from the best epoch, and every result in the rest of this chapter uses them.
The gap between the two curves is the model memorizing. It grows every epoch, because the training names repeat their product lines and phrasing, and this model has no **regularization**, nothing in its training that discourages memorizing, so early stopping is its only defence.
**Working rule:** Always record validation loss per epoch and keep the best epoch's weights, never the last. When the curves part early, as they do here after a few epochs, the model has more capacity than its training text supports: regularize it, use a smaller model, or get more text of the same kind.
## Compare with the baseline {#compare-with-the-baseline}
The real test is the one every model in this book takes: perplexity per token on the validation names, against the [4-gram baseline](#the-baseline). Before you run the comparison, predict which model wins. The neural model can relate similar contexts, which the 4-gram can't; the 4-gram has exact counts for every context it has seen.
Example `neural-vs-baseline` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib import neural
from lib.names import name_tokens
from lib.ngrams import KneserNey, perplexity
train, validation = name_tokens("train"), name_tokens("validation")
print("window model validation perplexity:", f"{neural.perplexity(validation):.1f}")
print("4-gram baseline validation perplexity:", f"{perplexity(KneserNey(train, 4), validation):.1f}")
```
Output:
```text
window model validation perplexity: 36.4
4-gram baseline validation perplexity: 23.1
```
The window model's validation perplexity is 36.4, against the baseline's 23.1. The counted model wins, by a wide margin.
To see where, split the validation tokens by whether their three-token context occurs in the training names, and score both models on each part:
Example `seen-and-unseen-contexts` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
import numpy as np
from lib import neural
from lib.names import name_tokens
from lib.ngrams import KneserNey, ngrams, padded
train, validation = name_tokens("train"), name_tokens("validation")
counted = KneserNey(train, 4)
pairs = [(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)]
seen = np.array([c in counted.totals[4] for c, _ in pairs]) # was this 3-token context in the training names?
from_counts = np.array([counted.probability(c, t) for c, t in pairs])
from_network = neural.token_probabilities(validation)
print("validation tokens with a context seen in training:", f"{seen.mean():.0%}")
for part, where in [("seen", seen), ("unseen", ~seen)]:
for model, p in [("4-gram", from_counts), ("network", from_network)]:
print(f"{model} perplexity, {part} contexts: {math.exp(-np.log(p[where]).mean()):.1f}")
```
Output:
```text
validation tokens with a context seen in training: 70%
4-gram perplexity, seen contexts: 17.9
network perplexity, seen contexts: 26.1
4-gram perplexity, unseen contexts: 41.2
network perplexity, unseen contexts: 78.5
```
70% of the validation tokens have a context the training names contain, and on those the 4-gram scores 17.9 and the network 26.1. On the unseen contexts, where the network's ability to relate contexts should help most, the 4-gram scores 41.2 and the network 78.5. The network is worse on both, and by more on the unseen ones.
So this network doesn't lose only where counts are exact; it hasn't learned to generalize well either. Two features of this model and this text are the likeliest reasons. The network has no regularization and overfits after a few epochs, as the [training curve](#stop-at-the-best-epoch) showed. And the store's names are short and formulaic: three tokens of context cover most of what a name holds, which is the setting where Kneser-Ney's counts are strongest. Bengio and colleagues found the opposite on larger, more varied text: on the Brown corpus, about 800,000 training words, their best network alone had a test perplexity of 268, and mixed with a trigram model 252, against the best n-gram model's 312 ([Bengio et al., 2003](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf), section 4.2 and table 1). A learned model isn't better because it's learned; it has to be measured.
**Working rule:** Always compare a learned model with a strong counted baseline on the same text and tokens before claiming it helps, and split the comparison by seen and unseen contexts to see where each model wins. Don't conclude from one small dataset that neural models don't work, or that they always do.
## Mix the network with the counts {#mix-with-the-counts}
A model that's worse on average can still be useful, if its mistakes differ from the better model's. When two models make different mistakes, mixing their probabilities can beat both, which is what Bengio and colleagues did with their network and a smoothed trigram model ([Bengio et al., 2003](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf), section 4). The mix is a weighted average of the two models' probabilities for each token, as in [chapter 3's interpolation](#mix-short-and-long-contexts).
For the store, score the validation names with the network's share of the mix at 0, 0.2, 0.3, 0.5 and 1, where 0 is the counted model alone and 1 is the network alone:
Example `mix-neural-and-counts` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
import numpy as np
from lib import neural
from lib.names import name_tokens
from lib.ngrams import KneserNey, ngrams, padded
train, validation = name_tokens("train"), name_tokens("validation")
from_network = neural.token_probabilities(validation)
counted = KneserNey(train, 4)
from_counts = np.array([counted.probability(c, t) for text in validation for c, t in ngrams(padded(text, 4), 4)])
for share in [0.0, 0.2, 0.3, 0.5, 1.0]: # the network's share of the mix; the counts get the rest
mixed = share * from_network + (1 - share) * from_counts
print(f"validation perplexity, network share {share:.1f}: {math.exp(-np.log(mixed).mean()):.1f}")
```
Output:
```text
validation perplexity, network share 0.0: 23.1
validation perplexity, network share 0.2: 21.9
validation perplexity, network share 0.3: 22.1
validation perplexity, network share 0.5: 22.9
validation perplexity, network share 1.0: 36.4
```
The mix beats both models. With a network share of 0.2, the validation perplexity is 21.9, better than the baseline's 23.1 and far better than the network's 36.4 alone. The network is worse on average, but where the counts give a token very little probability, the network's share keeps it from being tiny, and that lowers the average surprise.
The kept epoch and the share were both chosen by looking at the validation names, so the gain is slightly optimistic; the book's final comparison, on the test names, will check it. And the mix only helps because the two models' errors differ: if they made the same mistakes, averaging them would gain nothing.
**Working rule:** When a learned model and a counted one are close, try mixing their probabilities before choosing one, and choose the share on held-out text. Keep the share small when one model is much stronger, as here, and confirm the gain on a separate test set before relying on it.
## More text helps only if it's the same kind {#same-kind-of-text}
If the network overfits because it learned from too little text, the obvious fix is more text. The store has plenty: its product descriptions. The training script can train the same model on names and descriptions together, and the book recorded that version too, scored on the same validation names.
For the store, compare the two recorded models on the validation names:
Example `more-text-of-another-kind` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import json
import math
from lib import neural
from lib.names import name_tokens
validation = name_tokens("validation")
for name in ["neural-lm", "neural-lm-all-text"]:
manifest = json.load(open(f"data/recorded/{name}/manifest.json", encoding="utf-8"))
print(f"learned from {manifest['settings']['text']}:", f"{neural.perplexity(validation, name):.1f}")
from lib import bpe
from lib.store import load_products
tokenizer = bpe.store_tokenizer()
names_tokens = sum(len(t) for t in name_tokens("train"))
description_tokens = sum(len(tokenizer.encode(d)) for d in load_products("train")["description"])
print("training tokens, names:", f"{names_tokens:,}")
print("training tokens, names and descriptions:", f"{names_tokens + description_tokens:,}")
curve = json.load(open("data/recorded/neural-lm-all-text/curves.json", encoding="utf-8"))
print("all-text model, validation perplexity by epoch:", ", ".join(f"{math.exp(c['validation_loss']):.1f}" for c in curve))
```
Output:
```text
learned from names: 36.4
learned from all: 59.0
training tokens, names: 337,071
training tokens, names and descriptions: 2,744,320
all-text model, validation perplexity by epoch: 73.6, 64.0, 60.8, 59.0
```
The training names hold 337,071 tokens, and names and descriptions together 2,744,320, about eight times as many. The model that learned from names alone scores 36.4, and the one that learned from names and descriptions scores 59.0: worse, with eight times the text. That second model trained for four epochs, and its validation perplexity was still falling when it stopped, by smaller steps each epoch: 73.6, 64.0, 60.8, 59.0. Our judgement is that more epochs wouldn't bring it near the names model.
Descriptions are a different kind of text. They're written in full sentences, with words such as `your`, `this` and `perfect` that names rarely contain, and a model that learns mostly from them predicts names as if they were descriptions. The added text taught the model the wrong distribution for the task. A model learns the text it's given, and its perplexity measures how well it predicts text like that: data of another kind can make it worse, and needs to be weighted or kept apart.
**Working rule:** Before adding training text, check that it resembles the text you'll predict; train on both and measure on the target text, as here, rather than assuming more is better. When the extra text differs a lot, train on the target text alone, or lay out each text so the model can tell its kinds apart, as [chapter 8](#turn-products-into-batches) does with a separator between a product's name and its description.
## Suggest from similar contexts {#suggest-from-similar-contexts}
The network lost on average, including on contexts the training names lack, but averages hide single cases. [Chapter 3's autocomplete](#autocomplete-with-more-context) showed the 4-gram's worst case, `outdoor bar`, a context no training name contains, where it fell back to generic continuations.
For the store, ask the network for its three likeliest next tokens after five prefixes, each treated as the start of a name, and compare a few of its embeddings:
Example `neural-autocomplete` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import bpe, neural
from lib.ngrams import START
weights, vocabulary, manifest = neural.load()
index, width = {t: i for i, t in enumerate(vocabulary)}, manifest["settings"]["context"]
tokenizer = bpe.store_tokenizer()
for typed in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:
context = ([START] * width + tokenizer.encode(typed))[-width:]
logits = neural.forward(weights, np.array([[index[t] for t in context]]))[0]
probabilities = np.exp(neural.log_softmax(logits[None])[0])
best = np.argsort(-probabilities, kind="stable")[:3]
print(f"{typed:<12} -> " + ", ".join(f"{vocabulary[i].replace('', '')} {probabilities[i]:.2f}" for i in best))
outdoor_bar = ([START] * width + tokenizer.encode("outdoor bar"))[-width:]
p_stool = np.exp(neural.log_softmax(neural.forward(weights, np.array([[index[t] for t in outdoor_bar]])))[0, index["stool"]])
print("P(stool | outdoor bar):", f"{p_stool:.2f}")
table = weights["embedding"] / np.linalg.norm(weights["embedding"], axis=1, keepdims=True)
for a, b in [("patio", "outdoor"), ("patio", "kitchen"), ("patio", "wine")]:
print(f"cosine({a.replace('', '')}, {b.replace('', '')}) in the model's embeddings:", f"{float(table[index[a]] @ table[index[b]]):.2f}")
```
Output:
```text
bar -> with 0.22, cabinet 0.14, cart 0.13
patio bar -> stool 0.63, with 0.23, & 0.05
outdoor bar -> stool 0.40, with 0.13, & 0.12
kitchen bar -> & 0.53, with 0.17, cart 0.09
wine bar -> with 0.23, cabinet 0.16, 0.14
P(stool | outdoor bar): 0.40
cosine(patio, outdoor) in the model's embeddings: 0.16
cosine(patio, kitchen) in the model's embeddings: 0.07
cosine(patio, wine) in the model's embeddings: 0.10
```
After `outdoor bar`, the network suggests `stool` first, with probability 0.40, where the 4-gram put `&` and `with` ahead of it. The control prefixes show that `stool` isn't automatic after `bar`: after `bar` alone, `kitchen bar` and `wine bar`, it isn't among the network's three suggestions. And in the network's own embedding table, `patio` is a little closer to `outdoor`, with a cosine similarity of 0.16, than to `kitchen` or `wine`. That's consistent with the network having carried something from `patio bar` over to `outdoor bar`, though the similarity is weak, and one prefix proves little.
This is one case, not an average: the [seen and unseen split](#compare-with-the-baseline) showed the network worse on unseen contexts overall. It shows what a learned model can do that counting can't, and that this one does it only sometimes.
**Working rule:** Look at a learned model's predictions on contexts the training text lacks, with control prefixes beside them, but judge it by measurements over many contexts, not by examples. For autocomplete in production, use the mix, and test it on rare prefixes as well as common ones.
## What goes wrong in production {#neural-in-production}
A neural language model brings new failure modes with its new strengths:
- **Overfitting is silent.** Training loss always looks good. Without a validation curve, a model trained too long ships worse than one trained less. Keep the best epoch, and watch the gap.
- **Results depend on the machine.** Retraining on another processor gives slightly different weights. Save the weights you ship, the code and settings that made them, and the library's outputs for a few inputs, as the training script here does.
- **The output layer dominates.** Scoring every token in the vocabulary at every step is most of the computation and most of the parameters. A larger vocabulary costs here first.
- **Mismatched training text hurts.** Adding text unlike the target, as descriptions were for names, makes predictions worse while the training loss looks fine. Measure on the target text after every change to the training data.
- **Averages hide rare contexts.** A model can score well overall and still fail on the unusual prefixes that shoppers type when they look for something new. Score seen and unseen contexts separately, as this chapter did, and watch the unseen ones.
- **A mix needs both models at serving time.** Mixing a network with counts doubles what has to be loaded and kept current. Rebuild the counts whenever you retrain the network, so the two describe the same text.
## What this chapter decided {#neural-decisions}
- **The book's neural models are trained in PyTorch, once, in `recordings/`,** and the examples recompute them in NumPy from the saved weights. The NumPy forward pass reproduces PyTorch's logits to four decimal places, so the examples need no PyTorch.
- **Models keep the weights from their best validation epoch.** The window model's validation perplexity was lowest after epoch 5, and rose every epoch after, while training perplexity kept falling: the model was memorizing the training names.
- **On the store's names, the window model loses to the 4-gram baseline alone, on seen and unseen contexts alike, and beats it mixed.** A network share of 0.2 gives 21.9, the best of the book's models so far.
- **Learned models are compared with the baseline on seen and unseen contexts separately,** not only on average, because the two parts can tell different stories. Here the network lost on both, and by more on the unseen ones.
- **Training text must resemble the target text.** Adding the descriptions made the model worse at names, though names and descriptions together held about eight times as many tokens as the names alone, so every model's training text is chosen for the text it will be scored on.
- **Names are short, so from [chapter 8](#training-a-small-model) the book also measures models on the store's descriptions.**
# Attention
By the end of this chapter you can compute attention by hand: score how well each token's query matches every earlier token's key, turn the scores into weights, and average the values those weights select. You'll add the causal mask that keeps a language model from seeing the future, the scaling that keeps the weights from collapsing onto one token, and several heads that look for different things at once. Then you'll read what a trained model attends to, test what it copies, and measure how far back it looks.
This book follows one case throughout: an online furniture and home-goods store, which it calls the store, and the text of its product names and descriptions. [Chapter 5](#a-neural-language-model)'s window model predicted the next token of that text from exactly four earlier tokens, joined side by side, each slot with its own weights. That design has two limits. It can't see a fifth token back, however much it matters, and it learns about a token in the first slot separately from the same token in the third. The store's descriptions show why the first limit hurts: a description often repeats words from the product's name, tens of tokens later, as in `protect your outdoor patio furniture from harsh elements with this outdoor patio furniture cover`. A model that can look back at the name can see which word comes next. One that sees only the last four tokens, `this outdoor patio furniture`, has to guess among everything that follows those words in other products.
**Attention** lets every position look at every earlier one, and decide from their contents how much each matters. It was introduced to let a translation model look back at the words of the sentence it was translating ([Bahdanau et al., 2015](https://arxiv.org/abs/1409.0473)), and it is the core of the transformer ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)), the design of the open models this book reads in [chapter 10](#inside-an-open-model). This chapter builds attention on its own, then reads it inside the book's small transformer, which this book calls the store's model: a transformer trained once on the store's text, which [chapter 7](#the-transformer-block) assembles part by part.
## Look back with a weighted average {#look-back-with-a-weighted-average}
Attention replaces [chapter 5](#a-neural-language-model)'s fixed slots with a weighted average. At each position, the model computes a new vector as an average of vectors from the positions it can see, with weights it chooses from their content: high for the positions that matter here, low for the rest. The weights come from three vectors that each token's embedding is turned into, by three learned matrices:
- a **query**, what this position is looking for,
- a **key**, what this position offers to the positions that look at it,
- a **value**, what this position passes on when it's looked at.
The score of position j for position i is the dot product of i's query with j's key, divided by the square root of d, the size of each query and key, meaning how many numbers it holds. Softmax turns each row of scores into weights that are positive and sum to one, and the output at position i is the weighted sum of the values. In matrix form, for all positions at once:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
where Q, K and V hold every position's query, key and value as rows ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 3.2.1). The division by √d is the subject of a [later section](#scale-the-scores).
For the store, take four tokens, the start marker and `sled coffee table`, a table whose base is shaped like a sled's runners. Give each a random vector of 8 numbers, as an untrained embedding table would, and compute attention with random 8-by-4 query, key and value matrices:
Example `attention-by-hand` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
tokens = ["", "sled", "coffee", "table"]
rng = np.random.default_rng(2026)
x = rng.normal(0, 1, (4, 8)) # one 8-number vector per token, as an embedding table gives
w_query, w_key, w_value = (rng.normal(0, 0.5, (8, 4)) for _ in range(3))
queries, keys, values = x @ w_query, x @ w_key, x @ w_value # what each token looks for, offers, and passes on
scores = queries @ keys.T / np.sqrt(4) # how well each query matches each key, scaled
weights = np.exp(scores - scores.max(1, keepdims=True))
weights /= weights.sum(1, keepdims=True) # softmax along each row: each row sums to 1
output = weights @ values # each token: a weighted average of all the values
np.set_printoptions(precision=2, suppress=True)
print("attention weights (rows attend to columns):")
for token, row in zip(tokens, weights):
print(f" {token:<11}", row)
print("each row sums to 1:", bool(np.allclose(weights.sum(1), 1)))
print("output shape:", output.shape)
```
Output:
```text
attention weights (rows attend to columns):
[0.02 0.97 0.01 0. ]
sled [0.21 0.52 0.16 0.11]
coffee [0.04 0. 0.03 0.94]
table [0.16 0. 0.03 0.81]
each row sums to 1: True
output shape: (4, 4)
```
Each row of the printed weights is one position's average: which positions it draws from, and how much. Every row sums to one, as the line `each row sums to 1: True` confirms, and the output has one 4-number vector per position. The weights are random, because the matrices are, but they already show attention's freedom: `coffee` puts most of its weight on `table`, and the start marker on `sled`, because nothing restricts which positions a row can draw from. For predicting the next token, that freedom is a bug, as the [next section](#hide-the-future) shows.
Three properties of this arithmetic matter for everything that follows. The weights depend on content, through queries and keys, so the same token can attend to different places in different texts. The number of positions doesn't matter: the same three matrices work for a name of 4 tokens or a description of 150, so the model learns one way of looking back, not one per slot. And a weighted average ignores the order of what it averages. Without a mask, attention gives `bar` the same output in `bar stool` as in `stool bar`. The causal mask of the next section lets a little order back in, because each token then sees a different number of earlier tokens, and Haviv and colleagues found that language models can infer positions from that alone ([Haviv et al., 2022](https://arxiv.org/abs/2203.16634)). Models still add explicit position information, as [chapter 7](#the-transformer-block) does.
**Working rule:** Write out attention's four lines once, as this example does, and print the weight matrix: its rows say exactly where each position looks. Keep queries, keys and values as separate matrices while you're learning. Libraries often keep them in one matrix, as PyTorch's `MultiheadAttention` does with its `in_proj_weight` ([source](https://github.com/pytorch/pytorch/blob/v2.14.1/torch/nn/modules/activation.py)), and split the result, which computes the same thing.
## Hide the future with a causal mask {#hide-the-future}
In the [first section](#look-back-with-a-weighted-average), `coffee` drew most of its average from `table`, a later token. A language model can't be allowed to do that, because it predicts each token from the tokens before it. If position i could attend to position i + 1, it would see the answer: training would reward copying the next token, and the model would learn nothing it could use when generating text, where the next token doesn't exist yet. The fix is the **causal mask**: before softmax, set every score for a later position to minus infinity. The exponential of minus infinity is zero, so those positions get a weight of zero, and each position averages only over itself and the positions before it.
The mask also makes training efficient. With it, one pass over a text predicts every position at once, each from its own past, where [chapter 5](#a-neural-language-model)'s window model needed a separate window for every prediction.
For the store, apply the mask to the same four tokens' scores. Then test what the mask is for: replace the last token, `table`, with a different vector, and check whether the output at `coffee` changes, without the mask and with it:
Example `causal-mask` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
tokens = ["", "sled", "coffee", "table"]
rng = np.random.default_rng(2026)
x = rng.normal(0, 1, (4, 8))
w_query, w_key, w_value = (rng.normal(0, 0.5, (8, 4)) for _ in range(3)) # the same draws as attention-by-hand
allowed = np.tril(np.ones((4, 4), dtype=bool)) # position i may look at positions 0 to i only
def attend(x, mask):
scores = (x @ w_query) @ (x @ w_key).T / np.sqrt(4)
if mask:
scores = np.where(allowed, scores, -np.inf) # a score of minus infinity becomes a weight of 0
weights = np.exp(scores - scores.max(1, keepdims=True))
weights /= weights.sum(1, keepdims=True)
return weights, weights @ (x @ w_value)
weights, _ = attend(x, mask=True)
np.set_printoptions(precision=2, suppress=True)
print("weights with the causal mask:")
for token, row in zip(tokens, weights):
print(f" {token:<11}", row)
print("weight on later tokens:", float(weights[~allowed].sum()))
changed = x.copy()
changed[3] = rng.normal(0, 1, 8) # a different last token in place of `table`
for mask in [False, True]:
before, after = attend(x, mask)[1][2], attend(changed, mask)[1][2]
print(f"output at coffee changes with the last token, {'with' if mask else 'without'} the mask:",
bool(not np.allclose(before, after)))
```
Output:
```text
weights with the causal mask:
[1. 0. 0. 0.]
sled [0.29 0.71 0. 0. ]
coffee [0.58 0.01 0.41 0. ]
table [0.16 0. 0.03 0.81]
weight on later tokens: 0.0
output at coffee changes with the last token, without the mask: True
output at coffee changes with the last token, with the mask: False
```
The weight matrix is now lower-triangular, with zeros above the diagonal, so the total weight on later tokens is 0.0. The start marker can attend only to itself, so its row is a single 1. `sled` splits its weight between the start marker and itself. `coffee`, which put almost all its weight on the later `table` before the mask, now spreads it over the positions it's allowed, because softmax divides only among the positions left. The last row, `table`, is unchanged, since it had no later positions to lose.
The last two lines show the leak the mask closes. Without the mask, changing the last token changes the output at `coffee`: True. With it, `coffee`'s output can't depend on what comes later: False. A model trained without the mask can learn to copy each answer from the next position, so its training loss can fall close to zero while it learns nothing it can use to generate text. Nothing crashes along the way, which is what makes a wrong mask hard to notice.
**Working rule:** Apply the mask to scores before softmax, never to weights after it, so each row still sums to one. Test it the way this example does: check that the weight on later positions is exactly zero, and that changing a later token leaves earlier outputs unchanged. If a model's training loss falls far faster than expected, check the mask first.
## Scale the scores {#scale-the-scores}
Attention's formula divides every score by √d, and the reason is in the dot product. A dot product of two vectors adds up one product per number. If the numbers in queries and keys are independent and random, with mean 0 and variance 1, a score has variance d, so queries and keys of a larger size give larger scores ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 3.2.1, footnote 4). Softmax turns large differences between scores into extreme weights: one position gets nearly all the weight, and the rest nearly none. Vaswani and colleagues suspected that a model in that state learns slowly, because softmax's gradients are tiny when one weight is close to 1. Dividing by √d brings the scores back to variance 1, whatever the size.
The store's model makes this concrete: its heads work with queries and keys of 32 numbers. For the store, draw random queries and keys of three sizes, 4, 32 and 256, score each query against 16 keys with and without the division, and report the largest weight in each row, averaged over 1,000 queries. Before you run it, predict the largest weight with 256 numbers and no division:
Example `scaling-matters` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
rng = np.random.default_rng(2026)
for size in [4, 32, 256]:
queries, keys = rng.normal(0, 1, (1000, size)), rng.normal(0, 1, (16, size))
for divide in [False, True]:
scores = queries @ keys.T / (np.sqrt(size) if divide else 1)
weights = np.exp(scores - scores.max(1, keepdims=True))
weights /= weights.sum(1, keepdims=True)
label = "scaled" if divide else "unscaled"
print(f"size {size}, {label}, average largest weight in a row: {weights.max(1).mean():.2f}")
```
Output:
```text
size 4, unscaled, average largest weight in a row: 0.45
size 4, scaled, average largest weight in a row: 0.25
size 32, unscaled, average largest weight in a row: 0.78
size 32, scaled, average largest weight in a row: 0.24
size 256, unscaled, average largest weight in a row: 0.92
size 256, scaled, average largest weight in a row: 0.25
```
Without the division, the largest weight grows with the size: 0.45 with 4 numbers, 0.78 with 32, the size of the store's model's heads, and 0.92 with 256. At the larger sizes, each query attends mostly to one key before any training has happened. With the division, the largest weight stays at 0.25 at every size: spread over several keys, though not evenly, since 16 equal weights would be about 0.06 each.
That's the state training should start from. Every position looks at several others, so the gradients reach all of them, and training sharpens the weights where sharp weights help. A trained head can still put most of its weight on one position, as some of the [store's model's heads](#read-trained-heads) do; the scaling only keeps heads from starting there.
**Working rule:** Divide attention scores by the square root of the query's size, as the transformer paper does. A few models, such as T5, leave the division out and fold the factor into how the query weights start instead ([source](https://github.com/huggingface/transformers/blob/v4.46.0/src/transformers/models/t5/modeling_t5.py)), so check which a model does before you load its weights. If you change how queries and keys are computed or initialized, print the average largest weight of an untrained model, as here, and expect it to be well below 1.
## Several heads look for different things {#several-heads}
Each attention computation so far gives one set of weights per position, which can look for one kind of thing: the previous token, say, or the product's name. A model usually needs several at once. **Multi-head attention** runs several attention computations, called **heads**, side by side, each with its own query, key and value matrices, and each in a share of the vector's numbers: with 8 numbers and 2 heads, each head works in 4. The heads' outputs are joined side by side and mixed back together by one more learned matrix, the **output projection** ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 3.2.2).
Splitting the numbers keeps the cost about the same as one head over all of them, and gives each head its own pattern of weights. The store's model has 4 heads in each layer, each working in 32 numbers.
For the store, run two heads over four positions of random vectors, with the causal mask, and compare what the last position attends to in each:
Example `attention-heads` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
rng = np.random.default_rng(2026)
positions, dims, heads = 4, 8, 2
size = dims // heads # each head works in 4 of the 8 dimensions
x = rng.normal(0, 1, (positions, dims))
w_query, w_key, w_value, output_projection = (rng.normal(0, 0.5, (dims, dims)) for _ in range(4))
def split(m): # (positions, dims) -> (heads, positions, size)
return m.reshape(positions, heads, size).transpose(1, 0, 2)
queries, keys, values = split(x @ w_query), split(x @ w_key), split(x @ w_value)
scores = queries @ keys.transpose(0, 2, 1) / np.sqrt(size) # one score matrix per head
scores = np.where(np.tril(np.ones((positions, positions), dtype=bool)), scores, -np.inf)
weights = np.exp(scores - scores.max(-1, keepdims=True))
weights /= weights.sum(-1, keepdims=True)
joined = (weights @ values).transpose(1, 0, 2).reshape(positions, dims) # the heads' outputs side by side
output = joined @ output_projection # mixed back together by one more matrix
print("weights shape (heads, positions, positions):", weights.shape)
print("what the last token attends to, head 1:", np.round(weights[0, -1], 2))
print("what the last token attends to, head 2:", np.round(weights[1, -1], 2))
print("output shape:", output.shape)
```
Output:
```text
weights shape (heads, positions, positions): (2, 4, 4)
what the last token attends to, head 1: [0.41 0.18 0.29 0.11]
what the last token attends to, head 2: [0.04 0.32 0.14 0.5 ]
output shape: (4, 8)
```
The weights now have one matrix per head, of shape (2, 4, 4). The last position's rows differ between the heads, though both read the same input: head 1 puts its largest weight on the first position, and head 2 on the last, its own. With random matrices the difference means nothing yet, but it's the room training uses: each head can learn a different way of looking back, and the output projection combines what they found. The output has the input's shape again, (4, 8), so attention layers can be stacked, as [chapter 7](#the-transformer-block) does.
**Working rule:** As a rule of thumb, keep each head at least 32 numbers wide. In the transformer paper's experiments, 16 heads of 32 did as well as 8 heads of 64, while 32 heads of 16 did worse, and reducing the size of keys hurt ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), table 3). Check the weight tensor's shape, heads by positions by positions, whenever you change the model: a head axis in the wrong place mixes heads together without any error.
## Read what trained heads attend to {#read-trained-heads}
Random matrices show the arithmetic; a trained model shows what attention is for. The store's model has 4 layers, each with 4 heads of attention. It was trained once, in `recordings/transformer.py`, on the store's training names and descriptions, each product as one text: the start marker, the name, a separator token ``, the description and the end marker. Each layer's attention reads the output of the layer before, so later layers can attend on the basis of what earlier ones found. As in [chapter 5](#a-neural-language-model), the examples read the saved weights and recompute the model in NumPy, with `lib.transformer`, which also returns every head's weights.
For the store, take a validation product the model never trained on, `double chaise patio furniture cover`, whose description repeats the end of its name. Stop at the description's second `furniture`, where the next token is `cover`. Print the model's probability for `cover`, and each head's three largest weights in the first and last layers. `current:` marks the position being predicted from, `name:` a token in the name, and `description:` an earlier token in the description:
Example `trained-attention` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.store import load_products
_, vocabulary, manifest = transformer.load()
settings = manifest["settings"]
index = {t: i for i, t in enumerate(vocabulary)}
product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]
ids = transformer.product_ids(product["name"], product["description"], index, settings["longest"])
sep = ids.index(index[transformer.SEP])
position = ids.index(ids[sep - 1], sep) - 1 # just before the description first repeats the name's last token
logits, attention = transformer.forward(ids, keep=True)
probabilities = transformer.softmax(logits[position])
def label(i):
"""Where a token sits: the current position, the name, the description, or a marker."""
if i == position:
return f"current:{vocabulary[ids[i]]}"
if i == 0 or i == sep:
return vocabulary[ids[i]]
return f"{'name' if i < sep else 'description'}:{vocabulary[ids[i]]}"
print("layers:", settings["layers"])
print("heads per layer:", settings["heads"])
print("dimensions per head:", settings["dimensions"] // settings["heads"])
print("name:", " ".join(vocabulary[i] for i in ids[1:sep]))
print("description so far:", " ".join(vocabulary[i] for i in ids[sep + 1:position + 1]))
print(f"P(next token is {vocabulary[ids[position + 1]]}): {probabilities[ids[position + 1]]:.2f}")
for layer in [0, len(attention) - 1]:
for head, weights in enumerate(attention[layer]):
top = np.argsort(-weights[position], kind="stable")[:3]
print(f"layer {layer + 1}, head {head + 1}: " + ", ".join(f"{label(i)} {weights[position, i]:.2f}" for i in top))
name_cover = sep - 1 # the name's last token, `cover`
earlier = ids.index(ids[position], sep + 1) # the description's earlier `furniture`
last = attention[-1][:, position] # the last layer's heads, at the current position
head = int(last[:, name_cover].argmax())
print("last-layer head with the most weight on the name's cover:", f"head {head + 1}")
print("its weight on the name's cover:", f"{last[head, name_cover]:.2f}")
print(f"its weight on {vocabulary[ids[earlier + 1]].removesuffix('')}, after the description's earlier furniture:",
f"{last[head, earlier + 1]:.2f}")
```
Output:
```text
layers: 4
heads per layer: 4
dimensions per head: 32
name: double chaise patio furniture cover
description so far: protect your outdoor patio furniture from har sh elements with this outdoor patio furniture
P(next token is cover): 0.97
layer 1, head 1: current:furniture 0.80, description:furniture 0.11, description:elements 0.03
layer 1, head 2: current:furniture 0.31, name:furniture 0.13, description:furniture 0.11
layer 1, head 3: current:furniture 0.18, description:patio 0.18, 0.12
layer 1, head 4: description:protect 0.17, current:furniture 0.14, description:patio 0.11
layer 4, head 1: name:cover 0.48, name:double 0.18, name:furniture 0.13
layer 4, head 2: name:cover 0.37, 0.13, name:chaise 0.11
layer 4, head 3: description:patio 0.27, current:furniture 0.25, description:outdoor 0.21
layer 4, head 4: description:with 0.38, description:from 0.21, description:elements 0.06
last-layer head with the most weight on the name's cover: head 1
its weight on the name's cover: 0.48
its weight on from, after the description's earlier furniture: 0.00
```
The model gives `cover` a probability of 0.97, and the heads show where it may have looked. In the first layer, one head attends mostly to the current token, `furniture`, and the other three spread their weight over the description and the name so far. In the last layer, head head 1 puts 0.48 of its weight on `cover` in the name, many tokens back: the token that followed `furniture` in the name. The description's own earlier `furniture` was followed by `from`, and that head gives `from` only 0.00. A second last-layer head also puts its largest weight on the name's `cover`.
That pattern has a name. Olsson and colleagues found pairs of heads that work this way in small models made only of attention layers, and heads with the same behaviour in large models. They call the later head of each pair an **induction head**: a head in an earlier layer copies each token's predecessor into its position, and the induction head uses that to find earlier positions that followed the current token, and attends to them ([Olsson et al., 2022](https://arxiv.org/abs/2209.11895)). The weights above are consistent with that behaviour, but weights show where a head drew its average from, not what the model used, so the [next section](#test-what-the-model-copies) tests it.
**Working rule:** When you inspect a trained model, start with specific tokens in texts you understand, where you know what the model should look at, as here. Treat what the weights suggest as a hypothesis, and test it by changing the input, as the next section does.
## Test what the model copies {#test-what-the-model-copies}
If the model predicts `cover` by copying it from the name, then changing the name should change the prediction. If it predicts `cover` because descriptions of this kind often continue that way, the name shouldn't matter much. A test that changes the input and watches the output answers that, where attention weights can't.
For the store, keep the description and change the name: the name as sold, the name without `cover`, and the name ending in `set` instead. Print the probabilities of `cover` and `set` at the same position. Then count what follows `this outdoor patio furniture` in the training descriptions, the four tokens a window model would see:
Example `copy-from-the-name` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib import transformer
from lib.store import load_products
_, vocabulary, manifest = transformer.load()
index = {t: i for i, t in enumerate(vocabulary)}
product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]
cover, other = index["cover"], index["set"]
for name, label in [("double chaise patio furniture cover", "name as sold"),
("double chaise patio furniture", "name without cover"),
("double chaise patio furniture set", "name ending in set")]:
ids = transformer.product_ids(name, product["description"], index, manifest["settings"]["longest"])
sep = ids.index(index[transformer.SEP])
furniture = [i for i in range(sep + 1, len(ids)) if ids[i] == index["furniture"]]
p = transformer.softmax(transformer.forward(ids)[furniture[1]]) # at the description's second `furniture`
print(f"{label}, P(cover): {p[cover]:.2f}")
print(f"{label}, P(set): {p[other]:.2f}")
phrase = "this outdoor patio furniture "
after = [d.split(phrase, 1)[1].split()[0] for d in load_products("train")["description"]
if isinstance(d, str) and phrase in d and d.split(phrase, 1)[1].split()]
print("training descriptions with 'this outdoor patio furniture':", len(after))
print("followed by cover:", after.count("cover"))
```
Output:
```text
name as sold, P(cover): 0.97
name as sold, P(set): 0.02
name without cover, P(cover): 0.00
name without cover, P(set): 0.29
name ending in set, P(cover): 0.00
name ending in set, P(set): 0.70
training descriptions with 'this outdoor patio furniture': 6
followed by cover: 2
```
With the name as sold, the model gives `cover` 0.97. Take `cover` out of the name and its probability falls to 0.00. End the name in `set` instead, and the model gives `set` 0.70 and `cover` 0.00. The prediction follows the name. In the training descriptions, 6 contain `this outdoor patio furniture`, and 2 of them continue with `cover`. So the four tokens alone point only weakly at `cover`; the name decides it.
That's the behaviour the previous section's weights suggested, now shown by its effect: the model copies a word from the product's name, many tokens back, which a model limited to a window of four tokens can't do. One product isn't a measure of how often the model does it; the [next two sections](#attend-to-the-previous-token) measure its heads over many products.
**Working rule:** Test a claim about what a model uses by changing that input and measuring the output, as here, and include a change that should leave the output alone. Read attention weights for hypotheses, and use tests like this one for conclusions.
## Some heads attend to the previous token {#attend-to-the-previous-token}
One text can mislead; an average over many shows a head's habits. One habit worth measuring is attending to the previous token. A head that does it carries each token's predecessor forward into its position, so later layers see pairs of tokens, as [chapter 1](#predicting-the-next-word)'s bigram counts did, and the copying of the [previous sections](#read-trained-heads) depends on it: to find what followed `furniture` before, a head needs each earlier position to know its predecessor.
For the store, average each head's weight on the previous token over the first 300 validation products, find the head that puts the most there, and compare with an even spread, which puts 1 / (i + 1) on each of the i + 1 tokens position i can see:
Example `previous-token-heads` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
texts = product_texts("validation", 300)
layers, heads = len(transformer.forward(texts[0][0], keep=True)[1]), transformer.load()[2]["settings"]["heads"]
on_previous = np.zeros((layers, heads))
for ids, _ in texts:
_, attention = transformer.forward(ids, keep=True)
for layer, weights in enumerate(attention): # weights: (heads, positions, positions)
on_previous[layer] += weights[:, np.arange(1, len(ids)), np.arange(len(ids) - 1)].mean(1)
on_previous /= len(texts)
even = np.mean([np.mean(1 / np.arange(2, len(ids) + 1)) for ids, _ in texts]) # spread evenly over the visible tokens
print("average weight on the previous token (rows: layers, columns: heads):")
for layer, row in enumerate(on_previous, start=1):
print(f" layer {layer}: " + " ".join(f"{w:.2f}" for w in row))
layer, head = np.unravel_index(on_previous.argmax(), on_previous.shape)
print("head with the most weight on the previous token:", f"layer {layer + 1}, head {head + 1}")
print("its weight on the previous token:", f"{on_previous.max():.2f}")
print("an even spread's weight on the previous token:", f"{even:.2f}")
```
Output:
```text
average weight on the previous token (rows: layers, columns: heads):
layer 1: 0.03 0.06 0.13 0.10
layer 2: 0.32 0.13 0.27 0.24
layer 3: 0.13 0.33 0.09 0.07
layer 4: 0.11 0.07 0.35 0.27
head with the most weight on the previous token: layer 4, head 3
its weight on the previous token: 0.35
an even spread's weight on the previous token: 0.07
```
The head that puts the most weight on the previous token is layer 4, head 3, with 0.35 on average. An even spread, averaged the same way, would put 0.07 there, so that head favours the previous token about five times over. Several heads in the later layers put a quarter or more of their weight there, and the first layer's heads put much less: in the [trained-attention example](#read-trained-heads), one of them attended mostly to the current token.
No head is purely a previous-token head; each mixes it with other habits. A head's average says what it does most, not all it does, so describing a head as having one job oversimplifies.
**Working rule:** Describe a trained head by measured averages, such as its weight on the previous token, the current token or the start marker, over a few hundred texts. Compare each with what an even spread would give, averaged the same way, so a number such as 0.3 can be read as large or small.
## Reach past the window {#reach-past-the-window}
Attention's promise was to see past [chapter 5](#a-neural-language-model)'s four-token window, which predicts from the current token and the three before it. Whether the store's model does is measurable. For every description token in the first 300 validation products, leaving out the end marker, add up the weight each head puts on tokens 4 or more positions back, the nearest a four-token window can't see, and separately the weight on the two markers, `` and ``. Average over tokens, heads and layers, and compare with an even spread over the positions each token can see:
Example `attention-reaches-back` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
_, vocabulary, manifest = transformer.load()
end = vocabulary.index(transformer.END)
texts = product_texts("validation", 300)
layers = manifest["settings"]["layers"]
far, markers, even_far, even_markers, counted = np.zeros(layers), np.zeros(layers), 0.0, 0.0, 0
for ids, sep in texts:
queries = np.array([i for i in range(sep + 1, len(ids)) if ids[i] != end]) # the description's tokens
if len(queries) == 0:
continue # a product with no description
_, attention = transformer.forward(ids, keep=True)
stacked = np.stack(attention)[:, :, queries] # (layers, heads, description tokens, positions)
positions = np.arange(len(ids))
is_marker = np.isin(positions, [0, sep]) # and
beyond = (queries[:, None] - positions[None, :] >= 4) & ~is_marker # what a 4-token window can't see
far += (stacked * beyond).sum(-1).mean(1).sum(-1)
markers += (stacked * is_marker).sum(-1).mean(1).sum(-1)
even_far += (beyond.sum(1) / (queries + 1)).sum() # an even spread over the positions each token can see
even_markers += (2 / (queries + 1)).sum()
counted += len(queries)
print("description tokens measured:", f"{counted:,}")
print("average attention weight of a description token, by layer:")
for layer in range(layers):
print(f" layer {layer + 1}: 4 or more tokens back {far[layer] / counted:.2f}, on and {markers[layer] / counted:.2f}")
print("all layers, weight 4 or more tokens back:", f"{far.mean() / counted:.2f}")
print("an even spread's weight 4 or more tokens back:", f"{even_far / counted:.2f}")
print("all layers, weight on and :", f"{markers.mean() / counted:.2f}")
print("an even spread's weight on and :", f"{even_markers / counted:.2f}")
described = [(ids[1:sep], ids[sep + 1:]) for ids, sep in texts if any(i != end for i in ids[sep + 1:])]
print("products with a description:", len(described))
print("descriptions that repeat a token of the name:", sum(bool(set(n) & set(d)) for n, d in described))
```
Output:
```text
description tokens measured: 20,571
average attention weight of a description token, by layer:
layer 1: 4 or more tokens back 0.42, on and 0.12
layer 2: 4 or more tokens back 0.39, on and 0.04
layer 3: 4 or more tokens back 0.58, on and 0.13
layer 4: 4 or more tokens back 0.47, on and 0.13
all layers, weight 4 or more tokens back: 0.46
an even spread's weight 4 or more tokens back: 0.85
all layers, weight on and : 0.11
an even spread's weight on and : 0.05
products with a description: 262
descriptions that repeat a token of the name: 244
```
Across all layers, description tokens put 0.46 of their attention on tokens 4 or more back. An even spread would put 0.85 there, because most of what a description token can see is far back. So the heads favour nearby tokens, as you'd expect for predicting the next word, and still send a large share of their attention past the window, which no earlier model in this book could use. That matters for this text: of the 262 products with a description, 244 have a description that repeats at least one token of the name.
The markers get 0.11 of the attention, against 0.05 for an even spread: about twice as much as chance, though they predict no word themselves. One likely reason is a pattern found in larger models. Softmax makes every row of weights sum to one, so a head that finds nothing useful at a position still has to put its weight somewhere, and Xiao and colleagues found that language models put large weights on the first tokens of a text whatever they are, calling them **attention sinks** ([Xiao et al., 2024](https://arxiv.org/abs/2309.17453)). In the store's model, `` always comes first and `` is in every text, so both could serve; `` also marks where the description starts, which is useful in itself. This example can't tell the two apart.
**Working rule:** Measure how far back a trained model attends, as here, before deciding how long its context needs to be, and compare with an even spread so the number means something. Report the weight on markers separately, so marker weight doesn't look like long-range attention, and keep the first tokens when you cut a model's context to save space.
## What goes wrong in production {#attention-in-production}
Attention is the part of a language model whose cost grows fastest with text length, and the part most often misread:
- **Cost grows with the square of the length.** Every position scores every earlier one, so a text twice as long costs four times as much attention. Set a maximum length, as the store's model does at 160 tokens, and check how many of your texts the limit cuts.
- **A wrong mask fails silently.** A model that can see the future can reach a low training loss and still be useless for generating text. Test that the weight on later positions is zero, and that changing a later token leaves earlier outputs alone, in the training code and the generation code.
- **Attention weights aren't explanations.** A high weight says where a head drew its average from, not why the model predicted what it did. In the same model, very different attention weights can give the same prediction ([Jain and Wallace, 2019](https://aclanthology.org/N19-1357/), on recurrent and convolutional classifiers). Use weights to form hypotheses, and test them by changing the input, as this chapter did with the product's name.
- **Marker weight looks like importance.** Heavy weight on the first token or a separator may mean a head has nothing to do there. Don't remove those tokens, and don't read their weight as meaning.
- **Softmax overflows without care.** Subtract each row's largest score before exponentiating, as `lib.transformer.softmax` does. Make sure every row keeps at least one unmasked position, or the row becomes NaN: a causal mask always leaves each position itself, but a mask that hides padding can hide a whole row.
## What this chapter decided {#attention-decisions}
- **The book's models use causal, scaled, multi-head attention,** as Vaswani et al. defined it: scores divided by the square root of the size of each head's queries and keys, minus infinity on every later position before softmax, and heads that each work in a share of the numbers, mixed by an output projection.
- **The store's model has 4 layers of 4 heads, each head 32 numbers wide,** trained once in `recordings/transformer.py` on names and descriptions joined by a `` marker, and recomputed in NumPy by `lib.transformer`.
- **Claims about what a model uses are tested by changing its input.** Attention weights suggest hypotheses; changing the product's name showed that the model copies `cover` from it.
- **Trained heads are described by measured averages over many texts,** compared with an even spread: the weight on the previous token, the weight on tokens 4 or more back, and the weight on markers, counted separately.
- **On the store's descriptions, attention reaches well past [chapter 5](#a-neural-language-model)'s window.** Description tokens put 0.46 of their attention on tokens 4 or more back, and the model copies from the name, which a fixed window can't.
- **Attention needs explicit position information.** A weighted average ignores order, and the causal mask restores only a little of it, so [chapter 7](#the-transformer-block) adds rotary positions, as the store's model does inside every head.
# The transformer block
By the end of this chapter you can assemble a transformer block from its parts, joined by residual connections: attention, normalization, a gated feed-forward layer and position information. You can then stack blocks into a complete language model. You'll check each part on the store's trained model, check the whole model against PyTorch's own implementations, measure what each block and each detail contributes, and count where the parameters go.
[Chapter 6](#attention) built attention: each position looks back at earlier ones and takes a weighted average of what it finds. Attention alone isn't a language model. It moves information between positions but transforms little of it, and it has almost no sense of word order. Layers of it stacked directly also degrade: without the other parts of a block, the output of pure attention tends towards the same vector at every position, faster with every layer ([Dong et al., 2021](https://arxiv.org/abs/2103.03404)). The **transformer block** adds the parts that fix those problems around attention ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)).
This chapter builds the block in the form the LLaMA models popularized: normalization of each part's input with RMSNorm, a gated feed-forward layer and rotary position embeddings in every layer ([Touvron et al., 2023](https://arxiv.org/abs/2302.13971), section 2.2). SmolLM2, whose tokenizer [chapter 2](#read-an-open-tokenizer) read, is built the same way. Its 360-million-parameter model has 960 dimensions, 32 layers and 15 heads, and differs from the store's model in two details covered below: its heads share keys and values, and its rotary positions turn more slowly ([configuration](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/blob/f8027fd0eaeea54caa13c31d31b9fdc459c38b49/config.json)).
The store's model, whose heads [chapter 6](#read-trained-heads) read, is 4 of these blocks between an embedding table and an output layer; this book calls each block a layer of the model. At every position, each block does two things in order:
1. normalize the vector, run attention, and add the result back to the vector;
2. normalize again, run a feed-forward layer, and add that result back too.
The sections take those parts one at a time, starting with the additions that hold them together.
## Add each part's output to a residual stream {#add-to-the-residual-stream}
In [chapter 5](#a-neural-language-model)'s [window model](#predict-from-a-window), each layer replaced its input: the hidden layer's output was all the output layer saw. A transformer block adds to its input instead. Each part computes something from the current vector and adds its output to it: x ← x + attention(x), then x ← x + feed-forward(x). That's a **residual connection**. It comes from image models, where very deep stacks of layers trained well only once each layer added to its input instead of replacing it ([He et al., 2016](https://arxiv.org/abs/1512.03385)).
Two consequences follow. With the additions, the signal can pass "directly from one block to any other block", forwards and, as [gradients](#compute-gradients), backwards, so a deep stack trains far more reliably than it would without them ([He et al., 2016b](https://arxiv.org/abs/1603.05027)). And every position's vector becomes a shared workspace that each part reads from and writes to: the token's embedding goes in at the bottom, each part adds what it computes, and the output layer reads the sum at the top. Elhage and colleagues call this running sum the **residual stream** ([Elhage et al., 2021](https://transformer-circuits.pub/2021/framework/index.html)).
For the store, run its trained model on one validation product. At every layer, measure the length of what attention and the feed-forward layer add, and of the vectors after both additions, averaged over positions. A vector's length here is the square root of the sum of its squared numbers:
Example `residual-stream` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
weights, _, manifest = transformer.load()
ids, sep = product_texts("validation", 1)[0]
x = weights["embedding"][np.array(ids)]
print("length of the vectors entering layer 1:", f"{np.linalg.norm(x, axis=1).mean():.2f}")
for layer in range(manifest["settings"]["layers"]):
layer_weights = transformer.layer_weights(layer)
added_by_attention, _ = transformer.attention(transformer.rms_norm(x, layer_weights["norm1"]), layer_weights, manifest["settings"]["heads"])
x = x + added_by_attention
normalized = transformer.rms_norm(x, layer_weights["norm2"])
added_by_feed_forward = (transformer.silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]
x = x + added_by_feed_forward
print(f"layer {layer + 1}: attention adds {np.linalg.norm(added_by_attention, axis=1).mean():.2f}, "
f"feed-forward adds {np.linalg.norm(added_by_feed_forward, axis=1).mean():.2f}, "
f"vectors now {np.linalg.norm(x, axis=1).mean():.2f}")
print("length of the vectors leaving the last layer:", f"{np.linalg.norm(x, axis=1).mean():.2f}")
print("largest number in the stream:", f"{np.abs(x).max():.1f}")
```
Output:
```text
length of the vectors entering layer 1: 1.17
layer 1: attention adds 2.90, feed-forward adds 3.00, vectors now 4.65
layer 2: attention adds 2.05, feed-forward adds 2.90, vectors now 6.14
layer 3: attention adds 2.26, feed-forward adds 4.12, vectors now 8.46
layer 4: attention adds 4.08, feed-forward adds 6.33, vectors now 13.16
length of the vectors leaving the last layer: 13.16
largest number in the stream: 50.6
```
The vectors enter the first layer with an average length of 1.17, and leave the last with 13.16. Every layer adds to them, and what attention adds is comparable in size to what the feed-forward layer adds, though the feed-forward layer adds more at every layer, and by more in the later ones: both parts do real work at every layer. The vectors grow as they go, because each addition piles onto the last. A layer reading the long vectors near the top and one reading the short ones at the bottom would need very differently scaled weights, unless something rescales what each layer reads; the [next section](#normalize-before-each-part) does.
**Working rule:** Write every block as additions to the residual stream, never as replacements, and keep the stream's width the same through the whole model so the additions line up. When you debug a deep model, measure what each part adds, as here: a part that adds almost nothing is doing almost nothing.
## Normalize before each part {#normalize-before-each-part}
The residual stream grows through the layers, as the [previous section](#add-to-the-residual-stream) measured, so each part should read a rescaled copy of it. **Normalization** rescales a vector to a standard size before a part reads it, so the part's weights work the same whether the stream is short or long. The original transformer normalized after each addition, with **layer normalization**, which subtracts each vector's mean and divides by its standard deviation ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)). Xiong and colleagues showed that normalizing before each part instead trains more stably, because only the copy each part reads is rescaled, never the stream itself ([Xiong et al., 2020](https://arxiv.org/abs/2002.04745)). And **RMSNorm** drops the mean: it divides each vector by its root mean square, the square root of the average of its squared numbers, and multiplies each dimension by a learned gain ([Zhang and Sennrich, 2019](https://arxiv.org/abs/1910.07467)):
RMSNorm(x) = x / √(mean(x²) + ε) · g
where ε is a tiny number, 10⁻⁵ here, that keeps the division safe, and g is the gain, which starts at 1. Zhang and Sennrich found RMSNorm about as accurate as layer normalization, and faster. The store's model uses it before attention, before the feed-forward layer, and once more before the output layer.
For the store, take the residual stream of the same validation product at both ends of the model, entering the first block and leaving the last. Normalize every position's vector by hand, with the gain left at 1, and compare with `lib.transformer.rms_norm`:
Example `rms-norm-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
ids, _ = product_texts("validation", 1)[0]
streams, _ = transformer.run(ids) # the residual stream entering block 1 and leaving each block
gain = np.ones(streams[0].shape[1]) # the learned gain set to 1, to show the normalization alone
for label, x in [("entering block 1", streams[0]), ("leaving the last block", streams[-1])]:
root_mean_square = np.sqrt((x ** 2).mean(1, keepdims=True) + 1e-5) # one per position; 1e-5 is the ε
by_hand = x / root_mean_square * gain
print(f"root mean square {label}, before:", f"{root_mean_square.mean():.2f}")
print(f"root mean square {label}, after:", f"{np.sqrt((by_hand ** 2).mean(1)).mean():.2f}")
print(f"same as lib.transformer.rms_norm, {label}:", bool(np.allclose(by_hand, transformer.rms_norm(x, gain), atol=1e-4)))
print(f"largest mean of a position's numbers after, {label}:", f"{np.abs(by_hand.mean(1)).max():.2f}")
```
Output:
```text
root mean square entering block 1, before: 0.10
root mean square entering block 1, after: 1.00
same as lib.transformer.rms_norm, entering block 1: True
largest mean of a position's numbers after, entering block 1: 0.23
root mean square leaving the last block, before: 1.16
root mean square leaving the last block, after: 1.00
same as lib.transformer.rms_norm, leaving the last block: True
largest mean of a position's numbers after, leaving the last block: 0.26
```
Entering the first block, the vectors' root mean square is 0.10 on average; leaving the last, it's 1.16. After normalization, both are 1.00, and the hand calculation matches the library's at both ends: True and True. RMSNorm rescales but doesn't centre: a position's numbers can still average away from zero afterwards, by up to 0.26 here. That's the step it saves over layer normalization.
So though the stream grew many times longer across the layers, every attention and feed-forward layer read vectors of the same scale, times its learned gains.
**Working rule:** Normalize before each part of the block, not after, and normalize the stream once more before the output layer. Use RMSNorm unless you're matching a model that uses layer normalization. When you load another model's weights, check which normalization it uses, where, and with what ε, before anything else: the [mismatched-details example](#measure-mismatched-details) shows what a small mismatch costs.
## Transform each position with a gated feed-forward layer {#transform-each-position}
Attention moves information between positions; the **feed-forward layer** transforms it, at each position separately. It's the same idea as [chapter 5](#a-neural-language-model)'s [hidden layer](#predict-from-a-window): widen the vector, apply a function that isn't a straight line, and narrow the result back to the stream's width. The store's model widens 128 numbers to 352.
The form the LLaMA models use is **gated**. It computes two widened vectors from the input, passes one through a function called SiLU, and multiplies the two number by number, so the first decides how much of the second gets through ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)):
FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down
where ⊙ multiplies number by number, and SiLU(z) = z · σ(z), with σ the logistic function from [chapter 4](#learn-by-predicting-neighbours). SiLU passes large positive numbers almost unchanged and turns large negative ones to nearly zero, smoothly. Shazeer compared several gated forms with the plain one, at the same number of parameters, and found the gated ones predicted better; this form is called SwiGLU.
For the store, compute SiLU at three points. Then run the first block's real feed-forward layer on the same validation product, and count, at each position, how many of its hidden units the gate nearly shuts:
Example `gated-feed-forward` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
print("silu(-3), silu(0), silu(3):", ", ".join(f"{v:.2f}" for v in transformer.silu(np.array([-3.0, 0.0, 3.0]))))
ids, _ = product_texts("validation", 1)[0]
heads = transformer.load()[2]["settings"]["heads"]
first = transformer.layer_weights(0) # block 1's weights
x = transformer.load()[0]["embedding"][np.array(ids)]
attended, _ = transformer.attention(transformer.rms_norm(x, first["norm1"]), first, heads)
normalized = transformer.rms_norm(x + attended, first["norm2"]) # what block 1's feed-forward layer reads
gate = transformer.silu(normalized @ first["gate"]) # how much of each hidden unit to let through
up = normalized @ first["up"] # what each hidden unit carries
output = (gate * up) @ first["down"] # gated, then projected back to the stream's width
shut = (np.abs(gate) < 0.05).sum(1) # per position
print("hidden units:", gate.shape[1])
print("hidden units nearly shut at a position (gate under 0.05 in size), on average:", f"{shut.mean():.0f}")
print("fewest and most nearly shut at a position:", f"{shut.min()} and {shut.max()}")
print("positions:", len(ids))
print("output shape:", output.shape)
```
Output:
```text
silu(-3), silu(0), silu(3): -0.14, 0.00, 2.86
hidden units: 352
hidden units nearly shut at a position (gate under 0.05 in size), on average: 49
fewest and most nearly shut at a position: 33 and 67
positions: 133
output shape: (133, 128)
```
SiLU of −3, 0 and 3 gives -0.14, 0.00, 2.86: nearly zero for the negative number, nearly unchanged for the positive one. Of the first block's 352 hidden units, the gate nearly shuts 49 at a typical position, between 33 and 67 depending on the position. Which units are open depends on the input, so each position uses its own mix of them. The output has one vector per position at the stream's width again, (133, 128), ready to add.
The feed-forward layers hold most of a block's parameters: in the store's model, three matrices of 128 by 352, against attention's four of 128 by 128. Geva and colleagues found that in trained models these layers work like stores of patterns: each hidden unit responds to particular patterns in the text, and pushes the prediction towards particular next tokens ([Geva et al., 2021](https://arxiv.org/abs/2012.14913)).
**Working rule:** Use a gated feed-forward layer with SiLU. Set its width so its three matrices hold as many parameters as a plain layer four times the stream's width, the original transformer's ratio, 2,048 for 512 ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 3.3): that's about 8/3 times the width. SmolLM2-360M uses exactly 8/3, 2,560 for 960. The store's model rounds 8/3 of its width up to a multiple of 32, a rule of thumb that keeps matrix sizes convenient for hardware. When you load another model, check whether its feed-forward layer is gated, since a gated layer has three matrices where a plain one has two.
## Give attention positions with rotations {#rotary-positions}
A weighted average ignores the order of what it averages, as [chapter 6](#look-back-with-a-weighted-average) showed, so attention gets at most a weak sense of position from the causal mask alone. The original transformer added a vector for each position to each token's embedding ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 3.5). The LLaMA models instead use **rotary position embeddings**, RoPE, which work inside attention, at each layer ([Su et al., 2021](https://arxiv.org/abs/2104.09864); [Touvron et al., 2023](https://arxiv.org/abs/2302.13971), section 2.2). Before scoring, each query and key is rotated by angles that grow with its position. The vector's d numbers are taken in pairs, number i with number i + d/2 in the store's model, and each pair (a, b) is treated as a point on a plane and turned around the origin. At position p, pair i turns by the angle p·θᵢ, where θᵢ = 10000^(−i/(d/2)):
(a, b) → (a cos pθᵢ − b sin pθᵢ, a sin pθᵢ + b cos pθᵢ)
So the first pairs turn fast, a full radian per position, and the last pairs slowly. The line `angles = ...` in `lib.transformer.rope` computes p·θᵢ for every position and pair, and the line after it applies the rotation.
Rotation has two properties that suit attention. It doesn't change a vector's length, so it adds position without distorting content. And if a query at position m is turned by mθᵢ and a key at position n by nθᵢ, their dot product depends only on the difference, (m − n)θᵢ: the score depends on how far apart they are, not on where they sit in the text. A head can learn to look 3 tokens back, and it works the same at the start of a name and deep in a description.
For the store's model's heads of 32 numbers, take a random query and key, rotate them for positions 0 to 19, and compare scaled scores, as [chapter 6](#attention) defines them, at three pairs of positions:
Example `rope-by-hand` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib.transformer import rope
rng = np.random.default_rng(2026)
size = 32 # one head's query and key, as in the store's model
query, key = rng.normal(0, 1, size), rng.normal(0, 1, size)
positions = 20
rotated_q = rope(np.tile(query, (positions, 1))) # the same query, rotated for each position 0 to 19
rotated_k = rope(np.tile(key, (positions, 1)))
kept = np.allclose(np.linalg.norm(rotated_q, axis=1), np.linalg.norm(query)) and \
np.allclose(np.linalg.norm(rotated_k, axis=1), np.linalg.norm(key))
print("length kept by the rotation, for the query and the key:", bool(kept))
for i, j in [(2, 5), (12, 15), (2, 8)]:
print(f"score with the key at {i} and the query at {j}:", f"{rotated_q[j] @ rotated_k[i] / np.sqrt(size):.3f}")
```
Output:
```text
length kept by the rotation, for the query and the key: True
score with the key at 2 and the query at 5: 0.217
score with the key at 12 and the query at 15: 0.217
score with the key at 2 and the query at 8: -0.765
```
The rotation keeps both vectors' lengths: True. The score between the key at position 2 and the query at position 5 is 0.217, and between positions 12 and 15 it's 0.217, the same, because both pairs are 3 apart. Between positions 2 and 8, 6 apart, it's -0.765: the same query and key, scored differently because they're further apart.
RoPE has no learned parameters. The angles are fixed, and the model learns queries and keys that make use of them. The store's model applies it to the queries and keys of every head in every layer, which gives [chapter 6's](#attend-to-the-previous-token) previous-token heads a direct way to find the previous token, alongside the weak signal the mask provides.
**Working rule:** Use rotary position embeddings, applied to queries and keys, never values. When you load another model's weights, check how its RoPE pairs the numbers and what base it uses. Some implementations pair neighbouring numbers; others pair the first half with the second, as `lib.transformer.rope` does and as SmolLM2's configuration states with `"rope_interleaved": false`. SmolLM2 uses a base of 100,000 where the store's model uses 10,000.
## Tie the output layer to the embeddings {#tie-the-output}
With all the parts built, the whole model is short enough to read in one piece. `lib.transformer.block` is one block, exactly as the chapter has described it:
```python
def block(x: np.ndarray, layer_weights: dict[str, np.ndarray], heads: int, positions=rope):
"""One transformer block: attention, then the gated feed-forward layer, each added to the residual stream."""
attended, weights = attention(rms_norm(x, layer_weights["norm1"]), layer_weights, heads, positions)
x = x + attended
normalized = rms_norm(x, layer_weights["norm2"])
x = x + (silu(normalized @ layer_weights["gate"]) * (normalized @ layer_weights["up"])) @ layer_weights["down"]
return x, weights
```
`lib.transformer.run` passes each token's embedding through the blocks in turn, and these lines of `lib.transformer.forward` turn the final stream into one logit per token in the vocabulary, the scores [chapter 5](#a-neural-language-model)'s softmax turns into probabilities:
```python
streams, kept = run(ids, name, positions, skip)
weights, _, _ = load(name)
logits = rms_norm(streams[-1], weights["final_norm"]) @ weights["embedding"].T
```
The last line holds one more idea: the output layer reuses the embedding table, turned on its side, instead of learning a matrix of its own. A token's logit is the dot product of the final vector with that token's embedding. Press and Wolf found that this **weight tying** lowers perplexity as well as saving the output layer's parameters ([Press and Wolf, 2017](https://arxiv.org/abs/1608.05859)). SmolLM2 ties them too (`"tie_word_embeddings": true` in its configuration), since for a model of its size a separate output layer would be a large share of the whole. The [parameter count](#count-the-parameters) shows what tying saves in the store's model.
**Working rule:** Tie the output layer to the embeddings in a small model, and check whether a model you load ties them before you look for a separate output matrix. Keep your own forward pass short enough to read whole, as `lib.transformer` is, so that every detail the next sections test is visible.
## Check the model against PyTorch {#check-against-pytorch}
PyTorch trained the store's model once, in `recordings/transformer.py`, as it trained [chapter 5](#a-neural-language-model)'s (see [Train with PyTorch](#train-with-pytorch)), and the script saved PyTorch's own logits for every position of one validation product. The training script uses PyTorch's own implementations of RMSNorm, causal attention and SiLU, so comparing them with `lib.transformer` checks the NumPy code against a library's, not against a second copy of itself. RoPE is the exception: PyTorch has no built-in version, so both programs write it out, and the check confirms only that they agree. The [rope-by-hand example](#rotary-positions) checks its properties instead.
For the store, recompute PyTorch's logits with the NumPy model, and compare:
Example `transformer-same-as-pytorch` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
_, _, manifest = transformer.load()
check = np.load("data/recorded/transformer/check.npz") # one validation product's ids, and PyTorch's logits
ours = transformer.forward(list(check["ids"]))
print("PyTorch version:", manifest["torch"])
print("positions compared:", len(check["ids"]))
print("logits compared:", f"{ours.size:,}")
print("same as PyTorch to 4 decimal places:", bool(np.abs(ours - check["logits"]).max() < 1e-4))
```
Output:
```text
PyTorch version: 2.14.1+cpu
positions compared: 133
logits compared: 550,886
same as PyTorch to 4 decimal places: True
```
The NumPy model reproduces all 550,886 of PyTorch's logits, for 133 positions, to four decimal places: True, for PyTorch 2.14.1+cpu. Because the check covers a whole text, it covers the parts that depend on position, the mask and the rotations, as well as the norms, the gates and the tied output.
**Working rule:** Check your own forward pass against the training library's outputs on a full text, not a single position. Where the library has its own implementation of a part, train with it, so the check compares two independent implementations; where it doesn't, as with RoPE here, test the part's properties by hand.
## Leave out a block to see what it does {#leave-out-a-block}
The model is checked; now measure what its blocks contribute. Because every block adds to the residual stream, a block can be left out and the model still runs: the stream passes through it unchanged, and the next block reads it as before. That allows a simple experiment, called an **ablation**: remove one part, and measure what the model loses. It shows how much each block contributes to the trained model's predictions, which reading its weights can't.
For the store, score the names and descriptions of the first 100 validation products, 7,624 predictions, with the full model and then with each block left out in turn. Before you run it, predict which block's removal hurts most:
Example `skip-a-block` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
layers = transformer.load()[2]["settings"]["layers"]
texts = [ids for ids, _ in product_texts("validation", 100)]
def measure(skip=None):
"""Perplexity per token, and the average probability of the top token, with one block left out."""
surprise, top, count = 0.0, 0.0, 0
for ids in texts:
p = transformer.softmax(transformer.forward(ids, skip=skip)[:-1])
surprise -= np.log(p[np.arange(len(ids) - 1), ids[1:]]).sum()
top += p.max(1).sum()
count += len(ids) - 1
return np.exp(surprise / count), top / count
print("tokens scored, names and descriptions:", f"{sum(len(ids) - 1 for ids in texts):,}")
for skip in [None] + list(range(layers)):
label = "all blocks" if skip is None else f"without block {skip + 1}"
perplexity, top = measure(skip)
print(f"{label}:", f"{perplexity:,.1f}")
print(f"{label}, average probability of the top token:", f"{top:.2f}")
```
Output:
```text
tokens scored, names and descriptions: 7,624
all blocks: 24.3
all blocks, average probability of the top token: 0.42
without block 1: 17,896.9
without block 1, average probability of the top token: 0.21
without block 2: 77.8
without block 2, average probability of the top token: 0.29
without block 3: 64.3
without block 3, average probability of the top token: 0.34
without block 4: 77.3
without block 4, average probability of the top token: 0.37
```
With all four blocks, the [perplexity](#score-with-perplexity) per token is 24.3. This perplexity covers names and descriptions together, so it can't be set beside [chapter 3's baseline](#the-baseline) or [chapter 5](#a-neural-language-model)'s window model, which are scored on validation names alone; [chapter 8](#training-a-small-model) makes that comparison. Leaving out block 2, 3 or 4 more than doubles it, to 77.8, 64.3 and 77.3: each later block contributes a share of the model's skill, and none is dispensable. Leaving out block 1 is a different kind of failure. Perplexity goes to 17,896.9, worse than guessing evenly among all 4,142 tokens, which would score 4,142.
The difference is in what the later blocks expect. They were trained on a stream that block 1 had already written to, and without it they read vectors unlike any they saw in training. Their predictions become less confident, with the top token's average probability falling from 0.42 to 0.21, and they put that probability on the wrong tokens. Block 1 turns raw embeddings into the form everything after it is built on, partly by gathering context from the text so far, as three of [chapter 6's](#read-trained-heads) first-layer heads did. Blocks in a trained model aren't independent modules: each depends on what the earlier ones write.
**Working rule:** To learn what a part of a model does, remove it and measure, as here, rather than inferring from its weights. Expect the first block to matter most, since everything later is built on it; beyond that, measure, because here the second and last blocks mattered about equally, and more than the third. Read a small loss from removing a part with care: the remaining parts were trained alongside it, so the measurement says what the trained model relies on, not what the part could do alone.
## Measure what a mismatched detail costs {#measure-mismatched-details}
Every detail of a block has to match the weights it runs with, and a mismatch doesn't announce itself: the model still produces probabilities. The details most often wrong when loading another model's weights are in RoPE, because implementations differ in how they pair numbers and in their base, as SmolLM2's differs from the store's model's.
For the store, score the same 100 validation products with the trained weights, first as trained, then with RoPE pairing neighbouring numbers instead of halves, with RoPE's base at 100,000, SmolLM2's, instead of 10,000, and with no RoPE at all:
Example `mismatched-details` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.products import product_texts
texts = [ids for ids, _ in product_texts("validation", 100)]
def neighbours(x, base=10000.0):
"""RoPE that pairs neighbouring numbers, (0, 1), (2, 3) and so on, instead of the first half with the second."""
half = x.shape[-1] // 2
rotated = transformer.rope(np.concatenate([x[..., 0::2], x[..., 1::2]], axis=-1), base)
out = np.empty_like(x)
out[..., 0::2], out[..., 1::2] = rotated[..., :half], rotated[..., half:]
return out
for label, positions in [("as trained", transformer.rope),
("RoPE pairing neighbouring numbers", neighbours),
("RoPE with base 100,000", lambda x: transformer.rope(x, base=100000.0)),
("no RoPE", lambda x: x)]:
print(f"perplexity, {label}:", f"{transformer.perplexity(texts, positions=positions):,.1f}")
```
Output:
```text
perplexity, as trained: 24.3
perplexity, RoPE pairing neighbouring numbers: 112.2
perplexity, RoPE with base 100,000: 26.4
perplexity, no RoPE: 182.0
```
As trained, the perplexity is 24.3. Pairing neighbouring numbers instead of halves raises it to 112.2, and leaving RoPE out raises it to 182.0. Both would be caught by the first sample of generated text. The base is the dangerous one: at 100,000 the perplexity is 26.4, slightly worse, with no error and fluent-looking output, a mistake that could ship.
The other difference from SmolLM2 is **grouped-query attention**: its 15 query heads share 5 sets of keys and values, so its key and value matrices are a third of the size of its query matrix ([Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)). Loading such weights into a model that expects a key and value for every head fails with a shape error, which is the kind of mismatch you want.
**Working rule:** When you load another model's weights, compare its configuration with your code line by line: normalization and its ε, RoPE's pairing and base, the feed-forward layer's gating, tied outputs, and how many heads share keys and values. Then score text you know, as here, against the original implementation's outputs; a small loss with no error is the case to fear.
## Count where the parameters go {#count-the-parameters}
With every part in place, you can count what the model learns, and where. A model's parameters are what it can learn, and where they sit says what it spends that capacity on. For the store, count the trained model's parameters by part:
Example `transformer-parameters` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
from lib import transformer
weights, vocabulary, manifest = transformer.load()
settings = manifest["settings"]
groups = {"embedding (shared with the output)": ["embedding"], "attention": ["q", "k", "v", "o"],
"feed-forward": ["gate", "up", "down"], "normalization": ["norm1", "norm2", "final_norm"]}
for group, parts in groups.items():
count = sum(v.size for k, v in weights.items() if k.split(".")[-1] in parts)
print(f"{group}:", f"{count:,}")
print("total:", f"{sum(v.size for v in weights.values()):,}")
print("embedding's share of the total:", f"{weights['embedding'].size / sum(v.size for v in weights.values()):.0%}")
print("layers:", settings["layers"])
print("width:", settings["dimensions"])
print("heads:", settings["heads"])
print("feed-forward width:", settings["feed_forward"])
print("longest text, in tokens:", settings["longest"])
print("vocabulary:", f"{len(vocabulary):,}")
```
Output:
```text
embedding (shared with the output): 530,176
attention: 262,144
feed-forward: 540,672
normalization: 1,152
total: 1,334,144
embedding's share of the total: 40%
layers: 4
width: 128
heads: 4
feed-forward width: 352
longest text, in tokens: 160
vocabulary: 4,142
```
The model has 1,334,144 parameters, slightly fewer than the 1,394,732 of [chapter 5](#a-neural-language-model)'s window model. The embedding table, shared with the output layer, holds 530,176 of them, 40% of the total, because it has a row of 128 numbers for each of the 4,142 tokens. Those are the store tokenizer's vocabulary plus four markers: start, separator, end, and padding, which fills out short texts in a training batch. The feed-forward layers hold 540,672, attention 262,144, and the normalization gains 1,152.
The two models are about the same size and spend it differently. The window model put about three quarters of its parameters in its output layer, and could see four tokens back. The transformer puts most of its parameters in its blocks, shares its embeddings with its output, and can see every earlier token in a text of up to 160 tokens. [Chapter 8](#training-a-small-model) explains how the model was trained, and measures it against the baseline.
**Working rule:** Count parameters by part whenever you design or load a model, as here. At small sizes, the embedding table is a large share of the total, so tie it to the output layer, and choose the vocabulary size with that cost in mind, as [chapter 2's](#choose-how-many-merges) choice of merges did.
## What goes wrong in production {#transformer-in-production}
A transformer's parts have to fit together exactly, and most of what goes wrong is a part that almost fits:
- **Mismatched details can fail quietly.** Where the norms go, how RoPE pairs numbers and its base, whether the feed-forward layer is gated and whether the output is tied all have to match the trained weights. Some mismatches give nonsense at once; others, like the RoPE base [measured here](#measure-mismatched-details), cost a little accuracy with no error. Check against reference outputs.
- **Lengths beyond training are untested.** RoPE gives an angle for any position, but a model trained on texts of up to 160 tokens has never learned to use the angles beyond. Treat the training length as the model's limit until you've measured past it.
- **Large values can overflow at reduced precision.** Formats with fewer bits keep the same relative precision at every ordinary magnitude, but float16 can't hold values above 65,504, and in 8-bit formats a few large outliers ruin the precision of every other value ([Dettmers et al., 2022](https://arxiv.org/abs/2208.07339)). The store's model's stream peaks at 50.6 for the product measured here, far from float16's limit; check the largest values in your model's stream before serving it at lower precision.
- **Results depend on the machine.** As with every trained model in this book, keep the weights, the reference outputs and the exact training script together, and rerun the check after any change to the code or libraries.
- **Every part runs at every position.** The feed-forward layers cost the same at every position, and attention's cost grows with the square of the length, as [chapter 6](#attention-in-production) showed. Measure which dominates for your texts before you optimize either.
## What this chapter decided {#transformer-decisions}
- **The store's transformer uses the LLaMA-style block,** at a toy size: RMSNorm before each part, rotary positions in attention, a gated SiLU feed-forward layer, and an output layer tied to the embeddings. It has 4 layers, a width of 128, 4 heads and a feed-forward width of 352. SmolLM2-360M has the same kinds of parts, larger, with grouped-query attention and a RoPE base of 100,000.
- **The NumPy forward pass in `lib.transformer` is the book's reference,** checked against PyTorch's own RMSNorm, attention and SiLU on every position of a validation product, to four decimal places. Every later example that uses the model runs it.
- **Every block adds to the residual stream, and every part reads a normalized copy of it,** so the stream can grow through the layers while each part sees vectors of a fixed scale.
- **Parts are judged by measuring the model without them, or with them changed.** Leaving out any one block makes the store's model much worse, and leaving out the first makes it useless, so the book treats a trained model's blocks as one system. A wrong RoPE base costs a little and raises no error, so loaded weights are always checked against reference outputs.
- **The model is about the size of [chapter 5](#a-neural-language-model)'s window model, with its parameters spent on the blocks instead of the output layer.** The embedding table, shared with the output, is still 40% of it, so vocabulary size remains a cost the book watches.
# Training a small model
By the end of this chapter you can train a small transformer from start to finish and judge the result. You'll turn texts into batches, update the weights with AdamW, set the step size's schedule, read the training curve, measure the model fairly against the counted baseline, and find out what more width, more text and more training buy at this scale. You'll also see where the model beats counting, and where it needs counting beside it.
[Chapter 7](#the-transformer-block) assembled the store's model and used its trained weights to check every part. This chapter is about where those weights came from: one training run, recorded in `recordings/transformer.py`. The store is the online furniture and home-goods store this book follows, and its text is the names and descriptions of its products. The store's model learned from that text to predict each next token, and this chapter measures how well it does on the two jobs it's for: suggesting the next word of a product name in the search box, and predicting the store's descriptions, where text runs long enough for distant context to matter.
As in [chapter 5](#train-with-pytorch), training ran once, in PyTorch, because its results depend on the machine. The examples read what it saved: the weights, the curves and the settings, recorded in the run's manifest. To train it yourself, install `recordings/requirements.txt` in its own environment, which adds PyTorch's CPU build, and run `python recordings/transformer.py` from the book's folder: it writes the same files to `examples/data/recorded/transformer/`, and runs on a laptop's processor.
## Turn products into batches {#turn-products-into-batches}
Training starts by turning the store's catalog into numbers the model can learn from. Each product becomes one text: the start marker, the name, the separator ``, the description and the end marker, tokenized by the [store's tokenizer](#train-the-store-tokenizer). A text longer than the model's limit is cut. Each training step takes a **batch** of texts at once, so the arithmetic runs on whole matrices, and texts of different lengths are filled out to the longest in the batch with a padding token.
Each text then gives the model one prediction per position. The **inputs** are its tokens without the last, and the **targets** are its tokens without the first, so position i is trained to predict token i + 1. Thanks to the [causal mask](#hide-the-future), one forward pass makes every one of those predictions at once. The loss is averaged over the targets that aren't padding: the padding costs computation but teaches nothing.
For the store, tokenize every training product, shuffle the texts, and build one epoch's batches the way the training script does, measuring how much of each batch is padding:
Example `training-batches` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
import numpy as np
from lib import transformer
from lib.products import product_texts
_, vocabulary, manifest = transformer.load()
settings = manifest["settings"]
pad, end = vocabulary.index(transformer.PAD), vocabulary.index(transformer.END)
texts = [ids for ids, _ in product_texts("train")]
size = settings["batch"]
order = np.random.default_rng(settings["seed"]).permutation(len(texts)) # one shuffled epoch
padding, targets_seen, shares = 0, 0, []
for start in range(0, len(texts), size):
batch = [texts[i] for i in order[start:start + size]]
longest = max(len(ids) for ids in batch)
padded = np.array([ids + [pad] * (longest - len(ids)) for ids in batch]) # short texts filled out with
inputs, targets = padded[:, :-1], padded[:, 1:] # each position predicts the next token
padding, targets_seen = padding + (targets == pad).sum(), targets_seen + targets.size
shares.append((targets == pad).mean())
if start == 0:
first_shape = (inputs.shape, targets.shape)
print("training texts:", f"{len(texts):,}")
print("tokens in them:", f"{sum(len(ids) for ids in texts):,}")
print(f"texts cut at the length limit, {settings['longest']} tokens:", f"{np.mean([ids[-1] != end for ids in texts]):.1%}")
print("texts per batch:", size)
print("one batch's inputs and targets:", *first_shape)
print("share of targets that are padding, over one shuffled epoch:", f"{padding / targets_seen:.0%}")
print("least and most padding in a batch:", f"{min(shares):.0%} and {max(shares):.0%}")
print("steps per epoch:", f"{math.ceil(len(texts) / size):,}")
print("epochs in the recorded run:", settings["epochs"])
print("steps in the recorded run:", f"{manifest['steps']:,}")
```
Output:
```text
training texts: 34,252
tokens in them: 2,843,222
texts cut at the length limit, 160 tokens: 1.0%
texts per batch: 32
one batch's inputs and targets: (32, 151) (32, 151)
share of targets that are padding, over one shuffled epoch: 46%
least and most padding in a batch: 30% and 59%
steps per epoch: 1,071
epochs in the recorded run: 16
steps in the recorded run: 17,136
```
The training split holds 34,252 texts and 2,843,222 tokens, counting each text's start, separator and end markers, which [chapter 4's](#describe-by-neighbours) count of 2,744,320 leaves out, less what the length limit cuts, and only 1.0% of the texts are cut at the length limit. A batch of 32 texts gives inputs and targets of the shapes (32, 151) (32, 151): one row per text, one column per position. Over one shuffled epoch, 46% of the targets are padding, between 30% and 59% depending on the batch, because one long description sets the width for every shorter text beside it. One pass over the training texts, an **epoch**, takes 1,071 steps, and the recorded run took 17,136, 16 epochs.
Padding on that scale is waste: nearly half of each forward pass computes predictions the loss then ignores. It doesn't change what the model learns, since padding never counts, but it nearly doubles the cost of each step. Krell and colleagues found padding up to half of all tokens in common datasets, and proposed **packing**, several texts end to end in each row, with the attention kept from crossing between them ([Krell et al., 2021](https://arxiv.org/abs/2107.02027)). Putting texts of similar length in the same batch reduces padding too.
**Working rule:** Build batches as inputs and shifted targets, mask padding out of the loss, and shuffle the texts each epoch with a fixed seed, so the run can be repeated. Measure the share of padding in your batches, as here; if it's large, pack texts or group them by length before you buy a bigger machine.
## Update with AdamW and clipped gradients {#update-with-adamw}
With batches in hand, each step is the loop of [chapter 5](#take-gradient-descent-steps): forward pass, loss, [gradients](#compute-gradients), update. These lines of `recordings/transformer.py` are the whole of it, run once per batch:
```python
ids = pad([train[i] for i in order[start:start + SETTINGS["batch"]]])
logits = model(ids[:, :-1])
loss = torch.nn.functional.cross_entropy(logits.transpose(1, 2), ids[:, 1:], ignore_index=index[PAD])
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
scheduler.step()
```
The first three lines are the [previous section](#turn-products-into-batches): a padded batch, inputs without the last token, and a loss over the shifted targets that ignores padding. Two lines differ from [chapter 5](#a-neural-language-model).
The optimizer is **AdamW**, the [Adam](#train-with-pytorch) of [chapter 5](#a-neural-language-model) with **weight decay** applied separately: at every step, each weight is also reduced by its value times the step size times a decay factor, here 0.1 ([PyTorch's AdamW](https://docs.pytorch.org/docs/2.14/generated/torch.optim.AdamW.html)). Loshchilov and Hutter showed that applying the decay this way, apart from the gradient, works better with Adam than adding it to the loss ([Loshchilov and Hutter, 2019](https://arxiv.org/abs/1711.05101)). Decay is a form of [regularization](#stop-at-the-best-epoch), which chapter 5's window model lacked: it pulls every weight towards zero, so a weight stays large only while the training text keeps pushing it there.
The script also **clips** the gradients: when their combined length, measured as if every gradient were one long vector, exceeds 1.0, it scales them all down to that length ([PyTorch's clip_grad_norm_](https://docs.pytorch.org/docs/2.14/generated/torch.nn.utils.clip_grad_norm_.html)). One unusual batch then can't throw the weights far off. Pascanu and colleagues proposed clipping for exactly that: rare, very large gradients that would otherwise undo much of what training had learned ([Pascanu et al., 2013](https://arxiv.org/abs/1211.5063)).
**Working rule:** Use AdamW with a modest weight decay, and clip gradients by their combined length. Keep the training step short enough to read whole, as here, so that every choice in it is visible: what the loss ignores, what the optimizer decays and what gets clipped.
## Warm up, then decay the step size {#warm-up-then-decay}
The last line of the loop, `scheduler.step()`, changes the step size after every batch, following a **schedule**. For the first 200 steps, it rises in a straight line from nearly zero to its peak, 0.003: a **warm-up**, as the transformer paper's schedule had ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762), section 5.3). In Adam's first steps, its running average of squared gradients rests on only a few batches, so its step sizes vary widely; Liu and colleagues showed that warm-up works by reducing that variance ([Liu et al., 2020](https://arxiv.org/abs/1908.03265)). After the warm-up, the step size falls along half a cosine curve, to a tenth of the peak by the last step ([Loshchilov and Hutter, 2017](https://arxiv.org/abs/1608.03983)):
rate = peak × (0.1 + 0.9 × ½ (1 + cos(π × progress)))
where progress runs from 0 at the end of the warm-up to 1 at the last step. Large steps early cover ground; small steps late let the weights settle into a low point instead of bouncing around it.
For the store, compute the schedule by hand from the recorded settings, and compare it with the step sizes the training script saved as it ran, every tenth step:
Example `learning-rate-schedule` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import json
import math
from lib import transformer
_, _, manifest = transformer.load()
settings, total = manifest["settings"], manifest["steps"]
recorded_rates = json.loads((transformer.RECORDED / "curves.json").read_text())["rates"] # every 10th step's
def rate(step):
"""Warm up linearly to the peak, then follow half a cosine down to a tenth of it."""
if step < settings["warmup_steps"]:
return settings["peak_rate"] * (step + 1) / settings["warmup_steps"]
progress = (step - settings["warmup_steps"]) / (total - settings["warmup_steps"])
return settings["peak_rate"] * (0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * progress)))
print("warm-up steps:", settings["warmup_steps"])
print("peak step size:", settings["peak_rate"])
print("weight decay:", settings["weight_decay"])
for step in [0, settings["warmup_steps"] // 2, settings["warmup_steps"], total // 2, total - 1]:
print(f"step size at step {step:,}:", f"{rate(step):.2e}")
gap = max(abs(recorded - rate(10 * i + 1)) for i, recorded in enumerate(recorded_rates)) # saved after each step
print("same as the recorded step sizes:", gap < 1e-9)
```
Output:
```text
warm-up steps: 200
peak step size: 0.003
weight decay: 0.1
step size at step 0: 1.50e-05
step size at step 100: 1.51e-03
step size at step 200: 3.00e-03
step size at step 8,568: 1.68e-03
step size at step 17,135: 3.00e-04
same as the recorded step sizes: True
```
The step size starts at 1.50e-05, reaches 1.51e-03 halfway through the warm-up and its peak, 3.00e-03, at the warm-up's end. Halfway through training it's 1.68e-03, and at the last step 3.00e-04. The hand calculation matches every saved value: True.
The decay matters, and the recordings show how much. The comparison run trained for four epochs at the same width, with the same seed and the same batches, so its schedule finished its decay at step 4,284. It scored 27.6 on the validation names at that step. The recorded run, at the same step, was still near its peak step size, and scored 32.6. Same data, same steps: the difference is the decay.
**Working rule:** Warm the step size up over the first few hundred steps, then decay it, and match the schedule's length to the run's. Save the step size as training runs, as the training script does, and check it against the schedule you meant: a schedule bug looks like a training problem, and is far easier to find this way.
## Read the training curve {#read-the-training-curve}
With its batches, updates and schedule set, the store's model trained for 16 epochs, and the script measured it on the validation split after each one, on names and descriptions separately. It kept the weights from the epoch with the lowest validation loss on names and descriptions added together, as [chapter 5](#stop-at-the-best-epoch) kept its best epoch.
For the store, read the recorded curves as [perplexity](#score-with-perplexity) per token:
Example `transformer-training-curve` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import json
import math
from lib import transformer
_, _, manifest = transformer.load()
curves = json.loads((transformer.RECORDED / "curves.json").read_text())["epochs"]
print("perplexity per token; training mixes names and descriptions, as the training text does")
print("epoch training validation names validation descriptions")
for curve in curves:
print(f"{curve['epoch']:>5} {math.exp(curve['train_loss']):>8.1f} {math.exp(curve['validation_name_loss']):>16.1f}"
f" {math.exp(curve['validation_description_loss']):>23.1f}")
kept = next(curve for curve in curves if curve["epoch"] == manifest["kept_epoch"])
print("epoch kept, the lowest name and description losses added together:", manifest["kept_epoch"])
print("validation perplexity on names, kept epoch:", f"{math.exp(kept['validation_name_loss']):.1f}")
print("validation perplexity on descriptions, kept epoch:", f"{math.exp(kept['validation_description_loss']):.1f}")
print("validation perplexity on names, epoch 4:", f"{math.exp(curves[3]['validation_name_loss']):.1f}")
```
Output:
```text
perplexity per token; training mixes names and descriptions, as the training text does
epoch training validation names validation descriptions
1 131.4 50.5 50.4
2 41.4 38.8 36.2
3 32.8 34.5 31.5
4 29.0 32.6 29.3
5 26.6 30.9 27.8
6 24.8 30.2 26.4
7 23.3 28.9 25.0
8 21.8 28.1 23.9
9 20.5 27.1 22.9
10 19.2 26.5 22.2
11 17.9 25.8 21.6
12 16.8 25.1 20.8
13 15.8 24.6 20.4
14 14.9 24.2 20.0
15 14.2 24.0 19.9
16 13.8 24.1 19.9
epoch kept, the lowest name and description losses added together: 15
validation perplexity on names, kept epoch: 24.0
validation perplexity on descriptions, kept epoch: 19.9
validation perplexity on names, epoch 4: 32.6
```
Validation perplexity falls until epoch 15, the epoch kept, to 24.0 on names and 19.9 on descriptions, and then levels off: the last epoch is no better. As in [chapter 5](#a-neural-language-model), epoch 1's training perplexity is above validation, because it averages over the epoch's first batches, when the model knew nothing. The training column mixes names and descriptions, as the training text does, so it shows the trend, not a number to set beside either validation column.
[Chapter 5](#a-neural-language-model)'s window model stopped improving after epoch 5; this model kept improving for far longer. The two runs differ in several ways that could explain it. This model learned from names and descriptions, 2,843,222 tokens, against the window model's 337,071 from names alone; its step size decays; it has weight decay; and attention applies the same weights at every position. The run doesn't separate those reasons. What it does show is the gap between training and validation, which grows every epoch: by the end, the model predicts its training texts far better than new ones, so it's starting to memorize.
Descriptions score lower than names, but that doesn't mean the model is better at descriptions. Each kind of text has its own predictability: descriptions repeat phrases from their templates and their names, and names pack a new product into a few words. Perplexities compare models on the same text, never texts with each other.
**Working rule:** Record validation loss per epoch on each kind of text you care about, and keep the best epoch's weights. Read the gap between training and validation as the warning, not the training loss alone. Never compare perplexities across different kinds of text.
## Compare with the baseline on names {#compare-on-names}
With the training read, the result can be measured. The book's yardstick is the validation names: every model since [chapter 3](#counting-further-back) has been scored on them, and the [4-gram baseline](#the-baseline), trained on the training names, scores 23.1. That baseline is the strongest counted one for names: the same 4-gram trained on whole texts, names and descriptions, scores 27.7 on them, because descriptions dilute its counts of how names run.
The comparison has to count the same predictions. The 4-gram predicts each name's tokens and then its end marker; the transformer predicts each name's tokens and then ``, which plays the same part, since it marks where the name ends. A name needs nothing after it, so the transformer reads the name alone.
Before you run it, predict whether the transformer, alone, beats the counted baseline that [chapter 5](#a-neural-language-model)'s window model couldn't. Then score both models, and mixtures of the two, as [chapter 5](#mix-with-the-counts) did:
Example `transformer-on-names` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.names import name_tokens
from lib.ngrams import KneserNey, ngrams, padded
_, vocabulary, _ = transformer.load()
index = {token: i for i, token in enumerate(vocabulary)}
counted = KneserNey(name_tokens("train"), 4) # chapter 3's baseline, trained on names
from_counts, from_transformer = [], []
for name in name_tokens("validation"):
ids = [index[transformer.START]] + [index[token] for token in name] + [index[transformer.SEP]]
probabilities = transformer.softmax(transformer.forward(ids)[:-1]) # the name's tokens, then : its end
from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])
from_counts += [counted.probability(context, token) for context, token in ngrams(padded(name, 4), 4)] # then
from_counts, from_transformer = np.array(from_counts), np.array(from_transformer)
print("validation tokens scored, each name's end included:", f"{len(from_counts):,}")
for share in [0.0, 0.3, 0.5, 0.7, 1.0]:
mixed = (1 - share) * from_counts + share * from_transformer
print(f"validation perplexity on names, transformer share {share:.1f}:", f"{np.exp(-np.log(mixed).mean()):.1f}")
```
Output:
```text
validation tokens scored, each name's end included: 46,704
validation perplexity on names, transformer share 0.0: 23.1
validation perplexity on names, transformer share 0.3: 18.7
validation perplexity on names, transformer share 0.5: 18.4
validation perplexity on names, transformer share 0.7: 18.8
validation perplexity on names, transformer share 1.0: 24.0
```
Both models score 46,704 predictions. Alone, the transformer scores 24.0, slightly worse than the baseline's 23.1, though far better than [chapter 5](#a-neural-language-model)'s window model, 36.4. Mixed half and half, the two score 18.4, the best of any model in this book on names; chapter 5's best mix scored 21.9.
So on names, counting is still hard to beat alone, for the reason [chapter 5](#a-neural-language-model) gave: names are short and formulaic, and three tokens of context see most of what a name holds. But the transformer is wrong in different places from the counts, and the mix gains from both. The share was chosen on the validation names, so the gain is slightly optimistic; the test names stay untouched for the book's final comparison.
**Working rule:** Compare a new model with the strongest baseline you can build on exactly the same predictions, including how each marks the end of a text. When a learned model and a counted one are close, mix them, and choose the share on held-out text.
## Compare on descriptions {#compare-on-descriptions}
The [names comparison](#compare-on-names) favours counting, because names are short. Descriptions run many times longer, and [chapter 6](#test-what-the-model-copies) showed the model copying from the name many tokens back, so this is where its longer reach should pay. A fair baseline here is a 4-gram trained on the same texts the transformer learned from, the name, `` and description joined, and scored on the same tokens: each description's tokens and its end, with the same text before them.
For the store, fit that 4-gram on every training text, and score both models, and an even mix, on the descriptions of the first 500 validation products, leaving out the texts cut at the length limit and the products with no description. Then split the tokens, as [chapter 5](#compare-with-the-baseline) did, by whether their three-token context occurs in the training texts:
Example `transformer-on-descriptions` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.names import name_tokens
from lib.ngrams import KneserNey, ngrams, padded
from lib.products import product_texts
_, vocabulary, _ = transformer.load()
end = vocabulary.index(transformer.END)
as_tokens = lambda ids: [vocabulary[i] for i in ids[1:] if i != end] # padded() adds its own and
counted = KneserNey([as_tokens(ids) for ids, _ in product_texts("train")], 4) # whole texts: name, , description
from_counts, from_transformer, seen, products = [], [], [], 0
for ids, sep in product_texts("validation", 500):
if ids[-1] != end or sep == len(ids) - 2:
continue # cut at the length limit, or no description
pairs = list(ngrams(padded(as_tokens(ids), 4), 4))[sep:] # the description's tokens, then
from_counts += [counted.probability(context, token) for context, token in pairs]
seen += [context in counted.totals[4] for context, _ in pairs] # was this 3-token context in the training texts?
probabilities = transformer.softmax(transformer.forward(ids)[sep:-1])
from_transformer += list(probabilities[np.arange(len(probabilities)), ids[sep + 1:]])
products += 1
from_counts, from_transformer, seen = np.array(from_counts), np.array(from_transformer), np.array(seen)
perplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"
print("products scored:", products)
print("description tokens scored:", f"{len(from_counts):,}")
mixed = (from_counts + from_transformer) / 2
for label, p in [("4-gram baseline", from_counts), ("transformer", from_transformer), ("half and half", mixed)]:
print(f"{label}, validation perplexity on descriptions:", perplexity(p))
print("description tokens whose 3-token context is in the training texts:", f"{seen.mean():.0%}")
for part, where in [("seen", seen), ("unseen", ~seen)]:
print(f"4-gram, {part} contexts:", perplexity(from_counts[where]))
print(f"transformer, {part} contexts:", perplexity(from_transformer[where]))
names = [list(ngrams(padded(name + [transformer.SEP], 4), 4))[:-1] for name in name_tokens("validation")] # to
on_names = [counted.probability(context, token) for pairs in names for context, token in pairs]
print("this 4-gram, trained on whole texts, validation perplexity on names:", perplexity(np.array(on_names)))
```
Output:
```text
products scored: 446
description tokens scored: 35,651
4-gram baseline, validation perplexity on descriptions: 22.5
transformer, validation perplexity on descriptions: 20.1
half and half, validation perplexity on descriptions: 14.7
description tokens whose 3-token context is in the training texts: 74%
4-gram, seen contexts: 16.3
transformer, seen contexts: 16.6
4-gram, unseen contexts: 57.6
transformer, unseen contexts: 35.0
this 4-gram, trained on whole texts, validation perplexity on names: 27.7
```
On 35,651 tokens from 446 products, the 4-gram scores 22.5 and the transformer 20.1: on descriptions, the transformer beats counting alone, the first learned model in this book to beat a counted model of the same text on its own.
The split shows where. 74% of the tokens have a context the training texts contain, and on those the two models tie: 16.3 for the 4-gram, 16.6 for the transformer. On the rest, the transformer scores 35.0 where the 4-gram scores 57.6. Descriptions are formulaic enough that counts do as well as the transformer wherever they have seen the context; where they haven't, the transformer reads further back and relates what it sees to what it learned, and the counts can only back off.
That's also why the mix works so well. Half and half, the two score 14.7, far below either alone: on any one token, at least one of the two is often confident and right, and the even mix keeps half of that probability, enough to avoid the worst surprises of each.
**Working rule:** Measure a model on the kind of text it will predict, against a baseline that sees the same text, and expect the ranking to change from one kind of text to another. Split the comparison by seen and unseen contexts to see where each model wins. Try a mix before concluding that either model is enough: here, the mix gained more than any change to either model that this chapter measured.
## What width, data and steps buy {#what-width-data-and-steps-buy}
The recorded model is one choice among many: 128 numbers wide, trained on all the training text for 16 epochs. Studies of much larger models found that loss falls smoothly as models get bigger, see more text and train longer, and that bigger models learn more from each token they see ([Kaplan et al., 2020](https://arxiv.org/abs/2001.08361)). For a fixed budget of computation, Hoffmann and colleagues found that model size and training text should grow together ([Hoffmann et al., 2022](https://arxiv.org/abs/2203.15556)). And Muennighoff and colleagues found that repeating the same text for up to about four epochs is nearly as good as new text, with less gained from each further repetition ([Muennighoff et al., 2023](https://arxiv.org/abs/2305.16264)). Whether any of that holds at this toy scale is something to measure, not assume.
So the training script recorded seven cheaper runs beside the main one, each changing one thing:
- three widths, 64, 128 and 256, each trained for four epochs, with every head kept 32 numbers wide;
- random quarters and halves of the training products, each trained for four epochs;
- the same quarter and half trained for as many steps as all the data's four epochs, by repeating them more often.
The numbers are the validation perplexities the script recorded, at each run's best epoch and at its last. For the store, read all eight runs:
Example `width-data-and-steps` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import json
import math
from lib import transformer
runs = [("width 64, 4 epochs", "transformer-width-64"), ("width 128, 4 epochs", "transformer-width-128"),
("width 256, 4 epochs", "transformer-width-256"), ("width 128, 16 epochs", "transformer"),
("a quarter of the data, 4 epochs", "transformer-data-25"), ("half the data, 4 epochs", "transformer-data-50"),
("a quarter of the data, 16 epochs", "transformer-data-25-same-steps"),
("half the data, 8 epochs", "transformer-data-50-same-steps")]
for label, folder in runs:
run = transformer.RECORDED.parent / folder
manifest = json.loads((run / "manifest.json").read_text())
curves = json.loads((run / "curves.json").read_text())["epochs"]
best = min(curves, key=lambda curve: curve["validation_name_loss"] + curve["validation_description_loss"])
print(f"{label}, parameters:", f"{manifest['parameters']:,}")
print(f"{label}, steps:", f"{manifest['steps']:,}")
print(f"{label}, best epoch:", best["epoch"])
print(f"{label}, perplexity on names:", f"{math.exp(best['validation_name_loss']):.1f}")
print(f"{label}, perplexity on descriptions:", f"{math.exp(best['validation_description_loss']):.1f}")
print(f"{label}, perplexity on names, last epoch:", f"{math.exp(curves[-1]['validation_name_loss']):.1f}")
print(f"{label}, perplexity on descriptions, last epoch:", f"{math.exp(curves[-1]['validation_description_loss']):.1f}")
```
Output:
```text
width 64, 4 epochs, parameters: 478,656
width 64, 4 epochs, steps: 4,284
width 64, 4 epochs, best epoch: 4
width 64, 4 epochs, perplexity on names: 36.9
width 64, 4 epochs, perplexity on descriptions: 35.9
width 64, 4 epochs, perplexity on names, last epoch: 36.9
width 64, 4 epochs, perplexity on descriptions, last epoch: 35.9
width 128, 4 epochs, parameters: 1,334,144
width 128, 4 epochs, steps: 4,284
width 128, 4 epochs, best epoch: 4
width 128, 4 epochs, perplexity on names: 27.6
width 128, 4 epochs, perplexity on descriptions: 24.7
width 128, 4 epochs, perplexity on names, last epoch: 27.6
width 128, 4 epochs, perplexity on descriptions, last epoch: 24.7
width 256, 4 epochs, parameters: 4,273,920
width 256, 4 epochs, steps: 4,284
width 256, 4 epochs, best epoch: 4
width 256, 4 epochs, perplexity on names: 23.5
width 256, 4 epochs, perplexity on descriptions: 19.5
width 256, 4 epochs, perplexity on names, last epoch: 23.5
width 256, 4 epochs, perplexity on descriptions, last epoch: 19.5
width 128, 16 epochs, parameters: 1,334,144
width 128, 16 epochs, steps: 17,136
width 128, 16 epochs, best epoch: 15
width 128, 16 epochs, perplexity on names: 24.0
width 128, 16 epochs, perplexity on descriptions: 19.9
width 128, 16 epochs, perplexity on names, last epoch: 24.1
width 128, 16 epochs, perplexity on descriptions, last epoch: 19.9
a quarter of the data, 4 epochs, parameters: 1,334,144
a quarter of the data, 4 epochs, steps: 1,072
a quarter of the data, 4 epochs, best epoch: 4
a quarter of the data, 4 epochs, perplexity on names: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions: 54.1
a quarter of the data, 4 epochs, perplexity on names, last epoch: 51.9
a quarter of the data, 4 epochs, perplexity on descriptions, last epoch: 54.1
half the data, 4 epochs, parameters: 1,334,144
half the data, 4 epochs, steps: 2,144
half the data, 4 epochs, best epoch: 4
half the data, 4 epochs, perplexity on names: 35.2
half the data, 4 epochs, perplexity on descriptions: 34.8
half the data, 4 epochs, perplexity on names, last epoch: 35.2
half the data, 4 epochs, perplexity on descriptions, last epoch: 34.8
a quarter of the data, 16 epochs, parameters: 1,334,144
a quarter of the data, 16 epochs, steps: 4,288
a quarter of the data, 16 epochs, best epoch: 11
a quarter of the data, 16 epochs, perplexity on names: 40.0
a quarter of the data, 16 epochs, perplexity on descriptions: 39.6
a quarter of the data, 16 epochs, perplexity on names, last epoch: 44.1
a quarter of the data, 16 epochs, perplexity on descriptions, last epoch: 45.9
half the data, 8 epochs, parameters: 1,334,144
half the data, 8 epochs, steps: 4,288
half the data, 8 epochs, best epoch: 8
half the data, 8 epochs, perplexity on names: 30.4
half the data, 8 epochs, perplexity on descriptions: 28.6
half the data, 8 epochs, perplexity on names, last epoch: 30.4
half the data, 8 epochs, perplexity on descriptions, last epoch: 28.6
```
**Width** helps a lot, at a price. With the same steps, the names' perplexity falls from 36.9 at width 64 to 27.6 at 128 and 23.5 at 256. The widest model, trained for four epochs, even beats the recorded model trained for sixteen, 24.0 on names. But it has 4,273,920 parameters against 1,334,144, and each step costs about that much more.
**New text** helps at no extra cost per step. With the same steps, a random quarter of the products scores 40.0 on names at its best, half 30.4, and all of them 27.6. Doubling the text from half to all gained less than doubling the width from 128 to 256, but it cost no extra computation, and the first doubling, from a quarter to half, gained more than either.
**Repetition** helps for a while, then hurts. A quarter of the text seen four times scores 51.9; seen more often, it improves until epoch 11, then gets worse, ending at 44.1 after sixteen passes: the model memorizes what it has seen too often. Repeated text never caught up with new text at the same number of steps.
**More steps** help least. The recorded model's sixteen epochs improve on four, 24.0 against 27.6 on names, for four times the steps.
These runs also show the limits of this book's scale. The store's training text is fixed at what its catalog holds, 2,843,222 tokens. SmolLM2-360M, whose design [chapter 7](#the-transformer-block) compared with the store's model, was trained on 4 trillion tokens ([model card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/blob/f8027fd0eaeea54caa13c31d31b9fdc459c38b49/README.md)), more than a million times as much, which lets a model 960 numbers wide and 32 layers deep keep learning from new text instead of memorizing old.
**Working rule:** Get more text of the kind you'll predict before repeating what you have more than a few times; it costs nothing per step. With computation to spare, try a wider model for fewer epochs before training a narrow one longer, and measure both. Change one thing per run and hold the number of steps fixed where it matters, or the comparison measures two things at once.
## Suggest the next word in the search box {#autocomplete-in-the-search-box}
Averages say how good a model is; the search box shows what that means to a shopper. [Chapter 3's](#autocomplete-with-more-context) 4-gram suggested `&` and `with` after `outdoor bar`, a context no training name contains, and [chapter 5's](#suggest-from-similar-contexts) window model suggested `stool`. A few training descriptions do contain `outdoor bar`, and the transformer learned from them as well as from the names, while the 4-gram learned from names alone.
For the store, ask the transformer, and the half-and-half mix the [names comparison](#compare-on-names) favoured, for their three likeliest next tokens after the same five prefixes, each treated as the start of a name. A suggestion of `` means the model expects the name to end there:
Example `transformer-autocomplete` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import bpe, transformer
from lib.names import name_tokens
from lib.ngrams import KneserNey
_, vocabulary, _ = transformer.load()
index = {token: i for i, token in enumerate(vocabulary)}
counted = KneserNey(name_tokens("train"), 4)
tokenizer = bpe.store_tokenizer()
as_counted = [{transformer.SEP: ""}.get(token, token) for token in vocabulary] # ends a name, as does
show = lambda token: token.replace("", "")
for prefix in ["bar", "patio bar", "outdoor bar", "kitchen bar", "wine bar"]:
tokens = tokenizer.encode(prefix)
ids = [index[transformer.START]] + [index[token] for token in tokens]
from_transformer = transformer.softmax(transformer.forward(ids)[-1])
context = tuple([""] * 3 + tokens)[-3:]
from_counts = np.array([counted.probability(context, token) for token in as_counted])
mixed = (from_transformer + from_counts) / 2
for label, p in [("transformer", from_transformer), ("half and half", mixed)]:
top = np.argsort(-p, kind="stable")[:3]
print(f"{prefix:<12} {label:<13} -> " + ", ".join(f"{show(vocabulary[i])} {p[i]:.2f}" for i in top))
if prefix == "outdoor bar":
stool = index["stool"]
print("P(stool | outdoor bar), transformer:", f"{from_transformer[stool]:.2f}")
print("P(stool | outdoor bar), 4-gram:", f"{from_counts[stool]:.2f}")
print("P(stool | outdoor bar), half and half:", f"{mixed[stool]:.2f}")
```
Output:
```text
bar transformer -> with 0.17, & 0.14, and 0.09
bar half and half -> & 0.19, with 0.14, unit 0.07
patio bar transformer -> stool 0.51, table 0.25, set 0.02
patio bar half and half -> stool 0.69, table 0.13, & 0.01
outdoor bar transformer -> stool 0.50, with 0.05, table 0.04
outdoor bar half and half -> stool 0.28, & 0.12, with 0.12
P(stool | outdoor bar), transformer: 0.50
P(stool | outdoor bar), 4-gram: 0.06
P(stool | outdoor bar), half and half: 0.28
kitchen bar transformer -> room 0.14, with 0.08, cabinet 0.08
kitchen bar half and half -> with 0.14, & 0.12, room 0.07
wine bar transformer -> with 0.43, 0.22, cabinet 0.08
wine bar half and half -> with 0.30, 0.21, cabinet 0.13
```
After `outdoor bar`, the transformer gives `stool` a probability of 0.50, against the 4-gram's 0.06, and the mix 0.28, still its top suggestion. The control prefixes show the models aren't suggesting `stool` after every `bar`: after `kitchen bar` and `wine bar` they suggest other continuations, and after `wine bar` both expect, with good reason, that the name may already be complete. After `patio bar`, a context the names contain often, the mix is more confident in `stool` than the transformer alone.
One prefix is an example, not a measure. The measure is the names comparison, and there the mix won. For the search box, that's the model to serve.
**Working rule:** Serve the mix for autocomplete, and test it on rare prefixes as well as common ones, with control prefixes beside each, as here. Treat a single striking suggestion as an illustration of a measured gain, never as the evidence for it.
## What goes wrong in production {#training-in-production}
Training goes wrong in ways that look fine while they happen:
- **Padding wastes computation.** Nearly half of each step's predictions were padding here. Pack texts, or group them by length, before you buy more computation.
- **A schedule bug looks like a model problem.** As a rule of thumb, a warm-up that never ends, or a decay that starts too early, shows up as a loss that falls too slowly; the decay comparison above shows how much a schedule can move the result. Save the step size, and check it against the schedule you meant.
- **Choices made on validation data flatter the result.** The kept epoch and the mixing share were both chosen on the validation split, so its numbers are slightly optimistic. Keep a test split untouched until the end.
- **Repeated text gets memorized.** A small dataset seen many times looks fine on the training loss and poor on new text. Count how many times training sees each text, and prefer new text to more epochs.
- **One model is measured on one kind of text.** The transformer ranked second to the counts on names and first on descriptions. Score every model on each kind of text it will serve before choosing which to deploy.
- **Results depend on the machine.** PyTorch doesn't guarantee identical results across releases or platforms ([Reproducibility](https://docs.pytorch.org/docs/2.14/notes/randomness.html)). Record the weights, the curves and the exact script, as `recordings/transformer.py` does with the script's hash, and keep them together.
## What this chapter decided {#training-decisions}
- **The store's model is trained in one recorded run:** batches of 32 product texts, AdamW with weight decay 0.1, gradients clipped, a warm-up of 200 steps to a peak step size of 0.003, cosine decay to a tenth, and the weights kept from epoch 15 of 16.
- **Every comparison counts the same predictions.** On names, both models predict each name's tokens and its end, against [chapter 3](#counting-further-back)'s names-only baseline, the stronger one for names. On descriptions, a 4-gram trained on the same texts scores the same tokens, with the same text before them.
- **On the validation names, the transformer alone roughly ties the 4-gram baseline, and mixed half and half with it scores 18.4,** the best on names in this book so far. The mix is the model the book uses for autocomplete, replacing chapter 5's mix of the window model and the counts.
- **From this chapter on, models are also measured on the validation descriptions.** There the transformer beats the baseline alone, 20.1 against 22.5, by winning where the counts haven't seen the context, and the mix scores 14.7.
- **At this scale, new text is the cheapest gain, width the largest at a price, and repetition helps only for a few passes.** The recorded model stays at width 128, because every example from [chapter 6](#attention) on reads it and the examples have to run in seconds on a laptop; a wider one would score better.
# Choosing the next token
By the end of this chapter you can turn a language model's probabilities into text, and choose how: always the likeliest token, the likeliest whole text by beam search, or a sample shaped by a temperature, a top-k or top-p cut, a repetition penalty and a minimum length. You'll measure what each choice does to the text, and learn which failures no choice of setting can fix.
A language model gives a probability for every possible next token; it doesn't write. To write, something has to choose one token, append it, ask for the next probabilities, and repeat until the model predicts the end marker, ``. That choosing is called **decoding**, and it decides much of what the text reads like, with the model unchanged.
The store, the online furniture and home-goods store this book follows, has a reason to write. Of its 4,282 validation products, 597 have no description at all, and so do 14% of its training products. The store's model, the transformer that [chapter 8](#training-a-small-model) trained and measured, learned descriptions from the store's own text, so it can draft one from a product's name: start the text with the name and ``, as every training text was laid out, and let the model continue. This chapter does that with greedy decoding, beam search and sampling, using the book's shared module `lib.decoding`, which builds each method in NumPy on top of `lib.transformer`. Then it measures what each method writes against the store's real descriptions, and fixes two failures that the measurements expose: repetition, and descriptions that come out empty.
## Choose the likeliest token {#choose-the-likeliest-token}
The simplest decoder is **greedy**: at every step, take the single likeliest token. It needs no randomness, so it always writes the same text for the same start, and each step costs one [forward pass](#tie-the-output) of the model. The text it writes is, token by token, the model's best guess.
For the store, take a product with no description, `front porch welcome sign wall décor`, and decode greedily for up to 60 tokens. Then, for a second product, `5 light light bulb`, print the probability that the description ends before it starts:
Example `greedy-description` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import decoding, transformer
from lib.store import load_products
validation, training = load_products("validation"), load_products("train")
empty = lambda rows: rows["description"].fillna("").str.strip() == ""
print("validation products:", f"{len(validation):,}")
print("validation products with no description:", f"{empty(validation).sum():,}")
print("training products with no description:", f"{empty(training).mean():.0%}")
bulbs = training[training["name"].str.contains(r"\bbulbs?\b")]
print("training products named as bulbs with no description:", f"{empty(bulbs).mean():.0%}")
longest = 60
print("longest description, in tokens:", longest)
ids = decoding.generate(decoding.prompt("front porch welcome sign wall décor"), longest=longest) # always the likeliest
print("greedy description:", decoding.text(ids))
fourgrams = Counter(tuple(ids[i:i + 4]) for i in range(len(ids) - 3))
print("4-token phrases it uses twice or more:", " | ".join(decoding.text(list(g)) for g, n in fourgrams.items() if n > 1))
end = transformer.load()[1].index(transformer.END)
probabilities = transformer.softmax(decoding.next_logits(decoding.prompt("5 light light bulb")))
print("P(the description is empty), 5 light light bulb:", f"{probabilities[end]:.2f}")
```
Output:
```text
validation products: 4,282
validation products with no description: 597
training products with no description: 14%
training products named as bulbs with no description: 24%
longest description, in tokens: 60
greedy description: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a distressed finish . this welcome sign is made of wood and has a distressed finish that adds a rustic touch to your home . this sign is made from
4-token phrases it uses twice or more: this welcome sign is | to your home .
P(the description is empty), 5 light light bulb: 0.28
```
The greedy description reads like the store's text, because it is the store's text, recombined: `this welcome sign is a great way to add a touch of rustic charm to your home`. Then it starts to loop. It writes `features a distressed finish with a distressed finish`, and later `has a distressed finish` again, and it uses two four-token phrases twice, `this welcome sign is` and `to your home .`, which the example lists. Greedy decoding loops because the same likeliest continuation wins every time the text reaches a similar point: `distressed` is the model's favourite finish for a sign, so it comes back whenever a finish is due. Holtzman and colleagues found the same in large models: decoding that always maximizes probability leads to text that is "bland, incoherent, or gets stuck in repetitive loops" ([Holtzman et al., 2020](https://arxiv.org/abs/1904.09751)).
The description also says the sign is `made of natural wood`, which nothing in the product's name says, and the bulb shows a different failure. The model gives an immediate end, an empty description, a probability of 0.28. In the training text, 24% of the products named as bulbs have no description, against 14% of all products: the model learned the catalog as it is, gaps included, and a [later section](#forbid-an-empty-description) deals with it.
**Working rule:** Use greedy decoding when you want the same output every time and short text, such as a search-box suggestion, where the likeliest next word is the point. For anything longer than a phrase, expect it to repeat itself, and choose one of the methods in the [sections that follow](#search-for-the-likeliest-text).
## Search for the likeliest text {#search-for-the-likeliest-text}
[Greedy decoding](#choose-the-likeliest-token) picks the likeliest token at each step, which isn't the same as the likeliest text: a slightly less likely first word can lead to much likelier words after it. **Beam search** looks further. It keeps the k likeliest partial texts, the beam, scored by their **total log-probability**, the sum of the logarithms of their tokens' probabilities, where higher means likelier. At each step it extends every text in the beam by its k likeliest next tokens, and keeps the best k of the results. A text that reaches the end marker is finished and stays in the running. The heart of `lib.decoding.beam` is the line that scores each extension:
```python
candidates += [(score + log_p[token], written + [int(token)]) for token in np.argsort(-log_p, kind="stable")[:width]]
```
For the store, use a beam of 4 twice. First, complete a search query, `outdoor bar`, by up to 4 tokens, the end of the name included, the way a search box might offer whole phrases. Then write the welcome sign's description, and print the likeliest text overall, and the likeliest per token, its total log-probability divided by its length. Before you run it, predict the welcome sign's likeliest description:
Example `beam-search` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import decoding, transformer
_, vocabulary, _ = transformer.load()
width, longest = 4, 4
print("beam width:", width)
print("longest completion, in tokens, its end included:", longest)
start = decoding.prompt("outdoor bar")[:-1] # a search query: and the words typed so far
for score, written in decoding.beam(start, width=width, longest=longest, stops=(transformer.SEP,))[:2]:
print("completion of outdoor bar:", decoding.text([t for t in written if vocabulary[t] != transformer.SEP]),
f"(probability {np.exp(score):.3f})")
kept = decoding.beam(decoding.prompt("front porch welcome sign wall décor"), width=width)
score, written = kept[0] # the likeliest text overall
print("likeliest description:", decoding.text(written) or "(empty)")
print("its probability:", f"{np.exp(score):.2f}")
score, written = max(kept, key=lambda candidate: candidate[0] / len(candidate[1])) # the likeliest per token
print("likeliest description per token:", decoding.text(written))
```
Output:
```text
beam width: 4
longest completion, in tokens, its end included: 4
completion of outdoor bar: stool (probability 0.365)
completion of outdoor bar: stool cushion (probability 0.089)
likeliest description: (empty)
its probability: 0.15
likeliest description per token: welcome guests to your home with this welcome sign . crafted of iron , this welcome sign is a great way to add a rustic touch to your home .
```
For the query, beam search works as intended. Its top completions of `outdoor bar` are `stool`, then `stool cushion`, each with its probability: short, likely phrases, which is what a search box should offer. This uses the transformer alone; the mix that [chapter 8](#training-a-small-model) chose for autocomplete would need the 4-gram's probability for every candidate at every step, which `lib.decoding` doesn't compute.
For the description, the likeliest text overall is empty. Ending at once has a probability of 0.15, and every longer text multiplies many probabilities below one, so none comes close. Stahlberg and Byrne found the same in translation models: for most sentences the most probable translation was empty, and beam search worked only because it searched badly ([Stahlberg and Byrne, 2019](https://arxiv.org/abs/1908.10090)). Ranking by probability per token removes that bias, and gives a sensible description. It still invents a material, though: this time the sign is `crafted of iron`.
**Working rule:** Use beam search for short completions where the likeliest phrase is the goal, such as search queries, ranked by total probability. For longer text, rank finished texts per token, since total probability favours the shortest, and even then prefer the sampling methods of the [next sections](#sample-with-a-temperature), which write text closer to what people write.
## Sample, with a temperature {#sample-with-a-temperature}
Greedy decoding and beam search use only the top of the distribution. **Sampling**, which [chapter 1](#predicting-the-next-word) used to generate names, uses all of it: draw the next token at random, in proportion to its probability, so a token with probability 0.2 is chosen one time in five. Before sampling, the distribution can be made sharper or flatter with a **temperature**, T: divide every logit by T before the softmax. In symbols, with zᵢ the logit of token i, as in [chapter 5](#predict-from-a-window):
P(token i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
A temperature below 1 sharpens the distribution, giving more of the probability to the likeliest tokens; above 1, it flattens it, giving more to unlikely ones. A temperature near 0 approaches greedy decoding, and a temperature of 1 leaves the model's probabilities as they are. Dividing by T keeps the order of the tokens; it only changes how far apart their probabilities are.
For the store, stop the greedy description after its first few words, and look at the next token's distribution at three temperatures: the three likeliest tokens with their probabilities, how many tokens it takes to cover 90% of the probability, and how much probability falls outside the tokens that cover 90% at temperature 1:
Example `temperature-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import decoding, transformer
_, vocabulary, _ = transformer.load()
start = decoding.prompt("front porch welcome sign wall décor")
so_far = decoding.generate(start, longest=9) # the greedy description's first 9 tokens
logits = decoding.next_logits(start + so_far)
print("text so far:", decoding.text(so_far))
print("tokens in the vocabulary:", f"{len(vocabulary):,}")
cover = np.argsort(-transformer.softmax(logits), kind="stable") # at temperature 1, the likeliest first
cover = cover[:int(np.searchsorted(np.cumsum(transformer.softmax(logits)[cover]), 0.9)) + 1]
for temperature in [0.5, 1.0, 1.5]:
probabilities = transformer.softmax(logits / temperature) # divide the logits, then softmax
top = np.argsort(-probabilities, kind="stable")
covering = int(np.searchsorted(np.cumsum(probabilities[top]), 0.9)) + 1
print(f"temperature {temperature}, top 3:", ", ".join(f'"{vocabulary[i].replace("", "")}" {probabilities[i]:.2f}' for i in top[:3]))
print(f"temperature {temperature}, tokens needed to cover 90% of the probability:", covering)
print(f"temperature {temperature}, probability outside temperature 1.0's 90% cover:",
f"{1 - probabilities[cover].sum():.0%}")
```
Output:
```text
text so far: this welcome sign is a great way to add
tokens in the vocabulary: 4,142
temperature 0.5, top 3: "a" 0.61, "style" 0.20, "some" 0.11
temperature 0.5, tokens needed to cover 90% of the probability: 3
temperature 0.5, probability outside temperature 1.0's 90% cover: 0%
temperature 1.0, top 3: "a" 0.26, "style" 0.15, "some" 0.11
temperature 1.0, tokens needed to cover 90% of the probability: 78
temperature 1.0, probability outside temperature 1.0's 90% cover: 10%
temperature 1.5, top 3: "a" 0.09, "style" 0.06, "some" 0.05
temperature 1.5, tokens needed to cover 90% of the probability: 640
temperature 1.5, probability outside temperature 1.0's 90% cover: 42%
```
After `this welcome sign is a great way to add`, the model's own distribution, at temperature 1, gives its three likeliest tokens "a" 0.26, "style" 0.15, "some" 0.11, and needs 78 tokens to cover 90% of its probability. The remaining tenth is spread over the rest of the vocabulary's 4,142 tokens: a long tail, each token in it unlikely. At temperature 0.5 the three likeliest take "a" 0.61, "style" 0.20, "some" 0.11, and 3 tokens cover 90%. At 1.5 they shrink to "a" 0.09, "style" 0.06, "some" 0.05; covering 90% takes 640 tokens, and the tokens outside temperature 1's 90% now carry 42% of the probability.
That tail is the risk in sampling. Most tokens in it are wrong here, and at temperature 1 a sampler draws from it about once in every ten tokens, more often as the temperature rises.
**Working rule:** Treat temperature as the dial between predictable and varied text. Keep it at or a little below 1 for text that should stay close to what the training text usually says, such as product descriptions. A lower temperature makes text more predictable, not more accurate.
## Sample, and cut the tail {#sample-and-cut-the-tail}
Lowering the temperature shrinks the tail but never removes it. Two methods remove it directly, before sampling. **Top-k** keeps only the k likeliest tokens and renormalizes their probabilities to sum to one ([Fan et al., 2018](https://arxiv.org/abs/1805.04833)). **Top-p**, also called nucleus sampling, keeps the smallest set of likeliest tokens whose probabilities reach at least p ([Holtzman et al., 2020](https://arxiv.org/abs/1904.09751), equation 2). Both are options of `lib.decoding.truncate`, whose core is three lines: sort the tokens, add up their probabilities in that order, and find where the sum reaches p:
```python
order = np.argsort(-probabilities, kind="stable")
...
keep = int(np.searchsorted(np.cumsum(probabilities[order]), top_p)) + 1
```
Top-p adapts to the model's confidence, where top-k doesn't. At the point in the [previous section's example](#sample-with-a-temperature) where the model was unsure of the next word, it took 78 tokens to cover 90% of the probability, so top-p 0.9 would keep that many, while top-k 20 would keep only 20. After a word that the model predicts with near certainty, top-p keeps one or two tokens and top-k still keeps 20, most of them wrong. Holtzman and colleagues argued for top-p on exactly this ground: a fixed k is "sub-optimal across varying contexts".
For the store, write 40 tokens of the welcome sign's description four ways, with the same random seed: sampling at temperature 1, sampling at 0.7, top-k with k = 20, and top-p with p = 0.9. Then replay the temperature-1 sample, and list the tokens it drew from outside each step's 90% nucleus, the tokens top-p 0.9 would have cut:
Example `sample-descriptions` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import decoding, transformer
_, vocabulary, _ = transformer.load()
start, longest, seed = decoding.prompt("front porch welcome sign wall décor"), 40, 2026
print("longest description, in tokens:", longest)
print("random seed:", seed)
for label, settings in [("sampling, temperature 1.0", {}), ("sampling, temperature 0.7", {"temperature": 0.7}),
("top-k 20", {"top_k": 20}), ("top-p 0.9", {"top_p": 0.9})]:
ids = decoding.generate(start, rng=np.random.default_rng(seed), longest=longest, **settings)
print(f"{label}:", decoding.text(ids))
rng, written, tail = np.random.default_rng(seed), [], [] # replay temperature 1.0's draws, step by step
for _ in range(longest):
probabilities = transformer.softmax(decoding.next_logits(start + written))
token = int(rng.choice(len(probabilities), p=decoding.truncate(probabilities)))
order = np.argsort(-probabilities, kind="stable")
before = probabilities[order[:int(np.where(order == token)[0][0])]].sum() # the probability ranked above it
if before >= 0.9:
tail.append(vocabulary[token].replace("", ""))
written.append(token)
if vocabulary[token] == transformer.END:
break
print("temperature 1.0, tokens drawn from outside the 90% nucleus:", f"{len(tail)} of {len(written)}")
print("temperature 1.0, those tokens:", ", ".join(tail))
```
Output:
```text
longest description, in tokens: 40
random seed: 2026
sampling, temperature 1.0: enhance your home even glywheel design with this welcome welcome sign sign . this piece is ideal for both indoor and inviting . made in the usa , mini spirits decorate .
sampling, temperature 0.7: the sign is a great way to add some extra whimsy to your home and get colorful ! made of high-quality metal , this sign comes with a hanging dustroke hanger to help you gli
top-k 20: if you have a new product with a personalized sign wall décor that will help you entertain . look no further like a home decor , this sign is constructed to make life more wonderful . this welcome sign
top-p 0.9: express your guests with this cast iron sign from the nature of the world . it features a meditating podishade design . the hose and symbol in this metal piece will be
temperature 1.0, tokens drawn from outside the 90% nucleus: 5 of 36
temperature 1.0, those tokens: inviting, in, mini, spir, decorate
```
Sampling at temperature 1 wanders: it writes `welcome welcome sign sign`, invents a word, `glywheel`, and claims the sign is `made in the usa`. Only some of that comes from the tail: 5 of 36 tokens did, inviting, in, mini, spir, decorate. The doubled words and the made-up word were built from tokens inside the nucleus, which top-p would have kept. Cutting the tail removes the rarest mistakes, not all of them. At 0.7 the text holds together better, though it now says the sign is `made of high-quality metal`, where greedy decoding said wood. Top-k and top-p read more like the catalog than plain sampling, and still invent: top-p's sign is `cast iron`, with a `podishade` design.
Across this chapter, the welcome sign has been given a material six times, by greedy decoding, beam search, three of these samples and the [penalized greedy text](#penalize-repetition) below, and nothing in its name says what it's made of. The model writes what descriptions of welcome signs usually say, and they usually state a material. A decoding method changes which plausible material the model picks; none of them can make it pick the true one.
**Working rule:** For varied text, cut the tail with top-p around 0.9 or top-k around 20, with a temperature at or a little below 1, and fix the random seed when you need to reproduce a result. Don't expect any setting to make the text true: decoding chooses among what the model finds plausible, and checking facts is a separate job.
## Measure what each setting writes {#measure-each-setting}
Single samples show the flavour of each method; measurements show the trade-offs. Several properties of generated text can be measured without a person reading it:
- **repetition:** the share of four-token phrases that repeat an earlier one in the same description
- **the model's own [perplexity](#score-with-perplexity) of the text:** how surprising the model finds what it wrote, pooled over every token, where low means predictable
- **whether the description ends** within the limit, as real ones do
- **whether it uses a whole word of the product's name**, as a description of that product should
- **its average length** in tokens
For the store, draw 20 validation products at random from those with a description, write a description for each with every method, and compare them with the products' real descriptions, cut at the same 60-token limit:
Example `measure-decoding` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import decoding
from lib.products import product_texts
longest, sample, seed = 60, 20, 2026
described = [(ids, sep) for ids, sep in product_texts("validation") if sep < len(ids) - 2]
products = [described[i] for i in np.random.default_rng(seed).choice(len(described), sample, replace=False)]
print("products, drawn at random from the validation products with a description:", sample)
print("longest description, in tokens:", longest)
settings = [("real descriptions", None), ("greedy", {}), ("greedy, repetition penalty 1.3", {"penalty": 1.3}),
("sampling, temperature 1.0", {"sample": True}), ("sampling, temperature 0.7", {"sample": True, "temperature": 0.7}),
("top-k 20", {"sample": True, "top_k": 20}), ("top-p 0.9", {"sample": True, "top_p": 0.9})]
for label, chosen in settings:
rows = []
for number, (ids, sep) in enumerate(products):
start = ids[:sep + 1]
if chosen is None:
written = ids[sep + 1:][:longest]
else:
options = {option: value for option, value in chosen.items() if option != "sample"}
rng = np.random.default_rng(seed + number) if chosen.get("sample") else None
written = decoding.generate(start, rng=rng, longest=longest, **options)
rows.append(decoding.measures(start, written))
repeated, log_p, ended, uses_name, length = zip(*rows)
print(f"{label}, repeated 4-token phrases:", f"{np.mean(repeated):.0%}")
print(f"{label}, the model's perplexity of the text:", f"{np.exp(-np.concatenate(log_p).mean()):.1f}")
print(f"{label}, ended within {longest} tokens:", f"{np.mean(ended):.0%}")
print(f"{label}, uses a whole word of the name:", f"{np.mean(uses_name):.0%}")
print(f"{label}, average length in tokens:", f"{np.mean(length):.0f}")
```
Output:
```text
products, drawn at random from the validation products with a description: 20
longest description, in tokens: 60
real descriptions, repeated 4-token phrases: 0%
real descriptions, the model's perplexity of the text: 30.5
real descriptions, ended within 60 tokens: 35%
real descriptions, uses a whole word of the name: 95%
real descriptions, average length in tokens: 53
greedy, repeated 4-token phrases: 11%
greedy, the model's perplexity of the text: 2.7
greedy, ended within 60 tokens: 25%
greedy, uses a whole word of the name: 80%
greedy, average length in tokens: 52
greedy, repetition penalty 1.3, repeated 4-token phrases: 0%
greedy, repetition penalty 1.3, the model's perplexity of the text: 3.9
greedy, repetition penalty 1.3, ended within 60 tokens: 70%
greedy, repetition penalty 1.3, uses a whole word of the name: 80%
greedy, repetition penalty 1.3, average length in tokens: 41
sampling, temperature 1.0, repeated 4-token phrases: 0%
sampling, temperature 1.0, the model's perplexity of the text: 22.6
sampling, temperature 1.0, ended within 60 tokens: 50%
sampling, temperature 1.0, uses a whole word of the name: 50%
sampling, temperature 1.0, average length in tokens: 45
sampling, temperature 0.7, repeated 4-token phrases: 0%
sampling, temperature 0.7, the model's perplexity of the text: 7.2
sampling, temperature 0.7, ended within 60 tokens: 45%
sampling, temperature 0.7, uses a whole word of the name: 65%
sampling, temperature 0.7, average length in tokens: 45
top-k 20, repeated 4-token phrases: 0%
top-k 20, the model's perplexity of the text: 7.6
top-k 20, ended within 60 tokens: 50%
top-k 20, uses a whole word of the name: 80%
top-k 20, average length in tokens: 40
top-p 0.9, repeated 4-token phrases: 0%
top-p 0.9, the model's perplexity of the text: 10.7
top-p 0.9, ended within 60 tokens: 30%
top-p 0.9, uses a whole word of the name: 60%
top-p 0.9, average length in tokens: 47
```
The real descriptions repeat nothing (0%), and the model finds them fairly surprising: its perplexity of them is 30.5, higher than [chapter 8](#training-a-small-model)'s figure for descriptions because these are a different sample and only their first 60 tokens. Greedy text is the opposite: 11% of its phrases repeat, and its perplexity is 2.7, far more predictable than anything a person wrote. [Holtzman and colleagues](https://arxiv.org/abs/1904.09751) made the same observation at scale: human text isn't the text a model finds most probable.
No setting matches the real descriptions on every measure. Sampling at temperature 1 comes closest on perplexity, 22.6, but only 50% of its descriptions use a word of the product's name, against 95% of the real ones: it drifts off the product. Top-p 0.9 ends as often as the real text, 30% against 35%, with a perplexity of 10.7, but stays on the product in only 60%. Top-k 20 stays on the product most among the samplers, 80%, with no repetition, a perplexity of 7.6 and descriptions of 40 tokens on average. Twenty products is a small sample, so read these as directions, not exact rates.
What the measures can't say is whether a description is true. A description can be fluent, varied, on topic and wrong, and [the previous section's](#sample-and-cut-the-tail) were, about the sign's material.
**Working rule:** Compare decoding settings on measures like these, against real text of the same kind, before reading samples. Decide which measure matters most for the job: for product descriptions, staying on the product and not repeating come first. Then read a sample of the outputs anyway, for what no measure catches, and check every factual claim against the product's data.
## Penalize repetition {#penalize-repetition}
[Greedy decoding's](#choose-the-likeliest-token) loops can also be fixed without sampling. Keskar and colleagues proposed a **repetition penalty**: divide the logit of every token already generated by a factor θ above 1, so it becomes less likely ([Keskar et al., 2019](https://arxiv.org/abs/1909.05858), section 4.1). Dividing a negative logit would raise it instead, so implementations multiply negative logits by θ, as `lib.decoding.penalize` does, so the token becomes less likely either way. Keskar and colleagues found a factor around 1.2 balanced avoiding repetition against writing sensibly.
For the store, go back to the point where the [greedy description](#choose-the-likeliest-token) was about to write `distressed` a second time. Apply the penalty by hand to the next token's logits, check it against `lib.decoding.penalize`, and compare the probabilities before and after. Then write the whole description greedily with the penalty:
Example `repetition-penalty-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import decoding, transformer
_, vocabulary, _ = transformer.load()
start, factor = decoding.prompt("front porch welcome sign wall décor"), 1.3
written = decoding.generate(start) # greedy, as in greedy-description
distressed = vocabulary.index("distressed")
position = [i for i, token in enumerate(written) if token == distressed][1] # where greedy writes it a second time
before = written[:position]
print("text so far:", decoding.text(before))
print("penalty factor:", factor)
logits = decoding.next_logits(start + before)
seen = np.unique(before) # every token the description has used
penalized = logits.copy()
penalized[seen] = np.where(logits[seen] > 0, logits[seen] / factor, logits[seen] * factor) # less likely either way
print("same as lib.decoding.penalize:", bool(np.allclose(penalized, decoding.penalize(logits, before, factor))))
for label, adjusted in [("without a penalty", logits), ("with the penalty", penalized)]:
probabilities = transformer.softmax(adjusted)
top = np.argsort(-probabilities, kind="stable")[:3]
print(f"next token, {label}:", ", ".join(f"{vocabulary[i].replace('', '')} {probabilities[i]:.2f}" for i in top))
print(f"P(distressed), {label}:", f"{probabilities[distressed]:.2f}")
print("greedy with the penalty:", decoding.text(decoding.generate(start, penalty=factor)))
```
Output:
```text
text so far: this welcome sign is a great way to add a touch of rustic charm to your home . it is made of natural wood and features a distressed finish with a
penalty factor: 1.3
same as lib.decoding.penalize: True
next token, without a penalty: distressed 0.20, black 0.03, weathered 0.03
P(distressed), without a penalty: 0.20
next token, with the penalty: black 0.04, weathered 0.03, distressed 0.03
P(distressed), with the penalty: 0.03
greedy with the penalty: this welcome sign is a great way to add style and functionality to any room . the worn look of these beautiful signs are made from natural coir , so it can be used indoors or outdoors .
```
With a factor of 1.3, the hand calculation matches the library's (True). Without the penalty, `distressed` has probability 0.20 and wins; with it, 0.03, and another token wins. The penalized greedy description doesn't loop. It does still invent: this time the sign is made of `natural coir`, a fibre from coconut husks.
The penalty has a side effect the [measurements](#measure-each-setting) show. It penalizes every token already used, including common words such as `the` and `and` that a description needs many times, but never the end marker, which hasn't been used yet. So the end becomes relatively likelier, and descriptions end sooner: 70% of the penalized greedy descriptions ended within the limit, against 25% without the penalty, with an average length of 41 tokens against 52. Its repetition fell to 0%.
**Working rule:** Add a repetition penalty around 1.2 to 1.3 when you decode greedily or at a low temperature, and check what it does to length, since it shortens text. As a rule of thumb, if descriptions come out clipped, leave common function words and punctuation out of the penalty.
## Forbid an empty description {#forbid-an-empty-description}
The [greedy example](#choose-the-likeliest-token) found the model ready to write nothing for a bulb, and [beam search](#search-for-the-likeliest-text) found nothing to be the likeliest description for a sign. Both come from the catalog: many products have no description, so the model learned that ending at once is common. For a store that wants to fill exactly those gaps, the model has to be stopped from reproducing them. The simple fix is a **minimum length**: forbid the end marker, by setting its logit to minus infinity, until the description has some number of tokens. `lib.decoding.generate` takes it as `shortest`.
For the store, take the bulb, `5 light light bulb`, and decode greedily without a minimum and with one of 10 tokens:
Example `minimum-length` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
from lib import decoding, transformer
end = transformer.load()[1].index(transformer.END)
start, shortest = decoding.prompt("5 light light bulb"), 10
print("P(the description is empty):", f"{transformer.softmax(decoding.next_logits(start))[end]:.2f}")
print("greedy description:", decoding.text(decoding.generate(start)) or "(empty)")
print("minimum length, in tokens:", shortest)
print("greedy description, with the minimum:", decoding.text(decoding.generate(start, shortest=shortest)))
```
Output:
```text
P(the description is empty): 0.28
greedy description: (empty)
minimum length, in tokens: 10
greedy description, with the minimum: this light is a great light for a light , a light , and more light . it is made of metal and has a long , high gloss finish that will make it a perfect accent piece for your home . this light is dimmable , compatible with most standard bulbs , which are not included .
```
The model gives the empty description a probability of 0.28, and greedy decoding takes it: the description comes out empty. With the minimum, it writes one, and two failures from earlier in this chapter appear in it. It repeats itself, `a light , a light , and more light`, and it states things the name doesn't: that the bulb is `made of metal`, `dimmable`, and `compatible with most standard bulbs`. The minimum stopped the gap; it can't supply the facts.
**Working rule:** Set a minimum length whenever the training text includes empty or very short examples of what you're generating. Combine it with the other settings in this chapter, not instead of them, and expect the text it forces out to need the same checking as any other.
## What goes wrong in production {#decoding-in-production}
Decoding is where a model's text meets its readers, and most of its failures are quiet:
- **Fluent text states made-up facts.** Six descriptions in this chapter gave the welcome sign a material, and they disagreed. Never publish generated product text without checking each factual claim against the product's data; [chapter 11](#from-predicting-to-following) measures how often a model invents.
- **Gaps in the training text become outputs.** The model predicts an empty description for a bulb, because the catalog has many, and beam search finds the empty description likeliest of all. Forbid the end marker until a [minimum length](#forbid-an-empty-description), or the model will reproduce the gaps you're trying to fill.
- **Greedy text loops.** Use sampling with a cut tail, or a repetition penalty, for anything longer than a phrase, and measure repetition on a sample of outputs.
- **Sampled text isn't reproducible by default.** Fix and record the random seed with every generated text, so a complaint about one description can be traced and repeated.
- **Settings drift with the model.** A retrained model has a different distribution, so a temperature or top-k tuned on the last model can write differently with the next. Rerun the measurements of this chapter after every retrain, before the new model writes anything a shopper sees.
- **Each token costs a forward pass.** `lib.decoding` reruns the model over the whole text for every new token, which is simple and fine for 60 tokens. Serving systems keep every layer's keys and values from earlier positions, so each new token's matrix multiplications cover one position, though attention still reads the cached keys and values of every earlier one ([Pope et al., 2022](https://arxiv.org/abs/2211.05102)).
## What this chapter decided {#decoding-decisions}
- **Autocomplete uses the likeliest tokens.** Search-box suggestions show the top few tokens of the mix chosen in [chapter 8](#training-a-small-model), and beam search, ranked by total probability, can extend them to short phrases.
- **Descriptions use top-k 20 sampling,** the setting that stayed on the product most often without repeating, on the measures above, with a fixed, recorded seed and a minimum length so they can't come out empty. With only twenty products behind it, the choice is rechecked on a larger sample before any description reaches a shopper.
- **Beam search ranks longer texts per token,** because total probability favours the empty text. Short completions of search queries are ranked by total probability, where the likeliest phrase is what a shopper wants.
- **Decoding settings are chosen by measurement against real text:** repetition, the model's own perplexity, whether descriptions end, whether they use the name, and their length, over a random set of products, before samples are read, and again after every retrain.
- **Generated descriptions are drafts, never a source of product facts.** A person or a checking step corrects every claim that the name and the catalog can't support: every method here invented a material for a product whose name gives none.
# Inside an open model
By the end of this chapter you can read an open language model with what this book has built: count where its parameters go, work out what it costs to run, score text with it fairly against a model with a different tokenizer, find the attention patterns of chapter 6 in its heads, and test what it copies. You'll also see what its size buys on the store's text, and what it doesn't.
[Chapter 9](#choosing-the-next-token) ended with descriptions from the store's own model that need checking before anyone reads them. The store's model is the small transformer that chapters 6 to 9 built, trained and used, on one catalog: the names and descriptions of the online furniture and home-goods store this book follows. **SmolLM2-360M**, an open model released by Hugging Face under the Apache 2.0 licence, is built from the same parts, as [chapter 7](#the-transformer-block) showed. But it has more than two hundred times as many parameters, and it was trained on 4 trillion tokens ([model card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/blob/f8027fd0eaeea54caa13c31d31b9fdc459c38b49/README.md)) of web text, code, mathematics and synthetic textbooks ([Allal et al., 2025](https://arxiv.org/abs/2502.02737), section 6). For the store, the question is whether a free, downloadable model trained on so much more text could do its own model's jobs: suggesting the next word in the search box, and predicting its descriptions. [Chapter 2](#read-an-open-tokenizer) read SmolLM2's tokenizer; this chapter reads the model.
Running SmolLM2 needs PyTorch and a download of about 0.7 gigabytes, its weights stored as 16-bit numbers, so, like every result in this book that depends on the machine, it ran once. The script `recordings/open_model.py` loaded the **base** model, the one trained only to predict the next token, as the store's model was, at a pinned revision, a fixed version of its repository. It ran the model on the store's text and saved what this chapter reads: every weight's shape, its scores on the store's validation names and descriptions, its heads' attention habits, [chapter 6](#attention)'s copying test and [chapter 8](#training-a-small-model)'s search-box prefixes. A second script, `recordings/open_model_family.py`, downloaded only the configuration files of SmolLM2's three sizes, also at pinned revisions. The examples read both recordings, in `examples/data/recorded/`, through `lib.open_model`.
## Count the parameters at scale {#parameters-at-scale}
The parts are familiar, so the first question is where the parameters sit. [Chapter 7](#count-the-parameters) counted the store's model by part: 1,334,144 parameters, 40% of them in the embedding table it shares with its output.
For the store, count SmolLM2's recorded weights by part, the same way:
Example `open-model-parameters` (Python 3.12.15), passing in Edition 2026.10.3:
```python
import math
from lib import open_model
shapes = {name: shape for name, shape in open_model.read("parameters").items()
if name != "lm_head.weight"} # the output layer is tied to the embeddings: the same weights, counted once
parts = {"embedding (shared with the output)": ["embed_tokens"], "attention": ["q_proj", "k_proj", "v_proj", "o_proj"],
"feed-forward": ["gate_proj", "up_proj", "down_proj"], "normalization": ["layernorm", "model.norm"]}
total = sum(math.prod(shape) for shape in shapes.values())
for part, keys in parts.items():
count = sum(math.prod(shape) for name, shape in shapes.items() if any(key in name for key in keys))
print(f"{part}:", f"{count:,}", f"({count / total:.0%})")
print("total:", f"{total:,}")
config = open_model.read("manifest")["config"]
print("layers:", config["num_hidden_layers"])
print("width:", config["hidden_size"])
print("query heads:", config["num_attention_heads"])
print("key and value heads:", config["num_key_value_heads"])
print("feed-forward width:", config["intermediate_size"])
print("vocabulary:", f"{config['vocab_size']:,}")
print("one layer's query weights:", shapes["model.layers.0.self_attn.q_proj.weight"])
print("one layer's key weights:", shapes["model.layers.0.self_attn.k_proj.weight"])
```
Output:
```text
embedding (shared with the output): 47,185,920 (13%)
attention: 78,643,200 (22%)
feed-forward: 235,929,600 (65%)
normalization: 62,400 (0%)
total: 361,821,120
layers: 32
width: 960
query heads: 15
key and value heads: 5
feed-forward width: 2560
vocabulary: 49,152
one layer's query weights: [960, 960]
one layer's key weights: [320, 960]
```
SmolLM2 has 361,821,120 parameters, in 32 layers 960 numbers wide, with a feed-forward width of 2560 and a vocabulary of 49,152 tokens. The balance has shifted from the store's model. The embedding table, still shared with the output, holds 47,185,920 (13%); the feed-forward layers hold 235,929,600 (65%); attention holds 78,643,200 (22%); and the normalization gains almost nothing. The blocks grow with the number of layers times the square of the width, and the embedding table, for a fixed vocabulary, only with the width, so as a model grows, the blocks take over.
Attention holds fewer parameters than 15 full heads would, because of **grouped-query attention**. Each layer has 15 query heads but only 5 heads of keys and values, each shared by 3 query heads ([Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)). The weights show it: one layer's query weights have the shape [960, 960], and its key weights [320, 960], a third as many. Per layer, with the head size being the width divided by the number of query heads:
attention parameters = 2 × width² + 2 × width × key and value heads × head size
for the query and output weights, then the key and value weights. During generation, a model keeps every layer's keys and values for every earlier token, so sharing them shrinks that store, as a [later section](#cost-to-run) measures; Ainslie and colleagues found it costs little in quality.
**Working rule:** Count an open model's parameters by part from its weights before you use it, as here, and read the shapes for its design choices: tied embeddings, grouped keys and values, a gated feed-forward layer. Expect the blocks, not the embeddings, to hold most of a large model's parameters.
## Compare a family's sizes {#grow-the-same-design}
SmolLM2 comes in three sizes, each built from the same parts, and their configurations show how a model's makers spend more parameters. Each configuration file gives the layers, the width, the heads and the feed-forward width, and those are enough to count every parameter, part by part, with the arithmetic of [chapter 7](#count-the-parameters) and the formula above. `lib.open_model.parameters` does it.
For the store, count all three sizes from their recorded configurations, and check the count for the 360M model against the weights the first recording saved:
Example `open-model-family` (Python 3.12.15), passing in Edition 2026.10.3:
```python
import math
from lib import open_model
for name in open_model.SIZES:
config = open_model.config(name)
parts = open_model.parameters(config)
print(f"{name}, layers:", config["num_hidden_layers"])
print(f"{name}, width:", f"{config['hidden_size']:,}")
print(f"{name}, query heads:", config["num_attention_heads"])
print(f"{name}, query heads per key and value head:", config["num_attention_heads"] // config["num_key_value_heads"])
print(f"{name}, feed-forward width:", f"{config['intermediate_size']:,}")
print(f"{name}, RoPE base:", f"{config['rope_theta']:,}")
print(f"{name}, parameters:", f"{parts['total']:,}")
print(f"{name}, embedding's share:", f"{parts['embedding'] / parts['total']:.0%}")
recorded = sum(math.prod(shape) for key, shape in open_model.read("parameters").items() if key != "lm_head.weight")
print("SmolLM2-360M, parameters in the recorded weights:", f"{recorded:,}")
```
Output:
```text
SmolLM2-135M, layers: 30
SmolLM2-135M, width: 576
SmolLM2-135M, query heads: 9
SmolLM2-135M, query heads per key and value head: 3
SmolLM2-135M, feed-forward width: 1,536
SmolLM2-135M, RoPE base: 100,000
SmolLM2-135M, parameters: 134,515,008
SmolLM2-135M, embedding's share: 21%
SmolLM2-360M, layers: 32
SmolLM2-360M, width: 960
SmolLM2-360M, query heads: 15
SmolLM2-360M, query heads per key and value head: 3
SmolLM2-360M, feed-forward width: 2,560
SmolLM2-360M, RoPE base: 100,000
SmolLM2-360M, parameters: 361,821,120
SmolLM2-360M, embedding's share: 13%
SmolLM2-1.7B, layers: 24
SmolLM2-1.7B, width: 2,048
SmolLM2-1.7B, query heads: 32
SmolLM2-1.7B, query heads per key and value head: 1
SmolLM2-1.7B, feed-forward width: 8,192
SmolLM2-1.7B, RoPE base: 130,000
SmolLM2-1.7B, parameters: 1,711,376,384
SmolLM2-1.7B, embedding's share: 6%
SmolLM2-360M, parameters in the recorded weights: 361,821,120
```
The counts from the configurations match the weights exactly: 361,821,120 counted, 361,821,120 recorded. The three sizes have 134,515,008, 361,821,120 and 1,711,376,384 parameters. The smallest has 30 layers, 576 wide, with 9 query heads; the middle one 32 layers, 960 wide, with 15; and the largest 24 layers, 2,048 wide, with 32, fewer layers than either and much wider. Width, not depth, is where most of the largest model's extra parameters went.
The embedding table's share falls as the models grow, from 21% to 13% to 6%, because all three share one vocabulary while their blocks grow. And the largest differs in more than size. It doesn't group keys and values: it has 1 query head per key and value head, where the two smaller ones share each among 3. Its feed-forward layer is 8,192 wide, four times its width, where the 360M model's is 2,560, 8/3 of its width. And its rotary positions turn more slowly, with a base of 130,000 against 100,000.
**Working rule:** Count a model's parameters from its configuration, and check the count against its weights, as here; a mismatch means you've misread a detail. Compare a family's sizes to see how its makers scaled the design, and don't assume a larger model differs from a smaller one only in size.
## Count what it costs to run {#cost-to-run}
Parameters set what a model can learn; they also set what it costs to run. Its weights have to sit in memory, at two bytes each in 16-bit formats, the form SmolLM2's weights are stored in ([model card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/blob/f8027fd0eaeea54caa13c31d31b9fdc459c38b49/README.md)). And during generation, as [chapter 9](#decoding-in-production) noted, every layer keeps a key and a value for every earlier token, so each new token doesn't redo the work for the text before it. That **key and value cache** grows with the length of the text:
cache per token, in bytes = layers × 2 × key and value heads × head size × 2
counting a key and a value per layer, and two bytes per number.
For the store, compute both for the three sizes: the weights, the cache per token, and the cache for the longest text the models are set up for, 8,192 tokens, with and without grouped keys and values:
Example `open-model-memory` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from lib import open_model
longest = open_model.config("SmolLM2-360M")["max_position_embeddings"] # the longest text the models are set up for
print("longest text, in tokens:", f"{longest:,}")
for name in open_model.SIZES:
config = open_model.config(name)
head_size = config["hidden_size"] // config["num_attention_heads"]
weights = open_model.parameters(config)["total"] * 2 # 16-bit numbers: 2 bytes each
cache = config["num_hidden_layers"] * 2 * config["num_key_value_heads"] * head_size * 2 # a key and a value per layer
ungrouped = config["num_hidden_layers"] * 2 * config["num_attention_heads"] * head_size * 2
print(f"{name}, weights in 16-bit numbers, in GB:", f"{weights / 1e9:.1f}")
print(f"{name}, key and value cache per token, in KB:", f"{cache / 1e3:.0f}")
print(f"{name}, cache for the longest text, in GB:", f"{cache * longest / 1e9:.2f}")
print(f"{name}, cache for the longest text without grouping, in GB:", f"{ungrouped * longest / 1e9:.2f}")
```
Output:
```text
longest text, in tokens: 8,192
SmolLM2-135M, weights in 16-bit numbers, in GB: 0.3
SmolLM2-135M, key and value cache per token, in KB: 23
SmolLM2-135M, cache for the longest text, in GB: 0.19
SmolLM2-135M, cache for the longest text without grouping, in GB: 0.57
SmolLM2-360M, weights in 16-bit numbers, in GB: 0.7
SmolLM2-360M, key and value cache per token, in KB: 41
SmolLM2-360M, cache for the longest text, in GB: 0.34
SmolLM2-360M, cache for the longest text without grouping, in GB: 1.01
SmolLM2-1.7B, weights in 16-bit numbers, in GB: 3.4
SmolLM2-1.7B, key and value cache per token, in KB: 197
SmolLM2-1.7B, cache for the longest text, in GB: 1.61
SmolLM2-1.7B, cache for the longest text without grouping, in GB: 1.61
```
The weights take 0.3, 0.7 and 3.4 gigabytes. The 360M model's cache takes 41 kilobytes per token, so one text of the longest length needs 0.34 gigabytes, and would need 1.01 without grouping: grouping keys and values cut it to a third. The 1.7B model, which doesn't group them, needs 197 kilobytes per token, and 1.61 gigabytes for one long text, about half its weights' size. A service writing many descriptions at once keeps one cache per text, so with long texts, the caches can outgrow the weights.
**Working rule:** Budget a model's memory as its weights plus one cache per text in progress, computed from its configuration, as here. Prefer models that group keys and values when you serve long texts or many at once, and cap the text length you allow to what you've budgeted for.
## Score the store's text with a general model {#score-the-store-text}
A model with two hundred times the parameters, trained on more than a million times as many tokens as the store's catalog holds, might be expected to predict the store's text better than the store's own small model. Measuring that takes care, because the two models use different tokenizers: SmolLM2's vocabulary of 49,152 tokens splits text differently from the store's, so their [perplexities per token](#score-with-perplexity) count different things. What they share is the text. So divide each model's total log-probability of the same text by the number of words in it: a **perplexity per word**. In symbols, for a text of W words whose tokens the model gives probabilities p₁ to pₙ, however many tokens that takes:
perplexity per word = exp(−(log p₁ + … + log pₙ) / W)
Dividing each model's total by the same number of words puts both on one scale, however many tokens each spent.
Each model scores the same events. For a name, the store's model starts from `` and SmolLM2 from its own start token, `<|endoftext|>`, and both predict the name's words and then its end: `` for the store's model, a newline for SmolLM2. For a description, both are given the name and predict the description's words and its end. The names are the first 500 validation names, and the descriptions are [chapter 8's](#compare-on-descriptions) 446 products. The recording also scored each name after a label, `Product name: `, to test whether telling SmolLM2 what it's reading helps.
Before you run it, predict which model scores the store's names better per word, and by how much. Then compute the store's model's scores in NumPy, read SmolLM2's from the recording, and compare, per token and per word:
Example `open-model-on-the-store` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import math
from lib import bpe, open_model, transformer
from lib.products import product_texts
from lib.store import load_products
_, vocabulary, _ = transformer.load()
index = {token: i for i, token in enumerate(vocabulary)}
scores, tokenizer = open_model.read("scores"), bpe.store_tokenizer()
per = lambda log_p, count: f"{math.exp(-log_p / count):,.1f}"
# Names: each name's tokens and its end ( for the store's model, a newline for SmolLM2), from the start token.
names = list(load_products("validation")["name"])[:len(scores["names"])]
encoded = [[index[transformer.START]] + [index[t] for t in tokenizer.encode(name)] + [index[transformer.SEP]] for name in names]
store = sum(transformer.total_log_probability(ids, 1) for ids in encoded)
smollm2 = sum(row["log_p"] for row in scores["names"])
words = sum(row["words"] for row in scores["names"])
print("validation names scored:", len(names))
print("names, per-token perplexity, the store's model:", per(store, sum(len(ids) - 1 for ids in encoded)))
print("names, per-token perplexity, SmolLM2-360M:", per(smollm2, sum(row["tokens"] for row in scores["names"])))
print("names, per-word perplexity, the store's model:", per(store, words))
print("names, per-word perplexity, SmolLM2-360M:", per(smollm2, words))
print("names after a label, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["labelled names"]), words))
# Descriptions: chapter 8's products, each description's tokens and its end, given the name.
end = index[transformer.END]
store = sum(transformer.total_log_probability(ids, sep + 1) for ids, sep in product_texts("validation", 500)
if ids[-1] == end and sep < len(ids) - 2)
words = sum(row["words"] for row in scores["descriptions"])
print("descriptions scored:", len(scores["descriptions"]))
print("descriptions, per-word perplexity, the store's model:", per(store, words))
print("descriptions, per-word perplexity, SmolLM2-360M:", per(sum(r["log_p"] for r in scores["descriptions"]), words))
```
Output:
```text
validation names scored: 500
names, per-token perplexity, the store's model: 25.8
names, per-token perplexity, SmolLM2-360M: 564.8
names, per-word perplexity, the store's model: 199.3
names, per-word perplexity, SmolLM2-360M: 54,352.8
names after a label, per-word perplexity, SmolLM2-360M: 67,101.5
descriptions scored: 446
descriptions, per-word perplexity, the store's model: 37.0
descriptions, per-word perplexity, SmolLM2-360M: 61.2
```
Per token, on these names, the store's model scores 25.8 and SmolLM2 564.8. Per word, the gap is far wider: 199.3 against 54,352.8, because SmolLM2 spends more tokens on each of the store's words, and per token hides it. On the descriptions, per word, the store's model scores 37.0, and SmolLM2 61.2. The small model wins on both, by a wide margin on names.
Telling SmolLM2 it's reading a product name doesn't close the gap. After the label, it scores 67,101.5 per word, no better than without. So the gap isn't that SmolLM2 doesn't know a name is coming; it's the names themselves. Our judgement, untested here, is that two features of the store's text account for much of it: many names begin with a product line's made-up name, which the store's model has seen across its catalog, and the store's text is lowercased with spaces around its punctuation, unlike most of what SmolLM2 learned from.
The lesson isn't that small models are better. A model predicts best the kind of text it learned from: SmolLM2 knows far more English than the store's model, and little of the store's habits. Teaching it those habits takes further training on the store's text, which [chapter 11](#adapt-a-model) describes.
**Working rule:** Compare models with different tokenizers per word, or per character, on exactly the same text and events, never per token. Before choosing a large general model over a small one trained on your own text, measure both on your text, as here; size doesn't make up for text unlike anything the model learned from.
## Find the same heads at scale {#same-heads-at-scale}
[Chapter 6](#attend-to-the-previous-token) measured the store's model's heads by their habits, against an **even spread**, which puts 1 / (i + 1) of a position's weight on each of the i + 1 tokens position i can see. The same measures work on any model, and SmolLM2 has 480 heads to measure.
For the store, read the recorded averages over the first 50 of the 446 products scored above, each text a name, a newline and its description: each head's average weight on the previous token and on the first token, and an even spread's weight on each, averaged the same way:
Example `open-model-heads` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · tokenizers 0.23.3), passing in Edition 2026.10.3:
```python
import numpy as np
from tokenizers import Tokenizer
from lib import open_model, transformer
from lib.products import product_texts
from lib.store import load_products
heads = open_model.read("heads")
previous, first = np.array(heads["previous"]), np.array(heads["first"]) # (layers, heads)
print("texts averaged:", heads["texts"])
print("heads in all:", previous.size)
print("heads with more than half their weight on the previous token:", int((previous > 0.5).sum()))
print("the most weight a head puts on the previous token:", f"{previous.max():.2f}")
print("average weight on the first token, layers 1 to 4:", f"{first[:4].mean():.2f}")
print("average weight on the first token, layers 5 to 32:", f"{first[4:].mean():.2f}")
print("heads with more than half their weight on the first token:", int((first > 0.5).sum()))
# An even spread, averaged the same way: each position i shares its weight equally among the i + 1 tokens it sees.
smollm2 = Tokenizer.from_file("data/smollm2-tokenizer/tokenizer.json")
end = transformer.load()[1].index(transformer.END)
rows = load_products("validation").head(500)
texts = [f"{name}\n{description}\n" for (ids, sep), name, description in
zip(product_texts("validation", 500), rows["name"], rows["description"]) if ids[-1] == end and sep < len(ids) - 2]
lengths = [len(smollm2.encode(text, add_special_tokens=False).ids) + 1 for text in texts[:heads["texts"]]]
print("an even spread's weight on the previous token, and on the first:",
f"{np.mean([np.mean(1 / np.arange(2, n + 1)) for n in lengths]):.2f}")
```
Output:
```text
texts averaged: 50
heads in all: 480
heads with more than half their weight on the previous token: 3
the most weight a head puts on the previous token: 0.79
average weight on the first token, layers 1 to 4: 0.13
average weight on the first token, layers 5 to 32: 0.61
heads with more than half their weight on the first token: 286
an even spread's weight on the previous token, and on the first: 0.05
```
A few heads specialize sharply. 3 put more than half their weight on the previous token, the most of them 0.79, against an even spread's 0.05; the store's model's most previous-token head put 0.35, measured the same way. With hundreds of heads, a model can afford to give a few of them one narrow job.
The first token's weight is the larger surprise. In the first four layers, heads put 0.13 of their weight on it on average; from layer 5 on, 0.61, against an even spread's 0.05, and 286 heads put more than half there. That's the **attention sink** that Xiao and colleagues found in large models ([Xiao et al., 2024](https://arxiv.org/abs/2309.17453)). They explain it by softmax: a head with nothing useful to attend to still has to put its weight somewhere, and the first token is always there. [Chapter 6](#reach-past-the-window) found the pattern only weakly in the store's model, whose description tokens put 0.11 of their attention on its two markers, against an even spread's 0.05.
**Working rule:** Describe a large model's heads by the same measured averages you'd use on a small one, compared with an even spread. Expect a few specialized heads and many that put their weight on the first token, and keep that token when you cut a model's context to save memory.
## Test copying at scale {#copying-at-scale}
[Chapter 6](#test-what-the-model-copies) showed the store's model copying `cover` from a product's name into its description, by changing the name and watching the prediction follow. The same test asks whether SmolLM2 does it too. Whether its training text held this product's text can't be checked, and the test doesn't need it: if SmolLM2 predicted `cover` from memory of the product, taking `cover` out of the name wouldn't move it.
For the store, read SmolLM2's recorded probabilities for `cover` and `set` after `this outdoor patio furniture`, in the description of `double chaise patio furniture cover`, with the name as sold, without `cover`, and ending in `set`:
Example `open-model-copying` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from lib import open_model
copying = open_model.read("copying")
for label, probabilities in copying.items():
for token, p in probabilities.items():
print(f"SmolLM2-360M, {label}, P({token.strip()}):", f"{p:.2f}")
```
Output:
```text
SmolLM2-360M, name as sold, P(cover): 0.75
SmolLM2-360M, name as sold, P(set): 0.00
SmolLM2-360M, name without cover, P(cover): 0.02
SmolLM2-360M, name without cover, P(set): 0.02
SmolLM2-360M, name ending in set, P(cover): 0.01
SmolLM2-360M, name ending in set, P(set): 0.77
```
With the name as sold, SmolLM2 gives `cover` 0.75. Take `cover` out of the name and it falls to 0.02; end the name in `set` and `set` gets 0.77. The store's model behaved the same way: 0.97, 0.00 and 0.70. Both models copy the next word of an earlier phrase when the phrase repeats, the behaviour of [induction heads](#read-trained-heads), heads that find an earlier copy of the current token and attend to what followed it. Olsson and colleagues found them in every model they studied with more than one layer ([Olsson et al., 2022](https://arxiv.org/abs/2209.11895)).
So copying isn't something scale adds; the store's 4-layer model learned it too. For a store, the habit is useful and risky at once: a model that copies words from the product's name stays on the product, and one that copies from the wrong product's text spreads its mistake.
**Working rule:** Test a behaviour by changing the input and watching the output, as here, and run the same test on every model you compare. When two models of very different sizes share a behaviour, it's a basic one; look for what differs elsewhere.
## Suggest the next word at scale {#suggestions-at-scale}
The search box is where the store's text matters most. [Chapter 8](#autocomplete-in-the-search-box) found the store's model alone giving `stool` a probability of 0.50 after `outdoor bar`, and the mix of the store's model and its counts, the model the search box serves, 0.28.
For the store, read SmolLM2's three likeliest next tokens after the same five prefixes, each given with nothing before it, as the store's model got them:
Example `open-model-autocomplete` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from lib import open_model
for prefix, recorded in open_model.read("prefixes").items():
shown = ", ".join(f"{token!r} {p:.2f}" for token, p in recorded["top"][:3])
print(f"{prefix:<12} -> {shown}")
print("P(stool | outdoor bar), SmolLM2-360M:", f"{open_model.read('prefixes')['outdoor bar']['P( stool)']:.4f}")
```
Output:
```text
bar -> 'ry' 0.05, 'ley' 0.05, 'b' 0.04
patio bar -> '\n' 0.21, 'be' 0.06, ',' 0.06
outdoor bar -> '\n' 0.28, ',' 0.06, '\n\n' 0.04
kitchen bar -> '\n' 0.37, '\n\n' 0.03, ',' 0.02
wine bar -> '\n' 0.20, ',' 0.09, '\n\n' 0.03
P(stool | outdoor bar), SmolLM2-360M: 0.0002
```
SmolLM2 gives `stool` after `outdoor bar` a probability of 0.0002. Its likeliest next token after `outdoor bar`, `patio bar`, `kitchen bar` and `wine bar` is a newline: it takes each prefix as a complete line, likely because short lines like these end in a newline in much of the text it learned from. After `bar` alone, it suggests pieces of longer words, such as `ry` and `ley`, as in `barry` or `barley`. None of that is wrong English; it's the wrong text. A search box needs the store's habits, and the store's own models have them. A search box also has to answer within a keystroke, and the store's model runs a small fraction of SmolLM2's arithmetic per token.
**Working rule:** Don't put a general model in front of shoppers without measuring its suggestions on your own prefixes against what you have. Where a model trained on your text suggests better, keep it, and measure a general model on the jobs where its broader knowledge could matter.
## Set the two models side by side {#side-by-side}
With every measure taken, the two models can be compared in one table. The store's model is the book's own, measured in chapters 6 to 8; SmolLM2-360M is read from its recordings in this chapter.
| | The store's model | SmolLM2-360M |
| --- | --- | --- |
| Parameters | 1,334,144 | 361,821,120 |
| Layers and width | 4 layers, 128 wide | 32 layers, 960 wide |
| Query heads, and key and value heads, per layer | 4 and 4 | 15 and 5 |
| Vocabulary | 4,142 | 49,152 |
| Text, in tokens: the catalog, or tokens trained on | 2,843,222, the store's catalog | 4 trillion, mostly general text |
| Per-word perplexity on the store's names | 199.3 | 54,352.8 |
| Per-word perplexity on the store's descriptions | 37.0 | 61.2 |
| P(`cover`) when the description repeats the name | 0.97 | 0.75 |
| The most weight one head puts on the previous token | 0.35 | 0.79 |
Read down the columns and the pattern of this chapter shows. SmolLM2 is far larger on every measure of size, and both models have learned the same basic behaviours, copying and attending to the previous token. Everything about the store's own text favours the small model, because that's all it learned. The store's choice isn't between a small model and a big one; it's which job each does.
**Working rule:** Set every model you're choosing between side by side on the measures that matter for your jobs, as here, with the same text and the same tests in every column. Choose a model per job, not one model for everything.
## What goes wrong in production {#open-models-in-production}
An open model brings its makers' choices with it, and they don't announce themselves:
- **Its text isn't your text.** SmolLM2 scored the store's names far worse than the store's own small model, and a label didn't help. Measure any model on your own text before you trust it there.
- **Its tokenizer isn't yours.** Per-token numbers from two tokenizers can't be compared; score per word or per character, on the same events.
- **Its details have to match the code that runs it.** Grouped keys and values, a RoPE base of 100,000 and tied embeddings are all in SmolLM2-360M's configuration. [Chapter 7](#measure-mismatched-details) measured what a wrong RoPE base costs: a little accuracy and no error. Load models with the code their makers publish, at a pinned version, or check every detail against it.
- **Its revision changes.** A model's repository can be updated under the same name. Pin the revision, as `recordings/open_model.py` does, and record it with every result.
- **Its first token matters.** Most of SmolLM2's later heads lean on the first token. Keep it when you trim context, and never let a pipeline drop it silently.
- **Its size sets its speed.** Every generated token runs every layer, so a model a few hundred times larger takes a few hundred times the arithmetic per token. Measure the time per token on the machine you'll serve from, at the text lengths you'll serve, before you promise a response time.
- **Its licence and terms travel with it.** SmolLM2 is Apache 2.0; other open models carry other terms. Check the licence of the exact model and revision before you build on it.
## What this chapter decided {#open-model-decisions}
- **The book's open model is SmolLM2-360M, read from two recordings** at pinned revisions: its run on the store's text, and the configurations of its family. Every example runs without PyTorch or a download, and the results can be checked against what was recorded.
- **Models with different tokenizers are compared per word,** on the same text and the same events: each name's words and its end, from the start token; each description's words and its end, given the name.
- **On the store's text, the store's small model beats SmolLM2,** 199.3 against 54,352.8 per word on names, and 37.0 against 61.2 on descriptions. The search box keeps the mix of the store's model and its counts that [chapter 8](#autocomplete-in-the-search-box) chose.
- **Heads are compared across models by the same measures,** against an even spread. SmolLM2 shows both of [chapter 6](#attention)'s patterns, more sharply: a few heads that specialize in the previous token, and many that put their weight on the first token.
- **Copying from earlier in a text is basic,** shared by a 4-layer model and a 32-layer one, and tested the same way in both.
- **A model's cost is counted from its configuration before it's served:** its weights, plus one key and value cache per text in progress. Among the SmolLM2 sizes, grouped keys and values cut the 360M model's cache to a third.
# From predicting to following instructions
By the end of this chapter you can explain how a model that predicts the next token becomes one that follows requests, build the two training losses that do it, and measure what that change does and doesn't fix. You'll compare a base model with its instruction-tuned version on the same requests, measure how often each makes up a product detail, check whether its confidence warns you when it does, and find where in a longer context a model stops finding what it was given.
This book follows one case throughout: an online furniture and home-goods store, which it calls the store, and the text of its product names and descriptions. Every model in the book so far has done one thing: predict the next token of text like its training text. [Chapter 10](#inside-an-open-model) read **SmolLM2-360M**, an open model trained that way on general text, and found it scored the store's names far worse, per word, than the store's own small model, and its descriptions worse. People don't want a continuation of their text; they want an answer to their request. Models are taught that in two further steps of training.
**Instruction tuning**, or supervised fine-tuning, continues training the model on examples of requests paired with good responses, so that predicting the next token of such a pair means answering the request. Wei and colleagues found that tuning on many tasks phrased as instructions made a model better at tasks it had never seen ([Wei et al., 2022](https://arxiv.org/abs/2109.01652)). **Preference tuning** then trains the model to prefer the better of two responses to the same request. Ouyang and colleagues had people rank the responses, and trained with reinforcement learning from a reward model learned from those rankings ([Ouyang et al., 2022](https://arxiv.org/abs/2203.02155)); **Direct Preference Optimization**, DPO, gets the same effect with a simple loss on the preference pairs themselves ([Rafailov et al., 2023](https://arxiv.org/abs/2305.18290)).
**SmolLM2-360M-Instruct** is the base model of [chapter 10](#inside-an-open-model) after both steps: supervised fine-tuning on public and curated datasets, then DPO on UltraFeedback ([model card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct/blob/a10cc1512eabd3dde888204e902eca88bddb4951/README.md)). UltraFeedback's preference pairs were scored by a large language model, not by people ([Cui et al., 2023](https://arxiv.org/abs/2310.01377)). This chapter calls SmolLM2-360M the base model and SmolLM2-360M-Instruct the tuned model. The script `recordings/instruction_tuning.py` ran both once, at pinned revisions, decoding greedily, so each answer is the model's likeliest and the run can be repeated exactly, and saved what they wrote and how likely they found each token it wrote. The examples measure everything from those files.
## Build the two training losses by hand {#build-the-losses}
Both steps of tuning are training, with the loop of [chapter 8](#update-with-adamw); what changes is the loss. Instruction tuning uses the ordinary next-token loss with one difference: only the response's tokens count. The request is read, as context, but the model isn't trained to predict it, since the point is to answer requests, not to write them.
For the store, lay a product out as a request and a response, its name as the request and its description as the response, as the store's model already reads it. Compute the store's model's loss over every token, and over the response's tokens only:
Example `instruction-loss-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import transformer
from lib.store import load_products
_, vocabulary, manifest = transformer.load()
index = {token: i for i, token in enumerate(vocabulary)}
product = load_products("validation").query("name == 'double chaise patio furniture cover'").iloc[0]
ids = transformer.product_ids(product["name"], product["description"], index, manifest["settings"]["longest"])
sep = ids.index(index[transformer.SEP]) # the name plays the request; the description, the response
probabilities = transformer.softmax(transformer.forward(ids)[:-1])
loss = -np.log(probabilities[np.arange(len(ids) - 1), ids[1:]]) # each token's loss, given the tokens before it
response = np.arange(1, len(ids)) > sep # the description's tokens and its end
print("tokens predicted in the text:", len(loss))
print("tokens the instruction loss counts, the response's:", int(response.sum()))
print("average loss over every token:", f"{loss.mean():.3f}")
print("average loss over the response only:", f"{loss[response].mean():.3f}")
```
Output:
```text
tokens predicted in the text: 96
tokens the instruction loss counts, the response's: 90
average loss over every token: 2.267
average loss over the response only: 2.132
```
Of the 96 tokens the model predicts, the instruction loss counts 90: the description's tokens and its end. The average loss over every token is 2.267, and over the response only 2.132. Training on the second number, over many requests and responses, teaches a model to produce responses.
DPO's loss works on a pair of responses to one request, one preferred and one rejected. It compares how much more likely the model being tuned makes each response than the **reference model** it started from, the model after instruction tuning, and rewards it for raising the preferred one more. With log-probabilities log π of each response under the tuned model and log π_ref under the reference, and a factor β that sets how far the model may move:
margin = β × [(log π(preferred) − log π_ref(preferred)) − (log π(rejected) − log π_ref(rejected))]
loss = −log σ(margin)
with σ the logistic function from [chapter 4](#learn-by-predicting-neighbours) ([Rafailov et al., 2023](https://arxiv.org/abs/2305.18290), equation 7). For the store, follow the arithmetic on made-up log-probabilities, with β = 0.1:
Example `dpo-loss-by-hand` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
beta = 0.1 # how far the tuned model may move from the reference
print("beta:", beta)
# Total log-probabilities of a preferred and a rejected answer to one request, under the model being tuned and under
# the reference model it started from. Made-up numbers, to follow the arithmetic.
cases = {"tuned model favours the preferred answer more than the reference does": (-12.0, -20.0, -14.0, -18.0),
"tuned model favours it less": (-15.0, -15.0, -14.0, -18.0)}
for label, (preferred, rejected, preferred_reference, rejected_reference) in cases.items():
margin = beta * ((preferred - preferred_reference) - (rejected - rejected_reference))
loss = np.log1p(np.exp(-margin)) # -log σ(margin), written so it can't overflow
print(f"{label}, margin:", f"{margin:.2f}")
print(f"{label}, loss:", f"{loss:.3f}")
```
Output:
```text
beta: 0.1
tuned model favours the preferred answer more than the reference does, margin: 0.40
tuned model favours the preferred answer more than the reference does, loss: 0.513
tuned model favours it less, margin: -0.40
tuned model favours it less, loss: 0.913
```
When the tuned model favours the preferred response more than the reference does, the margin is 0.40 and the loss 0.513; when it favours it less, the margin is -0.40 and the loss rises to 0.913. Lowering the loss pushes the preferred response's probability up and the rejected one's down, measured against where the model started.
**Working rule:** Mask the request out of the loss when you tune a model on requests and responses, and keep the reference model's log-probabilities beside the tuned model's when you train on preferences. Both are small changes to a loop you already have.
## Ask the base model and the tuned model {#ask-base-and-tuned}
The store needs descriptions for products that have none, the job [chapter 9](#choosing-the-next-token) gave its own small model. The base model, SmolLM2-360M, can't be asked, only continued, so it gets the start of a product listing: `Product:`, the name, and `Description:`. The tuned model, SmolLM2-360M-Instruct, gets a request: `Write a short product description for:` and the name.
For the store, read what each model wrote for [chapter 9](#choosing-the-next-token)'s product, `front porch welcome sign wall décor`, and measure all 8 products the recording described, each without a description to draw on:
Example `base-and-tuned` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import claims, recorded
described = recorded.read("instruction-tuning", "descriptions")
product = next(row for row in described if row["name"] == "front porch welcome sign wall décor")
print("product:", product["name"])
print("base model:", " ".join(product["base"]["text"].split()))
print("tuned model:", " ".join(product["tuned"]["text"].split()))
def repeats(text):
"""The share of four-word phrases that repeat an earlier one in the text."""
words = text.lower().split()
phrases = [tuple(words[i:i + 4]) for i in range(len(words) - 3)]
return sum(count - 1 for count in Counter(phrases).values()) / max(len(phrases), 1)
print("products described, none of which has a description to draw on:", len(described))
for role in ["base", "tuned"]:
texts = [row[role]["text"] for row in described]
invented = sum(bool(claims.materials(row[role]["text"]) - claims.materials(row["name"])) for row in described)
print(f"{role} model, descriptions that name a material the name doesn't:", f"{invented} of {len(described)}")
print(f"{role} model, repeated four-word phrases:", f"{sum(map(repeats, texts)) / len(texts):.0%}")
```
Output:
```text
product: front porch welcome sign wall décor
base model: Front porch welcome sign wall décor The front porch is a common feature of many homes. It is a place where people can relax and enjoy the outdoors. It is also a place where people can socialize and meet new friends. The front porch is a great place to hang out with your family
tuned model: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
products described, none of which has a description to draw on: 8
base model, descriptions that name a material the name doesn't: 2 of 8
base model, repeated four-word phrases: 14%
tuned model, descriptions that name a material the name doesn't: 1 of 8
tuned model, repeated four-word phrases: 0%
```
For the welcome sign, the base model writes about front porches: `The front porch is a common feature of many homes`, then about relaxing and meeting friends. It continues the text in a plausible direction, which isn't the one the store needs. The tuned model writes something shaped like a product description, `a statement piece that showcases your personality and style`, but calls the sign `a space` and `This wall`: fluent about a product it doesn't understand. Across the products, the base model's texts repeat 14% of their four-word phrases, and the tuned model's 0%. The recording can't say which step of tuning stopped the loops.
Neither model knows anything about the products beyond their names. The base model named a material the name doesn't give in 2 of 8 descriptions, and the tuned model in 1 of 8, and each of those came from the model, not from the store's data, as [chapter 9](#choosing-the-next-token)'s did.
**Working rule:** Use a tuned model when you need a request followed, and a base model when you need text continued in the style of what came before. Don't read fluency as knowledge: a tuned model writes a better-shaped description from the same lack of facts.
## Format the request as the model was tuned {#format-the-request}
The tuned model answered in the [previous section](#ask-base-and-tuned) because its request came in the layout it was tuned on, its **chat template**: special tokens that mark where a **system message**, instructions placed before the user's request, the request itself and the model's reply begin and end. The model learned to answer requests only in that layout, so a request in any other form is only text to continue. Libraries that load the model apply the template from the model's tokenizer files ([tokenizer configuration](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct/blob/a10cc1512eabd3dde888204e902eca88bddb4951/tokenizer_config.json)).
For the store, read exactly what the tuned model was given with its template, and compare its reply with the reply to the same request given as plain text:
Example `chat-template` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from lib import recorded
template = recorded.read("instruction-tuning", "template")
print("request:", template["with"]["prompt"].split("<|im_start|>user\n")[1].split("<|im_end|>")[0])
print("what the tuned model reads, with its chat template:")
print(template["with"]["prompt"])
print("tuned model, with the template:", " ".join(template["with"]["text"].split()))
print("tuned model, without it:", " ".join(template["without"]["text"].split()))
```
Output:
```text
request: Write a short product description for: front porch welcome sign wall décor
what the tuned model reads, with its chat template:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
Write a short product description for: front porch welcome sign wall décor<|im_end|>
<|im_start|>assistant
tuned model, with the template: "Welcome to our front porch welcome sign wall décor, a space that brings warmth and comfort to your home. This wall is a statement piece that showcases your personality and style. With a mix of rustic and modern elements, you can create a unique and inviting atmosphere that reflects your individuality.
tuned model, without it: .
```
With the template, the request sits between `<|im_start|>user` and `<|im_end|>`, after a system message the template adds by default, `You are a helpful AI assistant named SmolLM, trained by Hugging Face`, and before `<|im_start|>assistant`, where the model's reply begins. The model replies with the description of the previous section. Without the template, given the same words as plain text, the tuned model writes `.` and stops. Tuning didn't teach the model to follow requests in general; it taught it to follow requests in that layout.
The system message matters too: it's part of every request the model reads, so a different one, such as a description of the store and its tone, can change any reply. That makes it a setting to choose and record, like a decoding setting in [chapter 9](#choosing-the-next-token).
**Working rule:** Always format requests to a tuned model with its own chat template, applied by the library from the model's tokenizer files, never assembled by hand. Check the template's default system message, since the model reads it on every request, and record the full formatted prompt with every answer you keep.
## Measure made-up product details {#measure-made-up-details}
A model that writes about a product with no facts has to make them up. The fairer test gives it the facts. Both SmolLM2 models are shown a product's name and real description, and asked: `What material is this product made of? Answer in a few words, using only the description.` The tuned model gets it as a request in its template; the base model gets it as a listing to continue, ending `Question:`, the question, and `Answer:`. If the product's data names a material, a good answer names it; if not, a good answer says the description doesn't say.
Checking the answers takes a list of material words, from `wood` and `metal` to `polyester` and `leather`, in the book's shared module `lib.claims`. An answer is **supported** if every material it names is in the product's name or description, **made up** if it names one that isn't, **declined** if it says the data doesn't say, and **none** if it names no material and doesn't decline. The check has limits, and makes judgement calls: it counts engineered wood such as `mdf` and `rubberwood` as wood, it counts `brushed nickel finish` as naming nickel, and it misses a material the list doesn't include. Every answer it classes is recorded for anyone to read. Making up a detail the input doesn't support is one kind of what's called **hallucination** ([Ji et al., 2023](https://arxiv.org/abs/2202.03629)), and this test measures exactly that kind.
For the store, class each model's answers for 40 products drawn at random, split by whether the product's data names a material. Before you run it, predict whether the tuned model makes up fewer materials than the base model where the data names none:
Example `made-up-materials` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from collections import Counter
from lib import claims, recorded
asked = recorded.read("instruction-tuning", "answers")
print("products asked, each with its description given:", len(asked))
given = [bool(claims.materials(row["description"]) | claims.materials(row["name"])) for row in asked]
print("products whose name or description names a material:", sum(given))
print("products whose name and description name none:", len(given) - sum(given))
for role in ["base", "tuned"]:
for label, keep in [("data names a material", True), ("data names none", False)]:
rows = [row for row, named in zip(asked, given) if named == keep]
counts = Counter(claims.classify(row[role]["text"], row["description"], row["name"]) for row in rows)
for kind in ["supported", "made up", "none", "declined"]:
print(f"{role} model, {label}, {kind}:", counts[kind])
```
Output:
```text
products asked, each with its description given: 40
products whose name or description names a material: 28
products whose name and description name none: 12
base model, data names a material, supported: 24
base model, data names a material, made up: 2
base model, data names a material, none: 2
base model, data names a material, declined: 0
base model, data names none, supported: 0
base model, data names none, made up: 7
base model, data names none, none: 5
base model, data names none, declined: 0
tuned model, data names a material, supported: 23
tuned model, data names a material, made up: 3
tuned model, data names a material, none: 2
tuned model, data names a material, declined: 0
tuned model, data names none, supported: 0
tuned model, data names none, made up: 8
tuned model, data names none, none: 4
tuned model, data names none, declined: 0
```
| Answer | Base, data names a material | Tuned, data names a material | Base, data names none | Tuned, data names none |
| --- | --- | --- | --- | --- |
| Supported | 24 | 23 | 0 | 0 |
| Made up | 2 | 3 | 7 | 8 |
| None | 2 | 2 | 5 | 4 |
| Declined | 0 | 0 | 0 | 0 |
Of the products, 28 have data that names a material, and on those both models mostly answer from it. On the 12 that don't, the right answer is that the description doesn't say, and neither model gave it once. Both mostly made a material up instead, and the rest of their answers named none without saying why.
Instruction tuning didn't fix this, and on these products the base model, given a listing to continue, did no worse. The request said to use only the description; when the description had no answer, the models answered anyway, with a material that products like this one often have. Following the form of an instruction isn't the same as knowing when you don't know.
**Working rule:** Test a model on questions its input can't answer, not only on ones it can, and count made-up answers separately. Check every factual claim a model makes about a product against the product's data, automatically where you can, as `lib.claims` does for materials.
## Check whether confidence warns you {#check-confidence}
The [previous section](#measure-made-up-details) classed the tuned model's answers to the material question as supported or made up. If its made-up answers came with low confidence, low confidence could flag them. A model whose confidence matches how often it's right is **calibrated**, and Guo and colleagues found that the deep neural networks they studied often aren't: they were more confident than their accuracy warranted ([Guo et al., 2017](https://arxiv.org/abs/1706.04599)). The recording saved the probability the tuned model gave each token it wrote, its end token included, so a confidence per token can be computed for each answer, as a geometric mean:
confidence per token = exp((log p₁ + … + log pₙ) / n)
the exponential of the average log-probability, the same average that perplexity inverts, so a short answer and a long one are measured on the same scale.
For the store, compare the tuned model's confidence in its supported and made-up answers, and count how many of each a threshold of 0.5 per token would keep:
Example `answer-confidence` (Python 3.12.15 · numpy 2.5.4), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import claims, recorded
asked = recorded.read("instruction-tuning", "answers")
by_kind = {"supported": [], "made up": []}
for row in asked:
kind = claims.classify(row["tuned"]["text"], row["description"], row["name"])
if kind in by_kind:
by_kind[kind].append(np.exp(np.mean(np.log(row["tuned"]["token_probabilities"])))) # per token, on average
for kind, confidence in by_kind.items():
print(f"tuned model, {kind} answers:", len(confidence))
print(f"tuned model, {kind} answers, average probability per token:", f"{np.mean(confidence):.2f}")
threshold = 0.5
for kind, confidence in by_kind.items():
print(f"tuned model, {kind} answers above {threshold} per token:", f"{sum(c > threshold for c in confidence)} of {len(confidence)}")
```
Output:
```text
tuned model, supported answers: 23
tuned model, supported answers, average probability per token: 0.65
tuned model, made up answers: 11
tuned model, made up answers, average probability per token: 0.45
tuned model, supported answers above 0.5 per token: 21 of 23
tuned model, made up answers above 0.5 per token: 3 of 11
```
The made-up answers are less confident on average, 0.45 per token against 0.65 for supported ones. But a threshold of 0.5 per token would keep 21 of 23 supported answers and still let through 3 of 11 made-up ones. Confidence is a signal, not a filter: it shifts the odds, and it lets confident errors through. With so few answers, these are directions, not rates to rely on.
**Working rule:** Don't use a model's confidence alone to decide whether to trust an answer. Measure how its confidence relates to being right on your own questions, as here, and if you use a threshold, choose it from those measurements and count what it lets through.
## Find what a longer context loses {#find-what-context-loses}
The [material question](#measure-made-up-details) gave the model one product at a time. A store's assistant often has several products' data in front of it at once, such as search results, a category page or a comparison, and has to find the right product's details among the others. Liu and colleagues found that models use information at the start and the end of a long context better than information in the middle ([Liu et al., 2024](https://arxiv.org/abs/2307.03172)).
For the store, give the tuned model, SmolLM2-360M-Instruct, 10 product listings, the asked product among them, first, in the middle or last, and ask what that product is made of. Count the answers that name only its own materials, as `lib.claims` checks them, for the 14 products whose data names one:
Example `answer-position` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from collections import defaultdict
from lib import claims, recorded
context = recorded.read("instruction-tuning", "context")
others = recorded.read("instruction-tuning", "manifest")["settings"]["context_others"]
print("product listings in each context:", others + 1)
results = defaultdict(list)
for row in context:
if claims.materials(row["description"]) | claims.materials(row["name"]): # only products whose data names one
results[row["place"]].append(claims.classify(row["tuned"]["text"], row["description"], row["name"]) == "supported")
print("products asked about at each place:", len(results["first"]))
for place in ["first", "middle", "last"]:
print(f"product placed {place}, answers that name only its own materials:", f"{sum(results[place])} of {len(results[place])}")
```
Output:
```text
product listings in each context: 10
products asked about at each place: 14
product placed first, answers that name only its own materials: 12 of 14
product placed middle, answers that name only its own materials: 6 of 14
product placed last, answers that name only its own materials: 12 of 14
```
With the product first, 12 of 14 answers name only its materials; last, 12 of 14; in the middle, 6 of 14. A product in the middle is found least often, the pattern Liu and colleagues found in much larger models. The counts are small, but the gap is wide. The context here is short, a few thousand characters, well within the model's limit: the middle loses details long before a context is full.
**Working rule:** Put the details a model most needs at the start or the end of its context, not in the middle, and give it as little unrelated text as the task allows. Test retrieval from a longer context at each position, as here, before trusting a model to find details among many.
## Adapt a model to your text {#adapt-a-model}
[Chapter 10](#inside-an-open-model) found the general model far worse than the store's small model on the store's names, per word. Instruction tuning changes how a model answers, not what it knows about one store. The way to teach it the store's text is **fine-tuning**: continue training it on that text, the same training loop as [chapter 8](#update-with-adamw), starting from the general model's weights instead of random ones.
Full fine-tuning updates every weight, which for a model of hundreds of millions of parameters takes time and memory. **LoRA**, low-rank adaptation, freezes the model's weights and trains a small correction beside some of them. For a weight matrix that maps k numbers to d numbers, it trains a matrix B of d by r and a matrix A of r by k, and adds their product to the weights: W + BA ([Hu et al., 2022](https://arxiv.org/abs/2106.09685)). With a small rank r, there's far less to train and store, and Hu and colleagues found it matched full fine-tuning on the tasks they tested.
For the store, count what LoRA would train on SmolLM2-360M, adapting the query and value weights of every layer, at two ranks, against full fine-tuning. The query weights map 960 numbers to 960; the value weights, because keys and values are grouped, map 960 to 320:
Example `lora-parameters` (Python 3.12.15), passing in Edition 2026.10.3:
```python
from lib import open_model
config = open_model.config("SmolLM2-360M")
width, layers = config["hidden_size"], config["num_hidden_layers"]
key_value_width = config["num_key_value_heads"] * width // config["num_attention_heads"]
total = open_model.parameters(config)["total"]
for rank in [8, 64]:
# LoRA adds two small matrices beside each adapted weight: width x rank, then rank x output width.
added = layers * (rank * (width + width) + rank * (width + key_value_width)) # on the query and value weights
print(f"rank {rank}, parameters trained:", f"{added:,}")
print(f"rank {rank}, share of the model's parameters:", f"{added / total:.2%}")
print("parameters trained by full fine-tuning:", f"{total:,}")
```
Output:
```text
rank 8, parameters trained: 819,200
rank 8, share of the model's parameters: 0.23%
rank 64, parameters trained: 6,553,600
rank 64, share of the model's parameters: 1.81%
parameters trained by full fine-tuning: 361,821,120
```
At rank 8, LoRA trains 819,200 parameters, 0.23% of the model; at rank 64, 6,553,600, or 1.81%. Full fine-tuning trains all 361,821,120. A rank-8 correction is smaller than the store's own small model, and it can be stored and swapped per task while the base weights stay shared.
This book doesn't run the fine-tuning: our judgement is that even LoRA, over the store's text, would take far longer on a laptop's processor than any recording here. It's the next step for a store that wants a general model to write in its own voice: fine-tune on the catalog, then measure the result against the store's own model, per word, as [chapter 10](#inside-an-open-model) did. Fine-tuning has risks of its own. A model trained on the catalog learns its gaps along with its text, as the store's small model learned to leave a bulb's description empty in [chapter 9](#forbid-an-empty-description). And training on one narrow text can weaken what the model knew before; Biderman and colleagues found LoRA forgets less of it than full fine-tuning ([Biderman et al., 2024](https://arxiv.org/abs/2405.09673)). Both are measured the same way as everything else in this book: on held-out text, before and after.
**Working rule:** To teach a general model your text, fine-tune it on that text, with LoRA when memory or storage is tight, and measure the result on held-out text of the same kind. Don't expect instruction tuning or a cleverer prompt to supply knowledge the model never learned.
## Look back at the yardstick {#look-back-at-the-yardstick}
This book has scored every model the same way on names, from [chapter 3](#counting-further-back) on: perplexity per token of the store's tokenizer, on the store's validation names. The test names were kept aside for one final comparison, so that choices made by looking at the validation names, such as the kept epochs and the mixing shares, couldn't flatter the results. Now they're used. For the store, score the five models that were chosen between on the 4,281 test names, with the mixing shares chosen on validation:
Example `test-names` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.3:
```python
import numpy as np
from lib import neural, transformer
from lib.names import name_tokens
from lib.ngrams import KneserNey, ngrams, padded
# The test names, untouched until now: every model scores each name's tokens and its end. The mixing shares are the
# ones chosen on the validation names, in chapters 5 and 8.
test = name_tokens("test")
counted = KneserNey(name_tokens("train"), 4)
from_counts = np.array([counted.probability(c, t) for text in test for c, t in ngrams(padded(text, 4), 4)])
from_window = neural.token_probabilities(test)
_, vocabulary, _ = transformer.load()
index = {token: i for i, token in enumerate(vocabulary)}
from_transformer = []
for name in test:
ids = [index[transformer.START]] + [index[t] for t in name] + [index[transformer.SEP]]
probabilities = transformer.softmax(transformer.forward(ids)[:-1])
from_transformer += list(probabilities[np.arange(len(ids) - 1), ids[1:]])
from_transformer = np.array(from_transformer)
perplexity = lambda p: f"{np.exp(-np.log(p).mean()):.1f}"
print("test names:", f"{len(test):,}")
print("test perplexity, the 4-gram baseline:", perplexity(from_counts))
print("test perplexity, the window model:", perplexity(from_window))
print("test perplexity, the window model mixed with the counts, share 0.2:", perplexity(0.2 * from_window + 0.8 * from_counts))
print("test perplexity, the store's transformer:", perplexity(from_transformer))
print("test perplexity, the store's transformer mixed with the counts, share 0.5:", perplexity((from_transformer + from_counts) / 2))
```
Output:
```text
test names: 4,281
test perplexity, the 4-gram baseline: 22.9
test perplexity, the window model: 35.4
test perplexity, the window model mixed with the counts, share 0.2: 21.6
test perplexity, the store's transformer: 23.7
test perplexity, the store's transformer mixed with the counts, share 0.5: 18.3
```
Set side by side, the scores tell the book's story in one table, and the test column agrees with the validation one to within a few tenths: the choices made on validation flattered the results very little.
| Model | Chapter | Validation names | Test names |
| --- | --- | --- | --- |
| Counts of single tokens, a 1-gram | [3](#smooth-with-kneser-ney) | 687.2 | |
| Counts of pairs, a 2-gram | [3](#smooth-with-kneser-ney) | 46.9 | |
| Counts of four tokens, the 4-gram baseline | [3](#the-baseline) | 23.1 | 22.9 |
| The window model, a small neural network | [5](#compare-with-the-baseline) | 36.4 | 35.4 |
| The window model mixed with the counts | [5](#mix-with-the-counts) | 21.9 | 21.6 |
| The store's transformer | [8](#compare-on-names) | 24.0 | 23.7 |
| The store's transformer mixed with the counts | [8](#compare-on-names) | 18.4 | 18.3 |
Three lessons run through it. Counting goes a long way on short, formulaic text: no learned model in this book beat the 4-gram baseline on names alone, and both mixes beat it. A learned model earns its place by being wrong in different places from the counts, which is why every mix beat both its parts. And on longer text, where distant context matters, the transformer pulled ahead on its own, as [chapter 8](#compare-on-descriptions) measured on descriptions. Size alone didn't decide anything: [chapter 10's](#score-the-store-text) general model, hundreds of times larger, scored 54,352.8 per word on the store's names, against the store's transformer's 199.3.
None of these numbers says whether a model is useful. Perplexity measures how well a model predicts text like its training text, which is what autocomplete needs. It doesn't measure whether a description is true, whether an answer follows a request, or whether a shopper finds what they came for. This chapter's tests, made-up materials, confidence and the place of a fact in a context, are a first step past perplexity, toward measuring the job itself.
**Working rule:** Keep one fixed yardstick for comparing models, the same text, tokenizer and measure every time, as this book did. Then measure the job itself, with tests built for it, before choosing a model to serve it: a lower perplexity is a reason to test a model, not a reason to ship it.
## What goes wrong in production {#instruction-models-in-production}
A tuned model is the kind people can use without help, and its failures sound as fluent as its successes:
- **It answers when it should say it doesn't know.** The base and the tuned SmolLM2 models made up a material for most products whose data named none, and neither ever said the data didn't say. Check every factual claim against the product's data before it reaches a shopper.
- **Its confidence lets errors through.** Made-up answers were less confident on average, but a threshold that kept most good answers still let 3 of 11 made-up ones through.
- **Its format is part of its training.** Without its chat template, the tuned model wrote a full stop. Apply the template from the model's files, and record the full prompt with every answer.
- **The middle of a context gets lost.** Put the details that matter at the start or the end, and test retrieval at each position.
- **It doesn't know your text.** Instruction tuning teaches the form of answers, not a store's catalog. Fine-tune on the catalog, and measure.
- **Fluency hides gaps.** The tuned model's descriptions read better than the base model's and contain no more facts. Judge generated text by checks on its claims, not by how it reads.
- **Its behaviour depends on its revision and its template's default messages.** Pin the model's revision, as the recording does, and rerun these measurements after any update.
## What this chapter decided {#instruction-decisions}
- **The book compares base and tuned models on the same requests,** recorded once at pinned revisions and decoded greedily, and measures their answers with checks anyone can rerun, such as `lib.claims` for materials.
- **Made-up details are measured on questions the input can't answer** as well as ones it can. On the store's products, both SmolLM2 models made up a material for most products whose data named none, and neither declined once, so generated product facts are always checked against the product's data.
- **A model's confidence is a signal, not a filter.** It's used only with a threshold measured on the store's own questions, and the errors it lets through are counted.
- **The details a model needs go at the start or the end of its context,** never buried among other products.
- **Perplexity stays the book's yardstick for comparing models,** on the same text, tokenizer and events every time, and the job each model is chosen for is measured with tests built for that job.
- **Teaching a general model the store's text means fine-tuning,** measured per word against the store's own model; instruction tuning changes how a model answers, not what it knows. Measuring a model inside an application, beyond the next-token tests of this book, is a subject of its own.