---
title: "Evaluating LLM Applications"
book: evaluating-llm-applications
edition: 2026.11.3
verified: 2026-11-21
edition_code: 4V0Y-VKTV
canonical: https://rollingedition.com/evaluating-llm-applications
license: CC BY-NC 4.0
more_permissions: https://rollingedition.com/policies/reuse
attribution: "Evaluating LLM Applications, Edition 2026.11.3, examples passed Nov 21, 2026. By Rolling Edition, https://rollingedition.com/evaluating-llm-applications. Text CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), with added permissions (https://rollingedition.com/policies/reuse). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.11.3. Examples passed Nov 21, 2026. 54 of 54 examples pass. Written with AI. Verified by people.

To cite a section, name Edition 2026.11.3 and link to the section's permanent link: https://rollingedition.com/evaluating-llm-applications/s/{section_id}

# Measure an assistant before you change it

By the end of this chapter you can measure a documentation assistant with a fixed set of questions, read the result with its interval, and say when twenty questions are too few to tell two prompts apart.

The assistant in this book answers questions about sablefile, a command-line tool that keeps a project's generated files in step with the templates they come from. Users ask how to run a command or what a setting does, and the assistant answers from the tool's documentation, citing the section behind each statement. Suppose someone on the team wants a different prompt. They try it on three questions, the answers look better, and they ship it. Three questions can't show whether the change helped. Nobody can say what the assistant did before the change either, because nothing was written down.

So the first thing to build is a measurement: a set of questions with the answers they should get, a way to run the assistant on them the same way each time, and a number that comes with its size. Changes come after that. This chapter builds the measurement in its simplest form, and it ends by showing where that form stops.

## The tool and its docs {#the-tool-and-its-docs}

An evaluation is only as steady as the thing it measures. The sablefile docs come in three versions, 1.0, 1.1 and 2.0, and they disagree in places. An answer that is right for one version can be wrong for another, so every question has to name its version before anything else about it is settled.

Start with the docs. Load the three versions, count the sections in each, and count the section IDs that all three share.

Example `list-the-docs` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Loads the three doc versions and prints each one's section count, and how many section IDs all three share.

The shared IDs are the sections whose concept keeps one ID across versions (plan section 2.2)."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.docs import load_sections, versions

sections = load_sections()  # every version, sorted by version and section ID
all_versions = versions()  # the version labels present, sorted
for version in all_versions:
    print(f"sections in version {version}: {sum(1 for s in sections if s.version == version)}")

ids = {version: {s.section_id for s in sections if s.version == version} for version in all_versions}
shared = set.intersection(*ids.values())  # IDs present in every version
print(f"section IDs in all {len(all_versions)} versions: {len(shared)}")
print(f"section IDs in 2.0 but in neither 1.0 nor 1.1: {len(ids['2.0'] - ids['1.0'] - ids['1.1'])}")
```

Output:

```text
sections in version 1.0: 35
sections in version 1.1: 36
sections in version 2.0: 40
section IDs in all 3 versions: 35
section IDs in 2.0 but in neither 1.0 nor 1.1: 4
```

The output opens with the section counts: 35 in 1.0, 36 in 1.1, and 40 in 2.0.

Most sections keep their IDs across versions, since 35 IDs appear in all three. The last line matters most for what follows. 4 IDs exist only in 2.0, so an assistant that answers from the older pages has nothing to cite for them.

The versions also differ in what the tool does, and the difference that matters here is in `sync`, the command that writes the generated files. The 1.1 docs say that `sync` with no flags writes the files and records each one in `sablefile.lock`, the lock file in the project root. The 2.0 docs say that `sync` with no flag prints a plan and writes nothing, that `--apply` writes the files, and that the lock file lives at `.sablefile/lock.json`. A question about `sync` has one right answer for 1.1 and another for 2.0. An assistant that reads the wrong version's page sounds sure of itself and is wrong.

This book measures answers. Retrieval ranking, tracing, cost, latency, attacks and agents' task sets belong to other books, named where they come up.

**Working rule:** Write the version into every question, and let that version's own docs be the only source for its answer. When two versions use the same wording and give different answers, make them two items, one for each version. Never let the newest version's page stand in for an older one.

## The twenty items {#the-twenty-items}

A test set is a list of questions, each with the answer it should get. Twenty items is deliberately small. You can read each one, check its expected answer against the docs, and say why it is there. A set nobody reads can't tell you what it measures.

The items fall into four kinds. Answerable items have their answer on the page. Unanswerable items ask about something the docs don't cover, and the right reply is the decline. Version-dependent items have an answer that changes from one version to the next, and several of them have a twin: the same question asked about another version. Ambiguous items could mean two things, and their reference answer covers both readings.

Each item is printed in ID order, with its version, kind, twin and whether it was planted.

Example `twenty-questions` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Prints the twenty chapter items in id order: version, kind, planted flag and question, then the counts by kind.

The counts are the set's make-up that the chapter describes (plan section 3.4)."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.answers import chapter_items

items = chapter_items()  # sorted by id
for item in items:
    twin = item.twin_of if item.twin_of else "none"
    planted = f"planted ({item.planted_why})" if item.planted else "not planted"
    print(f"{item.id} | version {item.version} | {item.kind} | {planted} | twin {twin} | {item.question}")

kinds = Counter(item.kind for item in items)
for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):
    print(f"items of kind {kind}: {kinds[kind]}")
print(f"planted items: {sum(1 for item in items if item.planted)}")
print(f"items in the dev split: {sum(1 for item in items if item.split == 'dev')}")
```

Output:

```text
q-0001 | version 1.1 | version-dependent | not planted | twin q-0002 | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
q-0002 | version 2.0 | version-dependent | not planted | twin q-0001 | In sablefile 2.0, if I run sync with no flags, are the target files written?
q-0003 | version 1.0 | version-dependent | planted (version-swap) | twin q-0004 | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
q-0004 | version 2.0 | version-dependent | not planted | twin q-0003 | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
q-0005 | version 2.0 | answerable | planted (near-miss-term) | twin none | Does sablefile 2.0 still read sablefile.lock in my project root?
q-0006 | version 2.0 | answerable | not planted | twin none | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
q-0007 | version 1.1 | answerable | not planted | twin none | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
q-0008 | version 2.0 | answerable | not planted | twin none | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
q-0009 | version 1.0 | answerable | not planted | twin none | What does one line of the 1.0 lock file hold?
q-0010 | version 2.0 | answerable | not planted | twin none | Will syncing the same content twice in 2.0 produce a different lock file each time?
q-0011 | version 1.1 | answerable | not planted | twin none | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
q-0012 | version 2.0 | answerable | not planted | twin none | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
q-0013 | version 1.0 | answerable | not planted | twin none | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
q-0014 | version 1.1 | answerable | not planted | twin none | What does sablefile init put into a new sablefile.toml in 1.1?
q-0015 | version 1.0 | unanswerable | planted (false-premise) | twin none | In sablefile 1.0, what does the --dry-run flag on sync print?
q-0016 | version 2.0 | unanswerable | not planted | twin none | Does sablefile 2.0 run on Windows?
q-0017 | version 2.0 | unanswerable | not planted | twin none | How fast is sablefile check on a project with several thousand generated files?
q-0018 | version 1.1 | unanswerable | not planted | twin none | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
q-0019 | version 2.0 | ambiguous | planted (ambiguous-target) | twin none | If I edit the file and then run check, will it fail?
q-0020 | version 2.0 | ambiguous | not planted | twin none | Does sync change the lock file?
items of kind answerable: 10
items of kind unanswerable: 4
items of kind version-dependent: 4
items of kind ambiguous: 2
planted items: 4
items in the dev split: 20
```

The last lines give the make-up of the set by kind, and answerable items are the largest group. 4 items are planted. A planted item was written to catch one particular mistake, and the set labels it, so a failure on that item can be read against its trap.

Each trap is simple. `q-0003` asks whether `${project}` is the right placeholder in 1.0, where the syntax is `{{project}}`. Its twin, `q-0004`, asks the same about 2.0, where `${project}` is right. `q-0005` asks whether 2.0 still reads `sablefile.lock` in the project root, which uses the 1.x file name for a version where that name matters only for migration. `q-0015` asks what the `--dry-run` flag does in 1.0, and the 1.0 docs name no such flag. `q-0019` asks whether a file fails a check after you edit it, without saying which file, so it has two readings.

**Working rule:** Write each item with its version, its kind and its expected answer before you run the assistant, and label any item written to catch a specific mistake. Keep the set small enough to read in one sitting, and never change an expected answer to fit what the assistant said.

## Retrieve, then answer {#retrieve-then-answer}

The assistant doesn't read all the docs for every question. For each question it takes the three sections most likely to hold the answer, from the index of the version the question names, and answers from those three alone. A wrong answer can therefore come from two places: the right section was never retrieved, or the model misread the section it was given. A measurement that can't tell these apart can't say what to fix.

The index is built with BM25, which scores each section by the words it shares with the question. The scoring runs through bm25s at 0.3.16. Take one question: whether an ignore entry such as `docs/**/*.tmp` catches a file two folders down in 1.1. Its output shows the words the index keeps from the question, and the top three sections, ordered by score and then by section ID.

Example `retrieve-top-sections` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Shows BM25 retrieval for one question (q-0007, version 1.1): the words kept, then the top three sections.

The sections come from the version's own index, ranked by score and then by section ID, so a tie has one order.
The tokeniser drops one-letter words, so the `3` of `3.5mm` is lost and only `5mm` is matched."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import bm25s

from lib.docs import load_sections
from lib.retrieval import Retriever
from lib.testset import load_items

item = next(item for item in load_items() if item.id == "q-0007")
print(f"question {item.id} (version {item.version}): {item.question}")

# The words the index actually matches on: the same tokeniser, stopwords and token pattern as lib.retrieval.
kept = bm25s.tokenize([item.question], stopwords="en", return_ids=False, show_progress=False)[0]
print(f"words kept for matching: {' '.join(kept)}")
dropped = bm25s.tokenize(["a 3.5mm jack"], stopwords="en", return_ids=False, show_progress=False)[0]
print(f"words kept from the text 'a 3.5mm jack': {' '.join(dropped)}")

hits = Retriever(load_sections(item.version)).retrieve(item.question, 3)  # top 3 of the version's index
for rank, hit in enumerate(hits, start=1):
    print(f"rank {rank}: {hit.section_id} | score {hit.score:.3f}")
```

Output:

```text
question q-0007 (version 1.1): In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
words kept for matching: ignore entry docs tmp catch file two folders down like docs tmp
words kept from the text 'a 3.5mm jack': 5mm jack
rank 1: ignore-recursive | score 9.156
rank 2: sync-create-dirs | score 3.386
rank 3: ignore-syntax | score 3.348
```

Read the two lines of kept words before the ranking. The tokeniser drops every token of one character. The `a` in the question goes, and so do both `1`s of `1.1`, so the version number never reaches the index. That does no harm here, because each question is searched only in the index of its own version. The same rule takes the `3` out of `3.5mm`: what is left of `a 3.5mm jack` is `5mm jack`, so a question about a 3.5 mm jack can match only on `5mm`. Neither loss raises an error, and the second one changes what the index can find.

The ranking puts `ignore-recursive` first, the section on recursive ignore entries, and the next two are the section on creating folders and the one on ignore syntax. Ties are broken by section ID, so the same question gives the same three sections on every run.

**Working rule:** Check retrieval before you check the answer. For each item, record the sections the assistant was given next to its reply. A wrong reply that lacks the right section is a retrieval problem, and a wrong reply that had the right section is a problem with the prompt or the model. Keep the wording of questions plain, because a tokeniser that drops one-character tokens will lose a measurement or a version number.

## Replay the answers {#replay-the-answers}

A model's reply to the same question can change with its runtime, its settings and the hardware it runs on. If every measurement calls the model again, each one measures something slightly different, and a change in the number might be noise. So the book records each reply once, from a pinned open model, and replays the recording. A replay returns the same text every time.

The recording is a folder of replies, one per item. It was made with a 4-bit build of Qwen3-4B-Instruct-2507, an open model pinned to one file and one revision, run at temperature zero on a CPU. It holds a reply for every item in the full set, and this chapter reads twenty of them by ID. Replay one of them: the reply to `q-0006`, which asks whether `check` can exit with status 0 when a file has drifted in 2.0.

Example `replay-one-answer` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Replays the recorded baseline reply to q-0006 and parses it: the sections it was given, the reply, its citations.

Nothing is generated here: the reply is the recorded text, read back as the same string."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant, chapter_items
from lib.contract import format_problems
from lib.prompts import P1_SHA256, P1_SYSTEM

assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)  # the prompt hash must match the manifest
item = next(item for item in chapter_items() if item.id == "q-0006")
print(f"question {item.id} (version {item.version}, {item.kind}): {item.question}")
for rank, section in enumerate(assistant.sections(item), start=1):  # the top 3, in rank order
    print(f"section in context, rank {rank}: {section.section_id}")

reply = assistant.reply(item)
print("recorded reply:")
print(reply.text)
print(f"citations, in order: {' '.join(reply.citations)}")
print(f"format problems: {', '.join(format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=False)) or 'none'}")
# A reply to an answerable item must answer and cite; this is the one fact the prose relies on here.
assert reply.citations and not reply.declined, "q-0006 should be answered, with at least one citation"
```

Output:

```text
question q-0006 (version 2.0, answerable): Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
section in context, rank 1: check-exit-status
section in context, rank 2: no-fail-flag
section in context, rank 3: quickstart-check
recorded reply:
Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag with `sablefile check`. This flag ensures that the check exits with status 0 even when files have drifted, without changing which files are reported. [sec:no-fail-flag]
citations, in order: no-fail-flag
format problems: none
```

The output lists the three sections the assistant was given, in rank order, and then prints the reply exactly as it was recorded. The reply says yes, and it cites `no-fail-flag`, which was among the three. Check the citation against that list. The reply cites a section it was given, so it didn't reach outside its context. An assert also requires the reply to be an answer with at least one citation, since a decline on this answerable item would be an error.

To put a live model in place of the recording, pass `live=True` to `Assistant`. That loads the pinned Qwen3-0.6B on the CPU and answers each question as it's asked. It is a smaller model than the 4B build the recording was made with, so its live replies, and any rate built on them, will differ from the recorded ones. This book quotes no live replies. Its numbers come from the recording, so you can check each one against a file.

**Working rule:** Replay for every comparison. Record again only when the model, the prompt or the sections the assistant sees have changed, and note in the recording what changed. Each recording names its prompt by hash, and the example checks that hash against the recording's manifest, so one recording can't stand in for a different prompt.

## Two code checks {#two-code-checks}

Some errors need no judge. A citation to a section that wasn't in the prompt, or a decline that also gives an answer, can be found by reading the reply's text. Checks in code cost nothing to run and give the same result each time, so the book runs both of them before any model is asked to judge a reply.

### Citations {#citations-in-code}

The assistant must cite every statement with the citation form, a section ID in square brackets, and it may cite only the IDs it was given. A citation can fail in two ways. The ID may not exist at all, or it may exist in another version, which is the mistake the version-swap items are written to provoke. The check calls a citation to any section that was not in the prompt `unknown-section`, whichever way it went wrong. It also flags an answer that has no citation.

The citation and format checks run over the twenty baseline replies, and the output counts each kind of failure.

Example `check-citations` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Runs the citation and format checks on the baseline replies to the twenty chapter items, and counts each problem.

A citation is a problem when its section was not in the item's prompt, which covers a section of another version."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant, chapter_items
from lib.contract import PROBLEMS, format_problems, parse_reply, section_ids_by_version
from lib.prompts import P1_SHA256, P1_SYSTEM
from lib.stats import pct

assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)
items = chapter_items()
counts = {name: 0 for name in PROBLEMS}  # items with each problem, by its fixed name
items_with_problems = 0
answered = 0
answered_uncited = 0
citations = 0
citations_outside = 0  # cited IDs that were not in the item's prompt
for item in items:
    reply = assistant.reply(item)
    problems = format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=item.kind == "unanswerable")
    items_with_problems += bool(problems)
    for name in problems:
        counts[name] += 1
    if not reply.declined and reply.text:
        answered += 1
        answered_uncited += not reply.citations
    citations += len(reply.citations)
    citations_outside += sum(1 for cid in reply.citations if cid not in assistant.context_ids(item))
assert citations_outside == 0, "every citation must point at a section that was in its item's prompt"
# A zero rate shows nothing unless the check can fire: a 1.1 section cited for a 2.0 item must be flagged as unknown.
planted_item = next(item for item in items if item.version == "2.0")
planted = sorted(section_ids_by_version()["1.1"] - assistant.context_ids(planted_item))[0]
planted_problems = format_problems(parse_reply(f"Yes. [sec:{planted}]"), context_ids=assistant.context_ids(planted_item), expect_decline=False)
assert "unknown-section" in planted_problems, "the check must fire on a planted bad citation"

print(f"items: {len(items)}")
print(f"items with a format problem: {pct(items_with_problems, len(items))}")
for name in sorted(counts):  # sorted by problem name
    print(f"items with problem {name}: {counts[name]}")
print(f"answered items: {answered}")
print(f"answered items with no citation: {pct(answered_uncited, answered)}")
print(f"citations: {citations}")
print(f"citations to a section not in the prompt: {pct(citations_outside, citations)}")
```

Output:

```text
items: 20
items with a format problem: 0.0% (0 of 20)
items with problem answered-declined-expected: 0
items with problem declined-with-text: 0
items with problem empty: 0
items with problem no-citation: 0
items with problem unknown-section: 0
answered items: 15
answered items with no citation: 0.0% (0 of 15)
citations: 16
citations to a section not in the prompt: 0.0% (0 of 16)
```

The baseline gave 16 citations in total, and each one points at a section it was given. The output puts rates on that: citations to a section outside the prompt come to 0.0% (0 of 16), and answered items with no citation come to 0.0% (0 of 15).

**Working rule:** Run the citation check on every reply, and report its rates apart from the answer rate, since the two count different mistakes. A rate of zero is only a result if the check could have found a problem, so keep a fixture reply that cites outside its prompt, and run the check on it, to show the check still fires.

### Declines {#declines-in-code}

When the docs don't answer a question, the assistant must reply with the decline phrase, "The docs don't say.", and nothing else. When the docs do answer, it must not use that sentence. These are two different errors, so each gets its own rate. A false refusal is an answerable item declined, and a false answer is an unanswerable item answered. Version-dependent and ambiguous items are left out of both rates. A version-dependent item has a rate of its own, version-correctness, which [chapter 9](grounded-answers) builds. An ambiguous item is judged by the pass rate alone, since a reply's text alone can't show which version or which reading was meant.

Both decline errors are counted over the baseline's replies.

Example `check-declines` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Counts the two decline errors over the baseline replies: answerable items declined, and unanswerable items answered.

Version-dependent and ambiguous items are left out of both rates: a version-dependent item is scored by its own rate,
version-correctness, and an ambiguous item has no rate yet. The last lines name any excluded item the baseline declined."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant, chapter_items
from lib.prompts import P1_SHA256, P1_SYSTEM
from lib.stats import pct

assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)
items = chapter_items()
answerable = [item for item in items if item.kind == "answerable"]
unanswerable = [item for item in items if item.kind == "unanswerable"]

false_refusals = [item.id for item in answerable if assistant.reply(item).declined]  # already in id order
false_answers = [item.id for item in unanswerable if not assistant.reply(item).declined]
print(f"answerable items: {len(answerable)}")
print(f"false refusal rate (answerable items declined): {pct(len(false_refusals), len(answerable))}")
for item_id in false_refusals:
    print(f"false refusal: {item_id}")
print(f"unanswerable items: {len(unanswerable)}")
print(f"false answer rate (unanswerable items answered): {pct(len(false_answers), len(unanswerable))}")
for item_id in false_answers:
    print(f"false answer: {item_id}")
print(f"items not counted here (version-dependent and ambiguous): {len(items) - len(answerable) - len(unanswerable)}")
excluded = [item for item in items if item.kind in ("version-dependent", "ambiguous")]
declined_excluded = [item.id for item in excluded if assistant.reply(item).declined]  # the decline is not counted
print(f"declined, not counted here: {', '.join(declined_excluded) or 'none'}")
```

Output:

```text
answerable items: 10
false refusal rate (answerable items declined): 0.0% (0 of 10)
unanswerable items: 4
false answer rate (unanswerable items answered): 0.0% (0 of 4)
items not counted here (version-dependent and ambiguous): 6
declined, not counted here: q-0004
```

Among the answerable and unanswerable items, the baseline makes neither error. The false refusal rate is 0.0% (0 of 10) of the 10 answerable items. The false answer rate is 0.0% (0 of 4) of the 4 unanswerable ones, so every unanswerable item got the decline, `q-0015` among them.

The rates leave out 6 items, and the last line names the one of them that the baseline declined: q-0004. Item `q-0004` asks whether `${project}` is the right placeholder in 2.0, where it is. The baseline replied with the decline phrase, but the 2.0 docs answer it, in the `placeholders` section, so the decline is a wrong reply. The rate above doesn't count it as a false refusal, because that rate covers answerable items only. Version-correctness does count it as a miss.

**Working rule:** Keep the two decline rates apart, and leave version-dependent and ambiguous items out of both. A reply counts as a correct decline only when it is the decline phrase and nothing else, so a reply that gives the sentence and then an answer counts as the error it is. Read each false refusal by hand before you change the prompt, since one of them may point at a real gap in the docs. Read each decline on a version-dependent item too, since the rate doesn't count those.

## One judge {#one-judge}

Citations and declines check the shape of a reply. They can't check whether the reply is right, and that's the question the team cares about. For that, the book uses a judge: a model that reads the question, a written reference answer and the assistant's reply, and returns PASS or FAIL. The judge here is Qwen3-4B-Instruct-2507, the same model that wrote the replies. A judge and an assistant that are one model can share blind spots, and then the judge misses the errors the assistant makes for the same reason. This book can't test for that with this setup. What it does instead is check the same model, under a calibration rubric, against public human ratings, and say what carrying those rates to this judge assumes, in [chapter 6](calibrate-the-judge).

The judge's rules are short, and they're worth reading in full. A reply is correct only if it says what the reference says. Other words are fine, and so is extra detail that doesn't contradict the reference. A reply that says something the reference doesn't support, or contradicts it, fails. If the reference has more than one reading, the reply must answer every one of them, and a partial answer fails. If the reference is "none", the docs don't answer, and the only correct reply is the decline. A reply that gives an answer and also the decline phrase fails. The judge writes its verdict on the first line and a short reason after it. The prompt asks for one sentence, and the judge doesn't always stop there.

Here is the judge's verdict on `q-0005`, the near miss. The assistant's reply to it says that 2.0 does not read `sablefile.lock`, and points vaguely at migration.

Example `judge-one-answer` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Reads the reference-guided judge's verdict on the baseline reply to q-0005, a planted near-miss item, from its recording.

The judge sees the question, the written reference and the reply; the first line of its reply is the verdict."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant, chapter_items
from lib.contract import parse_verdict
from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA256
from lib.replay import ReplayModel

assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)
item = next(item for item in chapter_items() if item.id == "q-0005")
print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")
print(f"reference answer: {item.reference_answer}")
reply = assistant.reply(item)
print(f"assistant reply: {reply.text}")

judge = ReplayModel("judge-answers-baseline", REF_SHA256)  # the prompt hash must match the judge's manifest
raw = judge.reply(item.id, 0)
verdict = parse_verdict(raw)
assert verdict is not None, "the judge's first line must be a verdict"
print(f"judge verdict: {verdict}")
reason = [line.strip() for line in raw.splitlines() if line.strip()][1:]  # the line after the verdict
print(f"judge reason: {reason[0] if reason else 'none'}")
```

Output:

```text
question q-0005 (version 2.0, answerable, planted: near-miss-term): Does sablefile 2.0 still read sablefile.lock in my project root?
reference answer: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
assistant reply: No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. The first 2.0 command that reads an older lock file handles it as described in lock-migration. [sec:install-upgrade]
judge verdict: FAIL
judge reason: The response incorrectly states that sablefile 2.0 "does not read `sablefile.lock`" and that it "moves the lock file to a new place," which contradicts the reference answer that says sablefile 2.0 reads the old lock file as a fallback and only reads the new file once it exists. The reference explicitly states that it reads the old file as a fallback and warns users, not that it moves or stops reading it.
```

The reference answer says something different: 2.0 reads the old file only as a fallback, when no new file exists, and it warns you when it does. The judge's verdict is FAIL, and its reason says the reply contradicts the reference. The 2.0 migration section of the docs says what the reference says, so on this item the judge is right. One correct verdict doesn't show that the judge is reliable, though.

That is why the judge needs checking against people. A judge that agrees with your own labels is not thereby calibrated. Shankar and colleagues' [*Who Validates the Validators?*](https://arxiv.org/abs/2404.12272) (2024) is a study of nine participants, and it found that people refine their grading criteria while they grade outputs, so the criteria they settle on change as they read more outputs. A rubric written once can drift from what its graders come to mean. [Chapter 5](a-judge-with-a-rubric) says how to write one down, and that is worth doing for a reason of its own: a written rubric is what a second reader can check a verdict against, and what later verdicts can be compared with. The check against human ratings is in [chapter 6](calibrate-the-judge).

**Working rule:** Give the judge a written reference for every item, and read its FAIL verdicts as closely as its PASS verdicts before you trust a rate built on them. Don't count agreement with your own labels as calibration. Until a judge has been checked against human ratings, report its rate as the judge's rate, not as a measure of the assistant's quality.

## A pass rate and its interval {#a-pass-rate-and-its-interval}

A pass rate is a count divided by a denominator, and the denominator decides what the rate means. The same rate on a much larger set is a different claim, and a set of twenty moves a lot when one item changes. Here the denominator is every item in the set, and a correct decline counts as a pass.

```
pass rate = passed items / items
```

An interval answers a different question from the pass rate: how far would the rate move if the set had been another twenty items drawn the same way? The method is a percentile bootstrap. The bootstrap itself is Efron's: his 1979 paper [*Bootstrap Methods: Another Look at the Jackknife*](https://doi.org/10.1214/aos/1176344552), in the *Annals of Statistics*, set out resampling as a way to estimate how much a statistic varies. The percentile bootstrap is the simplest interval built on resampling, the middle 95% of the resampled rates. The procedure draws items from the set with replacement, without regard to kind, so one draw can hold any mix of kinds. It computes the pass rate of each draw, and repeats the draws many times. The interval shows how much twenty items could say.

Compute the rate and its interval for the baseline, then break the rate down by kind.

Example `pass-rate-interval` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The baseline's pass rate over the twenty chapter items, judged by the reference-guided judge, with its 95% interval.

Every item is in the denominator, and a correct decline counts as a pass. The interval resamples items."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import chapter_items
from lib.judging import passed, verdicts
from lib.stats import bootstrap_mean, interval_pct, pct

items = chapter_items()
found = verdicts("judge-answers-baseline", [item.id for item in items])  # keyed in sorted id order
scores = passed(found)  # 1.0 for PASS, 0.0 for FAIL, in the same order
print(f"items: {len(items)}")
print(f"pass rate: {pct(int(scores.sum()), len(items))}")
lo, hi = bootstrap_mean(scores)  # level 0.95 for one comparison; seed SEED, 2,000 resamples
print(f"95% interval of the pass rate: {interval_pct(lo, hi)}")

kind_of = {item.id: item.kind for item in items}
for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):
    ids = [item_id for item_id in found if kind_of[item_id] == kind]
    passes = sum(1 for item_id in ids if found[item_id] == "PASS")
    print(f"pass rate, {kind}: {pct(passes, len(ids))}")
```

Output:

```text
items: 20
pass rate: 85.0% (17 of 20)
95% interval of the pass rate: [70.0%, 100.0%]
pass rate, answerable: 90.0% (9 of 10)
pass rate, unanswerable: 100.0% (4 of 4)
pass rate, version-dependent: 75.0% (3 of 4)
pass rate, ambiguous: 50.0% (1 of 2)
```

The baseline's pass rate is 85.0% (17 of 20). Its 95% interval is [70.0%, 100.0%]. The interval sits the same distance either side of the rate. Its upper end sits at the ceiling for any rate. It reaches the ceiling because more than 2.5% of the draws contain no failing item, so the top of the middle 95% is that ceiling. That means the interval can't rule out an assistant that never fails, which is what the single number hides.

Answerable items pass at 90.0% (9 of 10), unanswerable items at 100.0% (4 of 4), version-dependent items at 75.0% (3 of 4), and ambiguous items at 50.0% (1 of 2). The three failures fall in three different kinds. The one answerable failure is `q-0005`, read above. The one version-dependent failure is the decline on `q-0004`, read above. The one ambiguous failure is `q-0019`, which the judge fails under the baseline's reply, as the example in the next section shows.

Only the answerable kind has more than a few items: 10 of them. Each other kind holds 4 items or fewer, so their lines are notes about a few items, not results.

The interval covers only the spread that comes from drawing items. It says nothing about whether the twenty items represent the questions real users ask, and no interval can say that. [Chapter 3](build-the-test-set) covers what a set must contain before a score on it means anything, and why this one is written from the docs rather than sampled from users.

**Working rule:** Report every pass rate with its denominator, its interval and its rate for each kind of item. Treat a kind with only a few items as a note, not a result, and say how many items it rests on.

## Two prompts {#two-prompts}

The team's proposed prompt is the change to measure. P1 is the baseline prompt. P2 keeps the same ground rules and changes the form of the reply: it asks the model to quote first the sentence its answer relies on, then give the answer, with the citation on each line. The question is whether the quote helps. Two pass rates over different items mostly measure the items, so the comparison has to pass the same items through both prompts. Each item then gets two verdicts, and the difference is taken item by item. That's a [paired comparison](#paired-comparisons).

Both prompts run over the same twenty items, and the output reports the difference with its interval.

Example `compare-two-prompts` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Compares the baseline prompt (P1) with prompt-b (P2) on the twenty chapter items: each pass rate, the paired difference, its interval and n.

The word 'better' is used only when the whole interval is above zero, and 'worse' only when it is below zero. A difference
whose interval includes zero is reported as no difference detected."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import chapter_items
from lib.judging import paired
from lib.stats import interval_points, pct

result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])
first, second = result.first, result.second  # 1.0 or 0.0 per item, in id order
print(f"P1 (baseline) pass rate: {pct(int(first.sum()), result.n)}")
print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), result.n)}")
print(f"items that P2 passes and P1 fails: {int(((second == 1) & (first == 0)).sum())}")
print(f"items that P1 passes and P2 fails: {int(((second == 0) & (first == 1)).sum())}")
print(f"paired items (n): {result.n}")
print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")
print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")
if result.diff_lo > 0:
    reading = f"P2 is better at n = {result.n}"
elif result.diff_hi < 0:
    reading = f"P2 is worse at n = {result.n}"
else:
    reading = f"no difference detected at n = {result.n}"
print(f"reading: {reading}")
# The two prompts tie: the same number of items pass under each, so the pass rates and the reading say the same thing.
assert int(first.sum()) == int(second.sum()), "the two prompts must tie: the same pass count under each"
```

Output:

```text
P1 (baseline) pass rate: 85.0% (17 of 20)
P2 (prompt-b) pass rate: 85.0% (17 of 20)
items that P2 passes and P1 fails: 1
items that P1 passes and P2 fails: 1
paired items (n): 20
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
reading: no difference detected at n = 20
```

On this set the two prompts tie. P1 passes 85.0% (17 of 20) of the items, and P2 passes 85.0% (17 of 20).

The paired view shows what sits under the tie. P2 passes 1 item that P1 fails, and P1 passes 1 item that P2 fails. These are the judge's verdicts, not facts about the prompts. The first is `q-0019`, and the judge's PASS on it breaks the judge's own rule 3. The difference, P2 minus P1, is +0.0 points, and its interval is [-15.0, +15.0] points. The interval includes zero, so at this size the example reports [no difference detected](#paired-comparisons).

Here the judge's verdicts on `q-0019` are read under both prompts, and the rule the PASS breaks is printed beside them.

Example `judge-gets-one-wrong` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Shows one judge verdict that breaks the judge's own rule: on q-0019, the judge passes P2's reply, which answers only one of the
two readings in the reference. The rule it breaks is rule 3 of the judge's prompt, printed below from that prompt.

The hand verdict is a person's reading, written into this file. It is printed beside the judge's verdict and is the only
number here that is not a recorded one; the count of passes with it is taken from the judge's own verdicts."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant, chapter_items
from lib.judging import verdicts
from lib.prompts import P1_SHA256, P1_SYSTEM, P2_SHA256, P2_SYSTEM, REF_SYSTEM
from lib.replay import ReplayModel

ITEM_ID = "q-0019"
HAND = {"q-0019": "FAIL"}  # read by hand against rule 3: the reply answers reading 1 and not reading 2

item = next(item for item in chapter_items() if item.id == ITEM_ID)
print(f"question {item.id} (version {item.version}, {item.kind}, planted: {item.planted_why}): {item.question}")
print(f"reference answer: {item.reference_answer}")
print(f"P2 reply: {Assistant('assistant-prompt-b', P2_SYSTEM, P2_SHA256).reply(item).text}")

raw = ReplayModel("judge-answers-prompt-b", None).reply(item.id, 0)  # the judge's recorded reply to P2's answer
print(f"judge verdict on P2's reply: {verdicts('judge-answers-prompt-b', [item.id])[item.id]}")
reason = [line.strip() for line in raw.splitlines() if line.strip()][1:]  # the line after the verdict
print(f"judge reason: {reason[0] if reason else 'none'}")
print(f"judge verdict on P1's reply to the same item: {verdicts('judge-answers-baseline', [item.id])[item.id]}")
print(f"hand verdict on P2's reply, under rule 3: {HAND[item.id]}")
rule_3 = next(line for line in REF_SYSTEM.splitlines() if line.startswith("3. "))
print(f"rule it breaks: {rule_3}")

ids = [item.id for item in chapter_items()]
judged = verdicts("judge-answers-prompt-b", ids)
passes = sum(verdict == "PASS" for verdict in judged.values())
by_hand = passes - sum(1 for item_id in HAND if judged[item_id] == "PASS" and HAND[item_id] == "FAIL")
print(f"P2 passes, as judged: {passes} of {len(ids)}")
print(f"P2 passes, with {ITEM_ID} read by hand: {by_hand} of {len(ids)}")
```

Output:

```text
question q-0019 (version 2.0, ambiguous, planted: ambiguous-target): If I edit the file and then run check, will it fail?
reference answer: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
P2 reply: Quote: "Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what drift looks like, open a generated file, change one line, and run the check again. The check prints `DRIFT` and the path of the file." [sec:quickstart-check]
Answer: Yes, if you edit the file and then run check, it will print `DRIFT` and the path of the file, indicating a drift [sec:quickstart-check].
judge verdict on P2's reply: PASS
judge reason: The response correctly identifies that editing a generated file and running check will result in a DRIFT message and path output, aligning with Reading 1 of the reference. It does not contradict or omit any part of the reference.
judge verdict on P1's reply to the same item: FAIL
hand verdict on P2's reply, under rule 3: FAIL
rule it breaks: 3. If the reference has several readings, the response must answer every reading. A partial answer is incorrect.
P2 passes, as judged: 17 of 20
P2 passes, with q-0019 read by hand: 16 of 20
```

Read this output as a record of the judge's verdicts. The reference for `q-0019` has two readings, and P2's reply answers only the first. Rule 3 of the judge's prompt says a partial answer fails, and the judge fails P1's reply to the same item, which also answers only the first reading. On P2's reply it passes, and its reason says the reply covers the reference and omits nothing. Read by hand, P2 passes 16 of 20 items, not 17 of 20. The tie is one judge error away from not being one. That is why the rates here are the judge's rates, and why [chapter 6](calibrate-the-judge) checks the judge against public human ratings.

**Working rule:** Compare prompts on the same items, paired, and report the difference with its interval. Call one prompt better only when the whole interval lies above zero, and worse only when it lies below zero. When the interval includes zero, say that no difference was detected at this size, which is not the same as saying the prompts are equal.

## What twenty questions can't show {#what-twenty-questions-cant-show}

This last example takes the same two prompts and asks what the twenty items can show about them. Twenty items turn out to say very little about these two prompts.

Example `twenty-cannot-tell` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Shows that twenty items cannot tell the two prompts apart: the interval of the paired difference includes zero.

The assert is the chapter's claim, checked against the numbers; the example prints no verdict word."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import chapter_items
from lib.judging import paired
from lib.stats import interval_points

result = paired("judge-answers-baseline", "judge-answers-prompt-b", [item.id for item in chapter_items()])
print(f"paired items (n): {result.n}")
differing = int((result.first != result.second).sum())
print(f"items where the two verdicts differ: {differing}")
print(f"difference, P2 minus P1: {100 * result.difference:+.1f} points")
print(f"95% interval of the difference: {interval_points(result.diff_lo, result.diff_hi)}")
print(f"interval width: {100 * (result.diff_hi - result.diff_lo):.1f} points")
# Zero inside the interval is the claim: no difference that twenty items can show.
assert result.diff_lo <= 0.0 <= result.diff_hi, "the interval of the difference must include zero"
# Two items carry the whole comparison: the pass rates differ only where these two verdicts differ.
assert differing == 2, "exactly two items must have differing verdicts"
```

Output:

```text
paired items (n): 20
items where the two verdicts differ: 2
difference, P2 minus P1: +0.0 points
95% interval of the difference: [-15.0, +15.0] points
interval width: 30.0 points
```

Only 2 items get different verdicts from the two prompts, so the difference can move only in coarse steps, one item at a time. The interval of the difference is 30.0 points wide. An assert in the code requires the interval to include zero, so the claim fails loudly if the numbers ever stop supporting it. An interval that wide can detect only a large change, and a smaller change that still matters to users can hide inside it.

In production, the same mistake shows up in three forms: trusting a number that can't carry the claim made for it. A team picks the prompt with the higher number on the twenty items it wrote first, and the report leaves out the interval, so nobody sees how little the number holds. A judge gets trusted because its verdicts look sensible, and nobody reads its FAIL reasons against the reference. A set of recorded replies, kept after the sections the assistant sees have changed, measures the old system while its numbers keep looking fine. You notice the first when a report carries an interval beside each rate. You notice the second when someone reads the judge's FAIL reasons against the references on every run. The third shows only if the recording also hashes the docs it was given. This book's manifests hash the prompt and the test set, so a prompt change is refused on replay, but a change to the sections goes unseen.

Two later chapters take the next steps. [Chapter 3](build-the-test-set) builds the larger set this chapter lacks, and [chapter 8](intervals-and-sample-size) works out how many items a difference needs before it can show.

The chapter made these decisions:

- Every question names its version, and its answer comes from that version's docs only.
- This chapter's set is twenty items, read in full, with the planted items labelled.
- Each recording is made once per model, prompt and docs, and replayed for every comparison.
- Citations and declines are checked in code before any judge reads a reply.
- The judge grades against a written reference, and its rate counts as the judge's rate until calibration checks it.
- A pass rate comes with its denominator, its interval and its rate for each kind of item.
- Prompts are compared on the same items, and only an interval wholly on one side of zero makes one of them better or worse.

**Working rule:** Report the interval beside every comparison. Twenty items can find broken answers; they can't rank two prompts.

# Read the replies to find what to measure

By the end of this chapter you can read a set of replies from an assistant, sort what is wrong with each one into a failure mode, and say which of those failures a check can see.

Chapter 1 ended on a limit: twenty items can find broken answers, but they can't rank two prompts. The assistant in this book answers from three versions of sablefile's docs, 1.0, 1.1 and 2.0, and a question about one version can get an answer that is right for another. A pass rate counts every such failure the same way and names none of them. A team that measures only with a pass rate therefore writes its checks for the failures it expects, and the failures it doesn't expect go unchecked. Reading the replies first is how you find the second kind. Chapter 1's interval says how far a score can be trusted. This chapter asks what each variant gets wrong, which a score can't say.

Reading before counting has support outside this book. [Chapter 1](measure-before-you-change) cited Shankar and colleagues' [*Who Validates the Validators?*](https://arxiv.org/abs/2404.12272) (2024) for the point that people refine their grading criteria while they grade outputs. That paper is mainly a system, EvalGen, and its user study of nine practitioners is the part this chapter relies on. The drift the study records happened while people graded outputs. That a change to the pipeline can make the criteria drift is the authors' discussion, not a finding of the study. It is one study, so this chapter treats it as a caution, not as a result the book measures.

This chapter uses the 80 dev items, the split of the test set this book works with while it develops its measures. It reads only the dev items, so nothing here is tuned to the held-out items, and the held-out decline counts appear in [chapter 10](declining-when-the-docs-dont-say). The reading produces three things: a note on each reply, a count of the defects those notes fall into, and a candidate check for each defect. Every reply here is a recording made once for each variant and replayed by the examples, so the reading is the same on every run. Quotes of replies come from the recorded outputs, and the reference answers are printed beside their items, so any line the prose quotes from them can be found in the example's output. The notes sit in a data file that the tally example reads.

Throughout, a reply counts as right when it gives the reference answer, or when it declines because the reference says the docs don't say. Its citations are checked on their own, by the citation check in [chapter 1](measure-before-you-change#citations-in-code) and the checks in [chapter 4](checks-in-code). Anything else is a failure, and the rest of the chapter says which kind. The same steps work on a replay file of your own. Group its replies by kind, read each one against its reference, code each by its main defect, and write one candidate check per mode. This chapter does that for the sablefile replies, so you can follow each step on data you can open.

This chapter doesn't choose the checks or write a judge's rubric, and it doesn't estimate how often each mode happens in real use. Those come in later chapters, and each one relies on the modes named here.

## What the replies look like {#what-the-replies-look-like}

Three variants of the assistant answered the same dev items, and every comparison in this chapter sets one variant's replies beside another's. The baseline is the setup chapter 1 measured. It retrieves sections only from the version the question asks about, so its context holds one version's docs. The unfiltered variant drops the baseline's version filter. Its context can hold sections from all three versions, each marked with its version, so the model can see a mismatch and can also copy the wrong version's text. The second-model variant keeps the baseline's prompt and hands the answer to a smaller model, Qwen3 with 0.6 billion parameters in an 8-bit build. The prompt is the same, so a difference from the baseline comes from the model, not the prompt.

Each dev item has a kind, and the kind says what a right reply looks like. An [answerable item](#the-twenty-items) is one the docs answer. An [unanswerable item](#the-twenty-items) asks for something the docs don't say, and the right reply is a decline in the words the assistant was given: "The docs don't say." A version-dependent item has an answer that changes with the version asked about, so the same question asked of two versions can have two right answers. An ambiguous item can be read two ways, and the docs answer each reading. Some items are [planted](#the-twenty-items), written to catch one specific mistake. The output gives each planted item its reason, which is one of a false premise, a near-miss term, a version swap, a stale answer or an ambiguous target. A reply that fails a planted item shows whether the assistant made the mistake that item was written for.

Reading one item takes three steps. Start with the question and its version, since a reply is right only for the version it names. Then read the reference answer, which is the author's statement of what a right reply says. Then read the three replies against it, one at a time. Only after that does a reader group the replies or count them.

The example that follows is where the chapter starts. It prints every dev item with its reference answer and each variant's reply, grouped by kind and sorted by ID, so the three replies to one question sit on adjacent lines. Read across an item's lines, not down one variant's column. The variants fail in different ways on the same question, and that difference is what to see. Each reply is cut to its first 120 characters, and its line says whether the reply declined and what it cited. A reply that holds the decline phrase inside other text is marked as a phrase, not a decline, so you can see where the parser draws the line.

Example `read-replies-dev` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Prints the dev items, grouped by kind and sorted by ID, with each item's reference answer and the baseline,
unfiltered and second-model replies.

The reference is printed in full under its item, since the reading compares each reply with it. Each reply is one line:
whether it declined, its citations, and its first 120 characters. The reader's job is to set these side by side and
see which failure each reply shows (the codes are in data/failure-codes.json)."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.contract import DECLINE, parse_reply
from lib.prompts import P1_SHA256
from lib.replay import ReplayModel
from lib.testset import load_items

RECORDINGS = {  # the three variants, in the order the chapter discusses them
    "baseline": "assistant-baseline",
    "unfiltered": "assistant-unfiltered",
    "second-model": "assistant-second-model",
}
KINDS = ("answerable", "unanswerable", "version-dependent", "ambiguous")  # plan section 3.1 order
SHOWN = 120  # characters of each reply printed; the full text is in the recording

# The prompt hash is checked here, so a recording made from another prompt fails to load.
models = {name: ReplayModel(recording, P1_SHA256) for name, recording in RECORDINGS.items()}
items = load_items(split="dev")  # dev only: the held-out items are not read in this chapter (plan section 3.4)

for kind in KINDS:
    group = [item for item in items if item.kind == kind]  # load_items already sorts by ID
    print(f"## {kind}: {len(group)} dev items")
    for item in group:
        planted = f"planted ({item.planted_why})" if item.planted else "not planted"
        print(f"{item.id} | version {item.version} | {planted} | {item.question}")
        reference = item.reference_answer or "none: the item is unanswerable, so the right reply declines"
        print(f"  reference: {reference}")
        for name in RECORDINGS:
            reply = parse_reply(models[name].reply(item.id, 0))
            if reply.declined:
                mark = "declined"
            elif DECLINE in reply.text:
                # The phrase is there, but with other text: the parser does not call that a decline.
                mark = "decline phrase with other text"
            else:
                mark = f"cites {' '.join(reply.citations)}" if reply.citations else "no citation"
            text = reply.text.replace("\n", " ")  # a reply may hold line breaks; one line keeps the output readable
            more = "..." if len(text) > SHOWN else ""
            print(f"  {name} ({mark}): {text[:SHOWN]}{more}")
```

Output:

```text
## answerable: 40 dev items
q-0005 | version 2.0 | planted (near-miss-term) | Does sablefile 2.0 still read sablefile.lock in my project root?
  reference: Only as a fallback. If the new file is missing and an old sablefile.lock is present, a 2.0 command reads the old file but warns you about it. Once the new file exists, 2.0 reads only that one, and sablefile never deletes the old file.
  baseline (cites install-upgrade): No, sablefile 2.0 does not read `sablefile.lock` in the project root. Version 2.0 moves the lock file to a new place. Th...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0006 | version 2.0 | not planted | Is there a way to make check exit with status 0 even when a file has drifted in 2.0?
  reference: Yes. Pass --no-fail to check. It still prints each DRIFT line, but it exits with status 0, so a script can report drift without failing. The flag changes nothing except the exit status.
  baseline (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in 2.0. Use the `--no-fail` flag wit...
  unfiltered (cites no-fail-flag): Yes, there is a way to make `check` exit with status 0 even when a file has drifted in version 2.0. Use the `--no-fail` ...
  second-model (cites no-fail-flag): The `--no-fail` flag is used to make the check exit with status 0 even when files have drifted. [sec:no-fail-flag]
q-0007 | version 1.1 | not planted | In 1.1, will an ignore entry of docs/**/*.tmp catch a file two folders down, like docs/x/y/a.tmp?
  reference: Yes. Version 1.1 adds the double-star pattern, which matches zero or more folders. So docs/**/*.tmp matches docs/x/y/a.tmp as well as files directly in docs.
  baseline (cites ignore-recursive): Yes, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:ignore-recursive...
  unfiltered (cites ignore-recursive): Yes, in version 1.1, an ignore entry of `docs/**/*.tmp` will catch a file two folders down, like `docs/x/y/a.tmp`. [sec:...
  second-model (no citation): The docs section does not allow for an ignore entry of docs/**/*.tmp to catch a file two folders down, as per the rules....
q-0008 | version 2.0 | not planted | In 2.0, is there a way to re-include something that an earlier ignore line had excluded?
  reference: Yes, with a leading exclamation mark. A line that starts with ! negates an earlier match, so a later line can restore a file that an earlier line ignored.
  baseline (cites ignore-syntax): Yes, in 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to negate an e...
  unfiltered (cites ignore-syntax): Yes, in version 2.0, you can re-include something that an earlier ignore line had excluded by using a leading `!` to neg...
  second-model (declined): The docs don't say.
q-0009 | version 1.0 | not planted | What does one line of the 1.0 lock file hold?
  reference: The SHA-256 hash of a generated file's content, then two spaces, then that file's path. The 1.0 lock has no mode field.
  baseline (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, two spaces, then the path. [sec:lockfile]
  unfiltered (cites lockfile): One line of the 1.0 lock file holds the SHA-256 hash of the file's content, then the file's mode, then the path. [sec:lo...
  second-model (cites lockfile): The lock file holds the SHA-256 hash of the file's content, two spaces, and the path. [sec:lockfile]
q-0010 | version 2.0 | not planted | Will syncing the same content twice in 2.0 produce a different lock file each time?
  reference: No. The 2.0 lock file is JSON with no timestamps, so the same content always gives back an identical file.
  baseline (cites lockfile): No, syncing the same content twice in 2.0 will not produce a different lock file each time. The lock file has no timesta...
  unfiltered (cites lockfile): No, syncing the same content twice in 2.0 will produce an identical lock file each time because the lock file has no tim...
  second-model (declined): The docs don't say.
q-0011 | version 1.1 | not planted | How many times does 1.1 retry a remote template fetch that keeps failing, and can a flag change that?
  reference: Five times, before the command gives up. In 1.1 that number is fixed, and no flag or setting can change it.
  baseline (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
  unfiltered (cites remote-retries): In version 1.1, sablefile retries a remote template fetch up to 5 times before giving up. The count is fixed and no flag...
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0012 | version 2.0 | not planted | I want sync to retry a slow remote template only twice in 2.0. What do I pass?
  reference: Pass --retries 2 to sync. The flag sets how many times a failed fetch is retried, in place of the default of five, and it only affects the command it is given to.
  baseline (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  unfiltered (cites retries-flag): Pass `--retries 2` to sync. [sec:retries-flag]
  second-model (cites retries-flag): `--retries 2` to pass. [sec:retries-flag]
q-0013 | version 1.0 | not planted | In 1.0, which table in sablefile.toml holds the values a placeholder can name?
  reference: The [values] table. Each name in it can be used as a placeholder, and a placeholder name in a template must match a key in that table.
  baseline (cites config-values): The `[values]` table holds the values a placeholder can name. [sec:config-values]
  unfiltered (cites config-values): The `[values]` table in sablefile.toml holds the values a placeholder can name. [sec:config-values]
  second-model (cites config-values): The values table holds the values a placeholder can name. [sec:config-values]
q-0014 | version 1.1 | not planted | What does sablefile init put into a new sablefile.toml in 1.1?
  reference: A [values] table with one entry, project, set to the folder's name, and no [[file]] tables. init writes no other file.
  baseline (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  unfiltered (cites init-command): `sablefile init` creates `sablefile.toml` in the folder where you run it, or in the folder you name. The new file has a ...
  second-model (declined): The docs don't say.
q-0051 | version 2.0 | planted (near-miss-term) | My 1.x ignore file uses shell globs, so does *.log only match files in the top folder in 2.0?
  reference: No. Shell glob rules no longer apply in 2.0. Entries use gitignore syntax, so *.log matches at any depth, and a trailing slash limits an entry to directories.
  baseline (cites ignore-syntax): In 2.0, entries use gitignore syntax. A line such as `*.log` matches at any depth. Therefore, `*.log` matches files in a...
  unfiltered (cites ignore-syntax): No, in 2.0, `*.log` matches at any depth, not just in the top folder. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0052 | version 2.0 | planted (near-miss-term) | I have {{project}} all through my templates from 1.x. Is that still valid in 2.0?
  reference: No. In 2.0 each placeholder is written as a dollar sign and braces, ${project}. The double-brace form is an error: rendering fails, sync reports the file and line, and it exits with status 1.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0053 | version 2.0 | planted (stale-answer) | I have an old sablefile.lock from 1.1. When I first run sync --apply under 2.0, what happens to it?
  reference: 2.0 writes a new .sablefile/lock.json holding each file's hash and, where the old file recorded one, its mode. The old file is not deleted, so you remove it yourself. A warning is printed when 2.0 reads the old file.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (cites lock-migration): The old `sablefile.lock` file from version 1.1 is not written to the new lock file, so it remains unchanged. [sec:lock-m...
q-0054 | version 2.0 | planted (stale-answer) | Does the 2.0 lock keep each generated file's mode, and what happens to modes from an old lock?
  reference: Yes. 2.0 records each generated file's mode in .sablefile/lock.json. When a 2.0 command migrates an older lock, it keeps each mode that the old file has.
  baseline (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  unfiltered (cites lock-mode lock-migration): Yes, the 2.0 lock keeps each generated file's mode, the Unix permission bits such as `0644`, in `.sablefile/lock.json` [...
  second-model (cites lock-mode lock-migration): The 2.0 lock keeps each generated file's mode, as specified in [sec:lock-mode]. When a lock file is migrated from `sable...
q-0055 | version 2.0 | planted (stale-answer) | How do I make sync retry a remote template fetch fewer times in 2.0?
  reference: Pass --retries N with the number you want, for example --retries 1. The flag sets the retry count for sync and diff; without it the default is five.
  baseline (cites retries-flag): Use the `--retries N` flag to set the number of times `sync` retries a failed fetch of a remote template, where N is a w...
  unfiltered (cites retries-flag): To make `sync` retry a remote template fetch fewer times in version 2.0, use the `--retries N` flag where N is a whole n...
  second-model (cites retries-flag): The `--retries` flag sets how many times `sync` and `diff` retry a failed fetch of a remote template, in place of the de...
q-0056 | version 1.0 | not planted | Does sablefile check need a network connection to run in 1.0?
  reference: No. check does not fetch remote templates or render templates, so it works without a network.
  baseline (cites check-command): No, sablefile check does not need a network connection to run. [sec:check-command]
  unfiltered (cites remote-fetch-failure): Yes, sablefile checks need a network connection to run in 1.0, as it requires fetching remote templates, and if a remote...
  second-model (declined): The docs don't say.
q-0057 | version 2.0 | not planted | Will check pick up a change that someone made on a remote template's server?
  reference: No. check never fetches remote templates, so only the next sync's output and the output of diff reflect that change.
  baseline (cites remote-retries): No, `check` does not see changes on a remote template's server. [sec:remote-retries]
  unfiltered (cites remote-retries): No, `check` does not fetch anything and never retries, so it will not pick up a change that someone made on a remote tem...
  second-model (cites remote-templates): Check does not see changes on a remote template's server. A change on the server affects what the next sync writes and w...
q-0058 | version 2.0 | not planted | If I run check --no-fail while drift is present, what does its printed output look like?
  reference: It still prints a DRIFT line and the path for each drifted file. The flag only changes the exit status to 0.
  baseline (cites quickstart-check): Run `sablefile check` after a sync. If no output appears, no generated file has drifted since the last sync. To see what...
  unfiltered (declined): The docs don't say.
  second-model (no citation): The check prints `DRIFT` and the path, and exits with status 1. A check with no drift exits with status 0. Pass `--no-fa...
q-0059 | version 1.1 | not planted | What is the mode that sablefile 1.1 stores for each generated file, and which command stores it?
  reference: The Unix permission bits of the file, such as 0644. sync saves it for every file it writes.
  baseline (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644` [sec:lock-mode]. ...
  unfiltered (cites lock-mode): The mode that sablefile 1.1 stores for each generated file is the Unix permission bits, such as `0644`. The `sync` comma...
  second-model (cites lock-mode lock-mode): The mode that sablefile 1.1 stores for each generated file is described in [sec:lock-mode]. The command that stores it i...
q-0060 | version 2.0 | not planted | What top-level keys does the 2.0 lock file JSON contain?
  reference: A format key, set to the string 2.0, and a files key that maps each generated path to its SHA-256 hash and mode.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (no citation): The 2.0 lock file JSON contains keys: `lock.json`.
q-0061 | version 1.0 | not planted | What is the usual way to preview sync's changes in 1.0 before writing?
  reference: Run sablefile diff. It shows the differences between each rendered template and its target on disk, without writing any file or changing the lock.
  baseline (cites diff-command): Run `sablefile diff` to preview sync's changes before writing. [sec:diff-command]
  unfiltered (cites sync-default): To see the changes before they are written, run `sablefile diff`, which shows the same differences without writing anyth...
  second-model (cites diff-command): The usual way to preview sync's changes in 1.0 before writing is to use `sablefile diff`. [sec:diff-command]
q-0062 | version 2.0 | not planted | Can I point sablefile init at a folder other than the current one in 2.0?
  reference: Yes. Give the folder as an argument, such as init path/to/folder. The folder must already exist, and a relative path is read from the folder where you run the command.
  baseline (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one. To do this, provide the path to the folder a...
  unfiltered (cites init-directory): Yes, you can point `sablefile init` at a folder other than the current one in 2.0. To do this, give the folder as an arg...
  second-model (declined): The docs don't say.
q-0063 | version 1.1 | not planted | If a target sits in folders that do not exist yet, what does sync do in 1.1?
  reference: It creates the missing folders before it writes the file, so no setup is needed. sync never removes a folder, even one that ends up empty.
  baseline (declined): The docs don't say.
  unfiltered (cites sync-create-dirs): If a target sits in folders that do not exist yet, sync creates the folder before it writes the file in version 1.1. [se...
  second-model (no citation): The lockfile is `sablefile.lock` in the project root. It holds one line for each generated file: the SHA-256 hash of the...
q-0064 | version 2.0 | not planted | After I delete a target from sablefile.toml, does sync --apply in 2.0 remove its now-empty folder?
  reference: No. A sync run never deletes a directory, which means any empty ones you leave behind remain.
  baseline (cites sync-create-dirs): No, sync --apply does not remove empty folders. sync never removes a folder, so empty folders you leave behind stay wher...
  unfiltered (cites sync-create-dirs): No, sync --apply in 2.0 does not remove its now-empty folder. sync never removes a folder, so empty folders you leave be...
  second-model (declined): The docs don't say.
q-0065 | version 1.0 | not planted | Where can template files sit inside a 1.0 project?
  reference: Anywhere in the project. The template key in each [[file]] table gives the path from the project folder, and a template is read only when a command renders it.
  baseline (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  unfiltered (cites templates-location): Template files can sit in any folder of the project. [sec:templates-location]
  second-model (no citation): Template files are ordinary text files, and they can sit in any folder of the project. The `template` key in a `[[file]]...
q-0066 | version 2.0 | not planted | Does sablefile run any code that is inside a template?
  reference: No. Rendering only substitutes placeholders. It executes nothing in the template, and the same template with the same values always gives the same output.
  baseline (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  unfiltered (cites template-render): No, sablefile does not run any code that is inside a template. Rendering does not run code in the template. [sec:templat...
  second-model (declined): The docs don't say.
q-0067 | version 1.1 | not planted | Can a [[file]] table in 1.1 take a template that lives at an https address?
  reference: Yes. The template key can name an https URL. sablefile fetches it when a command renders the template, and the fetched text is not copied into the project.
  baseline (cites remote-templates): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:remote-templates]
  unfiltered (cites config-files-table): Yes, a [[file]] table in 1.1 can take a template that lives at an https address. [sec:config-files-table]
  second-model (declined): The docs don't say.
q-0068 | version 2.0 | not planted | What happens when a remote fetch in 2.0 still fails after every retry?
  reference: The command halts, reporting the failing address in its error message, and ends with status 1. Targets it never wrote retain their previous lock records.
  baseline (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  unfiltered (cites remote-fetch-failure): If a remote template still cannot be fetched after the last retry, the command stops with an error that names the URL, a...
  second-model (declined): The docs don't say.
q-0069 | version 1.0 | not planted | What exit status does sync give in 1.0 when a template file is missing?
  reference: Status 1. sync cannot finish in that case, and its error message names the file that caused the failure.
  baseline (cites sync-exit-status): Sync exits with status 1 when it cannot finish, for example when a template file is missing or a remote template cannot ...
  unfiltered (cites sync-exit-status): sync exits with status 1 when a template file is missing. [sec:sync-exit-status]
  second-model (cites sync-exit-status): The exit status for sync in 1.0 when a template file is missing is 1. [sec:sync-exit-status]
q-0070 | version 1.1 | not planted | If sync in 1.1 exits with status 0, does that mean the generated text is what I want?
  reference: Not necessarily. Status 0 only shows that sync finished, so you should open the generated files and check them.
  baseline (cites sync-exit-status): No, if sync exits with status 0, it only means that sync completed successfully, not that the generated text is what you...
  unfiltered (cites sync-exit-status): No, if sync in 1.1 exits with status 0, it does not mean the generated text is what you want. A status of 0 shows that s...
  second-model (declined): The docs don't say.
q-0071 | version 1.0 | not planted | Where do I list paths for sablefile 1.0 to leave alone, and what happens to them?
  reference: In the ignore file .sablefileignore in the project root, with one entry per line. Sync leaves an ignored target unwritten, check and diff both pass over it, and no lock record is made for it.
  baseline (cites ignore-file): You list paths for sablefile 1.0 to leave alone in the ignore file, `.sablefileignore`, which sits in the project root. ...
  unfiltered (cites ignore-file): List paths for sablefile 1.0 to leave alone in the `.sablefileignore` file, which sits in the project root. Put one entr...
  second-model (no citation): The ignore file, `.sablefileignore`, sits in the project root and lists paths that sablefile must leave alone. A project...
q-0072 | version 2.0 | not planted | With --apply, does 2.0's sync write a target that my ignore file lists?
  reference: No. An ignored target is never written, and it gets no lock record either.
  baseline (cites ignore-file): No, with `--apply`, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  unfiltered (cites ignore-file): No, with --apply, 2.0's sync does not write a target that is listed in the ignore file. [sec:ignore-file]
  second-model (declined): The docs don't say.
q-0073 | version 2.0 | not planted | Which lines in a 2.0 ignore file are skipped as comments?
  reference: Lines that begin with a hash sign. Blank lines are also ignored.
  baseline (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Lines that start with `#` are comments. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0074 | version 1.1 | not planted | A check in 1.1 reports drift for a file I did not mean to change. What should I do first?
  reference: Start with diff, which shows how the file differs from its template. Keep wanted changes by moving them into the template before running sync; discard unwanted ones by reverting the file and running check again.
  baseline (cites troubleshoot-drift): Look at the change before you act. Run `sablefile diff` to see how the file differs from its template. [sec:troubleshoot...
  unfiltered (cites troubleshoot-drift): Look at the change before you act. [sec:troubleshoot-drift]
  second-model (cites check-command): Run `sablefile diff` to see how the file differs from its template. [sec:check-command]
q-0075 | version 2.0 | not planted | After installing the 2.0 archive, how do I confirm which version the shell is running?
  reference: Run sablefile --version. The output has the form sablefile 2.0, and that number should match the docs you are reading.
  baseline (cites install-check-version): Run `sablefile --version` to see the installed version. [sec:install-check-version]
  unfiltered (cites install-check-version): Run `sablefile --version` to see the installed version. The output has the form `sablefile 2.0`. Check that the number m...
  second-model (declined): The docs don't say.
q-0076 | version 1.0 | not planted | How do I remove sablefile 1.0 without losing my project's files?
  reference: Run python3 -m pip uninstall sablefile. That removes the command and its package, and leaves the project's own files alone, including the config, templates, outputs and lock.
  baseline (cites install-uninstall): To remove sablefile 1.0, run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package....
  unfiltered (cites install-uninstall): Run `python3 -m pip uninstall sablefile`. This removes the `sablefile` command and its package. It does not change your ...
  second-model (declined): The docs don't say.
q-0077 | version 2.0 | not planted | I installed sablefile but my shell says the command is not found. What should I check in 2.0?
  reference: Ask pip where it put the package, using pip show sablefile from that very same shell. If a location comes back yet the command fails, the scripts folder may be missing from your path.
  baseline (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  unfiltered (cites troubleshoot-command-not-found): If the shell reports that `sablefile` is not found, the command is not installed in the environment that is active. Run ...
  second-model (declined): The docs don't say.
q-0078 | version 1.1 | not planted | Does upgrading sablefile to a newer 1.x archive change any file in my project?
  reference: No. The upgrade writes no project file. Read the changelog in the newer version's folder first, since it lists each section whose behaviour changed.
  baseline (cites install-upgrade): No, installing a new version of sablefile does not write any project file. [sec:install-upgrade]
  unfiltered (cites install-upgrade): No, installing a new version does not write any project file. [sec:install-upgrade]
  second-model (declined): The docs don't say.
q-0079 | version 2.0 | not planted | After a 2.0 sync --apply, how do I confirm the template placeholder was filled in?
  reference: Open the target, such as README.md, and look for the project name in the spot where the placeholder stood.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0080 | version 1.0 | not planted | What is the difference between a placeholder and a value in sablefile 1.0?
  reference: A placeholder is a name in a template that sync replaces. A value is the string in the [values] table that the placeholder's name points to.
  baseline (cites glossary-terms): A placeholder is a name in a template that sync replaces with a value, written as `{{project}}` in this version. A value...
  unfiltered (decline phrase with other text): The sections do not define or differentiate between a placeholder and a value in sablefile 1.0. [sec:template-render] de...
  second-model (no citation): The difference between a placeholder and a value in sablefile 1.0 is that a placeholder is a name in a template that syn...
## unanswerable: 13 dev items
q-0015 | version 1.0 | planted (false-premise) | In sablefile 1.0, what does the --dry-run flag on sync print?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0016 | version 2.0 | not planted | Does sablefile 2.0 run on Windows?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0017 | version 2.0 | not planted | How fast is sablefile check on a project with several thousand generated files?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0018 | version 1.1 | not planted | Can sablefile 1.1 fetch a remote template from a private URL that needs a login token?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites remote-retries): Yes, sablefile 1.1 can fetch a remote template from a private URL that needs a login token, as long as the token is incl...
  second-model (declined): The docs don't say.
q-0034 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does sync --apply write to the lock file?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. Commit the lock file with the ...
q-0035 | version 1.0 | planted (false-premise) | Does check in sablefile 1.0 have a --no-fail option?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0036 | version 1.1 | planted (false-premise) | In sablefile 1.1, what does the --quiet flag on sync do?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (cites sync-default): The --quiet flag on sync is used to suppress the display of the diff output. It does not change the lock file or the syn...
q-0037 | version 1.0 | not planted | In sablefile 1.0, what happens when a template uses a placeholder whose name is not in [values]?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0038 | version 2.0 | not planted | Is there a maximum number of templates a 2.0 project can have?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0039 | version 1.1 | not planted | What exit status does sablefile diff return when it finds differences in 1.1?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (cites check-exit-status): When check finds a drifted file, it prints `DRIFT` and the path, and exits with status 1. [sec:check-exit-status]
  second-model (declined): The docs don't say.
q-0040 | version 2.0 | not planted | If a file listed in the lock is missing from disk, does check report it in 2.0?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0041 | version 2.0 | not planted | Is there a limit on how long a value in [values] can be?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0042 | version 1.0 | not planted | Can sablefile 1.0 set the output colour through a color key in sablefile.toml?
  reference: none: the item is unanswerable, so the right reply declines
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
## version-dependent: 17 dev items
q-0001 | version 1.1 | not planted | In sablefile 1.1, if I run sync with no flags, are the target files written straight away?
  reference: Yes. In 1.1 a plain sync writes each target and records it in the lock file. To preview the changes first, run diff, which shows them without writing anything.
  baseline (cites sync-default): Yes, if you run `sablefile sync` with no flags, the target files are written straight away. [sec:sync-default]
  unfiltered (cites sync-default): Yes, in sablefile 1.1, if you run `sync` with no flags, the target files are written straight away. This is because `sab...
  second-model (decline phrase with other text): The docs don't say. The docs don't say.
q-0002 | version 2.0 | not planted | In sablefile 2.0, if I run sync with no flags, are the target files written?
  reference: No. In 2.0 a plain sync prints a plan that lists what would be created or updated, and it writes nothing. Adding --apply makes sync write the targets and update the lock file.
  baseline (cites sync-default): No, the target files are not written when running `sablefile sync` with no flags. Instead, sync prints a plan and writes...
  unfiltered (cites apply-flag): No, in sablefile 2.0, if you run `sync` with no flags, the target files are not written. The `--apply` flag is required ...
  second-model (declined): The docs don't say.
q-0003 | version 1.0 | planted (version-swap) | In sablefile 1.0, is ${project} the right way to insert the project name into a template?
  reference: No. In 1.0 a placeholder is a name in double braces, written {{project}}. The text ${project} is not a placeholder in 1.0, so it is copied to the output unchanged.
  baseline (cites placeholders): No, `${project}` is not the right way to insert the project name into a template in sablefile. The correct way is to use...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0004 | version 2.0 | not planted | In sablefile 2.0, is ${project} the right way to insert the project name into a template?
  reference: Yes. In 2.0 a placeholder is a name with a dollar sign and braces around it, so the form ${project} is replaced by the project value. The old double-brace form is an error.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0021 | version 1.0 | not planted | In sablefile 1.0, does check give a nonzero exit status when a generated file has drifted?
  reference: No. In 1.0 check prints DRIFT and the path for a drifted file but exits with status 0, so a script has to read the printed lines to see drift.
  baseline (cites check-exit-status): No, check does not give a nonzero exit status when a generated file has drifted. The check command prints "DRIFT" and th...
  unfiltered (cites quickstart-check): Yes, in sablefile 1.0, `check` gives a nonzero exit status when a generated file has drifted. This is because, without t...
  second-model (cites check-exit-status): The check exits with status 0 even when files have drifted, so a script cannot detect drift from the exit status in this...
q-0022 | version 2.0 | not planted | In sablefile 2.0, does check give a nonzero exit status when a generated file has drifted?
  reference: Yes. check exits with status 1 when a file has drifted. Adding --no-fail changes that to status 0 while still printing the DRIFT lines.
  baseline (cites check-exit-status): Yes, check gives a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
  unfiltered (cites no-fail-flag no-fail-flag): No, in sablefile 2.0, `check` does not give a nonzero exit status when a generated file has drifted if the `--no-fail` f...
  second-model (cites check-exit-status): The check exits with a nonzero exit status when a generated file has drifted. [sec:check-exit-status]
q-0023 | version 1.0 | not planted | Does an ignore entry of *.log in sablefile 1.0 skip a target at docs/build.log?
  reference: No. In 1.0 each entry is a shell glob matched against the whole target path, and shell globs do not match across folders. The entry only covers files at the top level of the project.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 1.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (cites ignore-syntax): The docs section does not skip the target at docs/build.log. The ignore entry of *.log in sablefile 1.0 does not skip th...
q-0024 | version 2.0 | not planted | Does an ignore entry of *.log in sablefile 2.0 skip a target at docs/build.log?
  reference: Yes. In 2.0 entries use gitignore syntax, so a line such as *.log matches at any depth, including docs/build.log.
  baseline (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  unfiltered (cites ignore-syntax): Yes, an ignore entry of `*.log` in sablefile 2.0 skips a target at `docs/build.log`. [sec:ignore-syntax]
  second-model (declined): The docs don't say.
q-0025 | version 1.0 | planted (version-swap) | Does the lock file in sablefile 1.0 record each generated file's mode?
  reference: No. In 1.0 each lock line holds only the SHA-256 hash and the path, so no mode is recorded.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0026 | version 1.1 | not planted | Does the lock file in sablefile 1.1 record each generated file's mode?
  reference: Yes. Each lock line holds the SHA-256 hash, then the file's mode, then the path. check compares only the hash, so a change of mode alone is not reported as drift.
  baseline (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
  second-model (cites lock-mode): The lock file in sablefile 1.1 records each generated file's mode. [sec:lock-mode]
q-0027 | version 1.0 | not planted | How many times does sablefile 1.0 retry a remote template fetch that keeps failing?
  reference: Three retries, after which the command gives up. No flag or setting changes that count in 1.0.
  baseline (cites remote-retries): Three times. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.0 retries a remote template fetch up to 3 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.0, and no flag or setting changes it. [sec:remote-retries]
q-0028 | version 1.1 | not planted | How many times does sablefile 1.1 retry a remote template fetch that keeps failing?
  reference: Five retries, after which the command gives up. In 1.1 no flag or setting can change that count.
  baseline (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  unfiltered (cites remote-retries): Sablefile 1.1 retries a remote template fetch up to 5 times before giving up. [sec:remote-retries]
  second-model (cites remote-retries): The count is fixed in 1.1, and no flag or setting changes it. [sec:remote-retries]
q-0029 | version 1.0 | planted (version-swap) | In sablefile 1.0, does sync --apply create the missing folder for a target?
  reference: The 1.0 docs describe no --apply flag. Sync creates a target's missing folder by itself, before it writes the file, so a target such as docs/guide/index.md needs no setup.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0030 | version 2.0 | not planted | In sablefile 2.0, does sync --apply create the missing folder for a target?
  reference: Yes, but only because --apply is given. In 2.0 a plain sync creates no folder, since it writes nothing, and with --apply the folder is made before the target is written.
  baseline (cites sync-create-dirs): Yes, sync --apply creates the missing folder for a target. [sec:sync-create-dirs]
  unfiltered (cites apply-flag): Yes, sync --apply creates the missing folder for a target. [sec:apply-flag]
  second-model (cites sync-create-dirs): [sec:sync-create-dirs] Folders for targets   Yes, with `--apply`, sync creates the missing folders before writing the fi...
q-0031 | version 1.1 | planted (version-swap) | In sablefile 1.1, where is .sablefile/lock.json kept?
  reference: The 1.1 docs describe no .sablefile folder. The lock file in 1.1 is sablefile.lock in the project root, with one line for each generated file.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0032 | version 2.0 | not planted | In sablefile 2.0, where is the lock file kept?
  reference: In .sablefile/lock.json, a JSON file with a format string and one entry per generated file. A leftover sablefile.lock from 1.x is read with a warning and migrated when a command writes files.
  baseline (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  unfiltered (cites lock-migration): In sablefile 2.0, the lock file is kept in `.sablefile/lock.json`. [sec:lock-migration]
  second-model (cites lock-mode): The lock file is kept in `.sablefile/lock.json`. [sec:lock-mode]
q-0033 | version 2.0 | not planted | In sablefile 2.0, what does sync print for a target whose content already matches its template?
  reference: The word unchanged, as that target's plan line. The plan step itself writes nothing.
  baseline (declined): The docs don't say.
  unfiltered (cites apply-flag): For a target whose content already matches its template, sync prints "unchanged". [sec:apply-flag]
  second-model (declined): The docs don't say.
## ambiguous: 10 dev items
q-0019 | version 2.0 | planted (ambiguous-target) | If I edit the file and then run check, will it fail?
  reference: Reading 1: if you edited a generated file (a target), check prints DRIFT and the path for it and exits with status 1, because its hash no longer matches the lock.
Reading 2: if you edited a template, check prints nothing and exits with status 0, since a template change is not drift.
  baseline (cites quickstart-check check-exit-status): Yes, if you edit the file and then run `sablefile check`, it will report `DRIFT` and the path of the file, indicating a ...
  unfiltered (cites check-exit-status no-fail-flag): Yes, if you edit the file and then run `sablefile check`, it will detect the drift and print `DRIFT` and the path of the...
  second-model (cites check-exit-status): If you edit the file and then run `sablefile check`, it will fail because the check will detect a drift and print a `DRI...
q-0020 | version 2.0 | not planted | Does sync change the lock file?
  reference: Reading 1: a plain sync, with no flag, writes nothing, so the lock file is left alone.
Reading 2: sync --apply writes the targets and updates .sablefile/lock.json with the hash and mode of each file it wrote.
  baseline (cites sync-lock-update): Yes, sync changes the lock file when the `--apply` flag is used. After sync writes a target, it records that file's SHA-...
  unfiltered (cites diff-command): No, sync does not change the lock file. [sec:diff-command]
  second-model (cites sync-lock-update): The lock file is how check finds drift, so run sync after each change you intend to keep. [sec:sync-lock-update]
q-0043 | version 2.0 | planted (ambiguous-target) | Is it drifted now that I changed the template?
  reference: Reading 1: if 'it' means the target, it has not drifted. Nothing has edited the target since the last sync, so check prints nothing for it.
Reading 2: if 'it' means a target that you hand-edited after the last sync, it has drifted, and check prints DRIFT and exits with status 1.
  baseline (cites drift-meaning): No, it is not drifted now that you changed the template. A change to a template is not drift, because the generated file...
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0044 | version 2.0 | not planted | Does sync create a folder for a target?
  reference: Reading 1: a plain sync writes nothing, so it creates no folder.
Reading 2: sync --apply creates the target's missing folder before it writes the file.
  baseline (cites sync-create-dirs): Yes, sync creates a folder for a target if the folder does not exist, but only when the `--apply` flag is used. [sec:syn...
  unfiltered (declined): The docs don't say.
  second-model (cites sync-create-dirs): Yes, sync creates a folder for a target. [sec:sync-create-dirs]
q-0045 | version 2.0 | not planted | Does --retries change how often the command retries?
  reference: Reading 1: for sync and diff, yes. They use the count you give, in place of the default of five.
Reading 2: for check, no. check never fetches a remote template, so it never retries.
  baseline (cites retries-flag): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. [sec:retries-flag]
  unfiltered (cites retries-flag remote-retries): Yes, the `--retries` flag changes how often the command retries a failed fetch of a remote template. It sets the number ...
  second-model (cites retries-flag): The `--retries` flag changes how often the command retries a failed fetch of a remote template, in place of the default ...
q-0046 | version 2.0 | not planted | If I delete sablefile.lock, will check stop reporting drift?
  reference: Reading 1: if the old file was the only lock, deleting it leaves check with nothing to read, so check prints an error message instead of DRIFT lines.
Reading 2: once .sablefile/lock.json exists, 2.0 reads only that file, so deleting sablefile.lock changes nothing.
  baseline (declined): The docs don't say.
  unfiltered (declined): The docs don't say.
  second-model (declined): The docs don't say.
q-0047 | version 2.0 | not planted | When does a 2.0 project's old lock file get migrated?
  reference: Reading 1: a command that only reads, such as check, reads the old file and prints a warning, but writes nothing, so both files stay as they are.
Reading 2: a command that writes files, such as sync --apply, also writes .sablefile/lock.json and keeps each entry's mode where the old file has one.
  baseline (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  unfiltered (cites lock-migration): When a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.json`, it reads the old file and prints a war...
  second-model (no citation): The 2.0 project's old lock file is migrated when a 2.0 command finds an older `sablefile.lock` and no `.sablefile/lock.j...
q-0048 | version 1.0 | not planted | In 1.0, does check report the file if I change it?
  reference: Reading 1: if 'the file' is a template, no. check does not render templates, so it prints nothing for a template change.
Reading 2: if 'the file' is a generated target you edited, yes. check prints DRIFT and the path for it.
  baseline (declined): The docs don't say.
  unfiltered (cites check-command): Yes, in 1.0, `check` reports the file if it is changed. [sec:check-command]
  second-model (cites check-command): The check command does not report the file if it changes. [sec:check-command]
q-0049 | version 2.0 | not planted | Will init overwrite my config?
  reference: Reading 1: if sablefile.toml already exists, init stops with an error and leaves it alone.
Reading 2: if you mean another file in the folder, such as a README, init writes no other file, so it does not touch that file.
  baseline (cites init-command): No, init will not overwrite your config. It stops with an error if `sablefile.toml` already exists, so it never overwrit...
  unfiltered (declined): The docs don't say.
  second-model (cites init-command): init stops with an error if `sablefile.toml` already exists, so it never overwrites your configuration. [sec:init-comman...
q-0050 | version 1.1 | not planted | Does the 1.1 lock file care about a file's mode?
  reference: Reading 1: yes, the lock records each generated file's mode alongside its hash.
Reading 2: no, for comparison. check compares only the hash, so a mode-only change is not reported as drift.
  baseline (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. [sec:lock-mode]
  unfiltered (cites lock-mode): Yes, the 1.1 lock file cares about a file's mode. The mode field is described in lock-mode, and sync records the mode of...
  second-model (cites lock-mode): The 1.1 lock file does not care about a file's mode. The mode field is described in lock-mode. [sec:lock-mode]
```

Take item q-0005. It asks whether sablefile 2.0 still reads sablefile.lock in the project root, and it is a planted near-miss term, a name close to one the docs use, which catches a careless reply. The baseline says no and cites a section about upgrading. The reference answer says that 2.0 falls back to the file, with a warning, so the baseline's no is wrong. The unfiltered and second-model variants both decline, and the docs do answer the question, so their declines are wrong as well.

The three replies to q-0005 fail in two different ways. A decline on an answerable item and a confident wrong answer are different failures, and a check that catches one doesn't catch the other. A pass rate would count all three replies as failures and lose that difference. The note on this reply in the data file begins "says 2.0 never reads sablefile.lock; reference says it falls back to it with a warning." Each note says what the reply did and what the reference says, in one sentence, so the grouping that follows rests on something a reader can check. A count can come from different defects, and two replies that both fail a pass rate can be different failures. That is why the replies come before the count.

The comparison has limits, and they bound what the counts can claim. The dev items were written to probe the assistant, and the planted items are not a random sample of what users ask. So the counts say which variant fails in which way on these items. They don't say how often a user would meet each failure. A sample of questions drawn the way users ask them would be needed for that.

**Working rule:** Before you count anything, read replies from every variant, grouped by kind, and write down what is wrong with each one. A count can say that a variant fails. Only the replies say how.

## Coding the failures {#coding-the-failures}

Reading becomes counting once each reply has a note and a mode. The drafting model wrote one note per reply, as the one on q-0005 does, and then grouped the notes. Each reply got one mode, its main defect, and a reply with no defect got `ok`. The codes were made by the AI model that drafted this book, as one coder; no second coder checked them, and nobody measured agreement on the codes. The 80 dev items give 240 replies, 80 from each variant. The codes sit in a data file beside the examples, one entry per reply.

The counts carry the label "coded by the drafting model (one coder; not human labels)", and the label means what it says. This chapter has no agreement figure, and no one else has reproduced the counts. [Chapter 1](measure-before-you-change) made the same point: agreement with your own labels doesn't calibrate a judge. [Chapter 6](calibrate-the-judge) takes that up.

The tally example turns the notes into counts, and it checks the codes first. Each dev item has exactly one code per variant, no code names a mode the file doesn't define, and the file carries the label. Then it prints one row per mode and one column per variant, and one labelled line for each count, which are the numbers the prose below quotes.

Example `tally-failure-modes` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Counts the drafting model's failure codes over the dev items, per variant, and prints one table and one labelled line
per count.

The point rests on the table's reading of each mode: the codes were made by the drafting model, as one coder, and are not
human labels. The codes live in data/failure-codes.json; the checks below stop a reply being counted twice or left out.
The labelled lines are the numbers the chapter's prose quotes."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import json
from collections import Counter
from pathlib import Path

from lib.testset import load_items

CODES = Path(__file__).resolve().parent / "data" / "failure-codes.json"
VARIANTS = ("baseline", "unfiltered", "second-model")

book = json.loads(CODES.read_text(encoding="utf-8"))
assert book["labelled"] == "coded by the drafting model (one coder; not human labels)", "the codes are the model's, not human labels"  # the label the prose uses
assert book["variants"] == list(VARIANTS), "the variants in the file are the three the chapter names"

dev_ids = {item.id for item in load_items(split="dev")}
n = len(dev_ids)
codes = book["codes"]
assert {code["item"] for code in codes} <= dev_ids, "a code names an item that is not in the dev split"
pairs = [(code["item"], code["variant"]) for code in codes]
# One code per dev item and variant: a pair repeated would be counted twice, a missing one would go unseen.
assert len(pairs) == len(set(pairs)) == n * len(VARIANTS), "expected one code per dev item and variant"
assert all(code["mode"] in book["modes"] for code in codes), "a code uses a mode the file does not define"

counts = Counter((code["variant"], code["mode"]) for code in codes)
print(f"dev items: {n}")
print(f"replies coded: {len(codes)}")
print(f"labelled: {book['labelled']}")
print(f"{'mode':<22}" + "".join(f"{variant:>14}" for variant in VARIANTS))
for mode in book["modes"]:  # the file's order: ok first, then the failure modes
    total = sum(counts[(variant, mode)] for variant in VARIANTS)
    if total == 0:  # a mode no variant shows is printed as zero of N, so the reader sees it was looked for
        print(f"{mode}: 0 of {n}")
        continue
    print(f"{mode:<22}" + "".join(f"{counts[(variant, mode)]:>14}" for variant in VARIANTS))
totals = {variant: sum(counts[(variant, mode)] for mode in book["modes"]) for variant in VARIANTS}
assert all(total == n for total in totals.values()), "each variant's counts add up to the dev items"
print(f"{'all modes':<22}" + "".join(f"{totals[variant]:>14}" for variant in VARIANTS))
print()
for variant in VARIANTS:  # one labelled line per count, the form the chapter's prose binds to
    for mode in book["modes"]:
        print(f"{mode}, {variant}: {counts[(variant, mode)]}")
    print(f"defective, {variant}: {n - counts[(variant, 'ok')]}")
```

Output:

```text
dev items: 80
replies coded: 240
labelled: coded by the drafting model (one coder; not human labels)
mode                        baseline    unfiltered  second-model
ok                                56            49            24
over-declining                    10            15            31
wrong-version-answer               2             3             0
wrong-answer                       3             5             9
missed-refusal                     0             2             2
incomplete-answer                  8             4             7
citation-problem                   1             2             6
malformed-reply                    0             0             1
all modes                         80            80            80

ok, baseline: 56
over-declining, baseline: 10
wrong-version-answer, baseline: 2
wrong-answer, baseline: 3
missed-refusal, baseline: 0
incomplete-answer, baseline: 8
citation-problem, baseline: 1
malformed-reply, baseline: 0
defective, baseline: 24
ok, unfiltered: 49
over-declining, unfiltered: 15
wrong-version-answer, unfiltered: 3
wrong-answer, unfiltered: 5
missed-refusal, unfiltered: 2
incomplete-answer, unfiltered: 4
citation-problem, unfiltered: 2
malformed-reply, unfiltered: 0
defective, unfiltered: 31
ok, second-model: 24
over-declining, second-model: 31
wrong-version-answer, second-model: 0
wrong-answer, second-model: 9
missed-refusal, second-model: 2
incomplete-answer, second-model: 7
citation-problem, second-model: 6
malformed-reply, second-model: 1
defective, second-model: 56
```

Read the `ok` row, then the rows under it. The baseline has the most replies with no defect, 56, and the second-model variant has the fewest, 24. Over-declining is the largest defect in each variant. In the second-model variant it is more than half of the defects: 31 of 56.

The baseline doesn't escape the worse defects. It gives 2 wrong-version answers, on q-0023 and q-0029, and the unfiltered variant gives the most, 3. Missed refusals come only from the variants, and the baseline has none. Every mode appears at least once. A mode that no variant showed would print as zero, so a reader could see it was looked for.

**Working rule:** Give each reply one note and one mode, and print one labelled line per count, so that every number the prose gives can be checked against the output.

## Over-declining {#over-declining}

An over-declining reply declines an item the docs answer. It is the defect a reader meets first, because a decline looks careful, and it fails the user as surely as a wrong answer does. The terms used here, the decline phrase, the unanswerable item and the variants, are defined in [what the replies look like](#what-the-replies-look-like). The count covers every item the docs answer, version-dependent items included, so for the baseline it is larger than the false refusal rate in [chapter 10](declining-when-the-docs-dont-say), which counts answerable items only. Item q-0004 is the version-dependent case that [chapter 1](measure-before-you-change#declines-in-code) explains in full. All three variants decline it with the same words, so the baseline does this too.

Counting declines has a trap, and it sits in the parser. Item q-0080 is answerable, and the unfiltered variant's reply to it gives two sentences saying the sections don't define the difference, then ends with the decline phrase. The docs answer the question, so the reply is an over-decline. The parser counts a decline only when the phrase is the whole reply, so it reads this one as an answer that cites a section. A reader who searches for the phrase finds it, and a check built on the parser misses it.

The check candidate is that a reply to an item whose reference names a supporting section must not decline, in any words. [Checks in code](checks-in-code) builds checks of that kind. Whether a decline is right on an item the docs don't cover is a judgement, which [chapter 10](declining-when-the-docs-dont-say) takes up.

**Working rule:** Count a decline by what the reply does, not by whether the phrase stands alone, and count over-declining apart from wrong answers, because their checks differ.

## Wrong-version answers {#wrong-version-answers}

A wrong-version answer gives another version's behaviour for the version the question asks about, and it can be fluent and cited while being wrong. In q-0029 the baseline says yes, `sync --apply` creates a missing folder in 1.0, and cites the section on creating folders. The note on the reply says that is the 2.0 behaviour, and that 1.0 has no `--apply` flag at all.

The unfiltered variant does this more often. In q-0021 it says yes to whether `check` gives a nonzero exit status for drift in 1.0, where the baseline and the second-model variant both answer that it does not. The note says the unfiltered reply gave the 2.0 rule for a 1.0 question. Its context holds every version, so the 2.0 rule sits beside the 1.0 one.

Citations don't settle it. In q-0023 the baseline answers yes about a 2.0 result, and it cites the 1.0 section. A check that every citation names a section of the asked version passes that reply. Only the claim, read against the 1.0 reference, shows the error.

**Working rule:** Check each reply's version in two places. Its citations must belong to the asked version, and its claim must match that version's reference. A citation check alone passes an answer that is right about the wrong version.

## Missed refusals {#missed-refusals}

A missed refusal answers an unanswerable item with content the docs don't hold. It is the mode a reader is most likely to believe, because the reply reads like any other answer. The terms used here are defined in [what the replies look like](#what-the-replies-look-like). The dev items show it only in the variants, 2 times in the unfiltered variant and 2 in the second-model variant, and each is short enough to read whole.

In q-0018 the unfiltered variant says a login token can be sent with a remote template fetch, and cites a section about retries. In q-0034 the second-model variant gives a tip about the lock file, cited to a section that doesn't cover the question. In q-0036 it says `--quiet` suppresses the diff output, citing a section that never mentions the flag. In q-0039 the unfiltered variant answers about `check` when the question asks about `diff`. The baseline declines every one of them.

A code check can require a decline on every unanswerable item, and it would catch each of them. On an answerable item the same fault goes unseen: a reply that cites a real section silent on its claim would pass, because it answers, as an answer is expected there. Reading the claim against the cited section is the work of [chapter 9](grounded-answers) and of [chapter 10](declining-when-the-docs-dont-say).

**Working rule:** Require a decline on every unanswerable item, and read every answer to one against the section it cites before you count it as right.

## From a failure to a check {#from-a-failure-to-a-check}

A failure mode is useful once it has a check candidate, and a candidate has four parts: the mode, the input the check reads, the claim it tests, and what it can't see. A candidate with no stated limit hasn't been thought through. A candidate is also not yet a check. [Checks in code](checks-in-code) builds the checks a program can run, and the judgements go to a judge. The list gives each of the seven failure modes from [coding the failures](#coding-the-failures) its candidate, and names the case that shows it.

- **Over-declining.** A reply to an item whose reference names a supporting section must not decline, in any words. The check can't tell a right decline on an item the docs don't cover from a wrong one, so that judgement goes to [chapter 10](declining-when-the-docs-dont-say).
- **Wrong-version answers.** Every citation must belong to the asked version, and the claim must match that version's reference. The first half is a code check in [checks in code](checks-in-code). The second, which catches q-0023, is version-correctness in [chapter 9](grounded-answers). It can't see a claim the reference doesn't cover.
- **Wrong answers.** The claim must match the reference answer for the asked version, and a reply to an ambiguous item must answer each reading, as [one judge](measure-before-you-change#one-judge) sets out. A judge with a written rubric compares each claim with its reference, as in [chapter 5](a-judge-with-a-rubric). Its test case is q-0005, where the baseline says 2.0 never reads sablefile.lock and the reference says it falls back to it. For ambiguous items the rubric must put both readings in the reference, as q-0048 needs: the unfiltered variant says yes and the second-model variant says no, and each is wrong about one reading. The judge can be wrong too, which [chapter 6](calibrate-the-judge) measures.
- **Missed refusals.** Every reply to an unanswerable item declines, or a reader checks its citation against the section it names. The first is a code check. The second is the work of [chapter 9](grounded-answers). The code check reads no section text, so on an answerable item it passes a reply that cites a real section silent on its claim.
- **Incomplete answers.** The reference lists each number, step and reading a right answer needs, and a code check could look for each one in the reply. This book builds no such check. In q-0011 the baseline says the retry count is fixed and that no flag changes it, but never gives the number. The unfiltered variant does give it, so a check for the reference's number would fail the baseline's reply. Its limit: a number can appear in the reply without being the answer.
- **Citation problems.** Every citation must be one of the item's supporting sections. In q-0057 the baseline is right that `check` doesn't see changes on a remote server, but it cites a section the item doesn't list as supporting, so the citation is unlisted. A code check against that list catches it. Its limit: a listed section can still fail to support the claim, which takes a reader, as [checks in code](checks-in-code) explains.
- **Malformed replies.** A check on the reply's shape, against the format the prompt asks for, catches a reply that comes out in the wrong order. The second-model reply to q-0030 puts its citation first and garbles its sentence, though its answer is right. Its limit: shape says nothing about the answer, so a reply can pass the shape check and still be wrong. The format checks in [checks in code](checks-in-code) build it.

Each candidate in the list names the reply it would have caught, and that is its first test. If a candidate doesn't flag the reply it was written for, it isn't yet a check for the mode.

Build the code checks first. They run in seconds and need no model call, so they can run on every change. A judge costs a model call for each reply and needs its own calibration, so it goes to the modes that a reference and a citation can't settle. Wrong answers need one for the claim. Wrong-version answers need one only for the part that reads a claim against its reference, and missed refusals need one only for the part that reads a claim against the section it cites.

In production the modes don't arrive labelled. A user reports a wrong reply, and the first question is which mode it is, because a check written for the wrong mode passes the reply that failed. When no mode fits, read the failing replies again, code them, add a mode and write its check. This is why the chapter reads and codes the replies before it counts anything.

The list isn't closed: the held-out items and any later variant will show modes these items don't, and each will need its own note and its own candidate.

The chapter made these decisions:

- One coder made the codes: the drafting model. They are labelled as such, and no agreement is measured.
- The dev items are the only items read here; the held-out decline counts appear in chapter 10.
- The baseline's replies are counted with the variants, so its 2 wrong-version answers are reported as such.

**Working rule:** Code the replies by their main defect before you choose a check, and write each check for one mode. A check built from the failures you expected tests for those failures, and it passes the ones you didn't expect.

# Build a test set you can defend, and read public benchmarks

By the end of this chapter you can build a test set whose score tells you something, check its schema and coverage with code, and say what a public benchmark score does and does not tell you about your own assistant. You will also see how a multiple-choice score can be impossible to tell apart from a habit of answering the same way.

Here the test set belongs to the sablefile assistant, a retrieval step and a model that answers questions from the tool's docs. Every number this book reports about the assistant comes from a set of questions with reference answers, so the set decides how much a score is worth. The book guards against three faults. If every question has an answer in the docs, an assistant that always answers will look good. If the questions reuse the docs' own sentences, retrieval matches on words and looks right. If the set is tuned until the score rises, the score records the tuning. The set is also the only place the questions are written down, so a fault in it can go unseen, and a score built on it inherits the fault without any sign of it.

This chapter builds the set the book uses and checks it with code. It then reads two public benchmarks, GSM8K and TruthfulQA, which many readers will have seen quoted, and says what their scores can still show. The sections follow the questions a team would ask in turn: what the set is for, how it is built and checked, how it is split, and what a public score adds.

## What a test set is for {#what-a-test-set-is-for}

A [test set](#the-twenty-items) is a fixed set of questions, paired with the answers you would accept, that stands in for the questions users will ask. Its job is to be asked the same questions each time something changes, so that a difference in score comes from the change and not from the questions. It does that job only while it stays fixed. This set was written from the docs rather than sampled from users, so it stands in for them and does not reproduce them.

Each row is an **item**: one test question with the fields a scorer needs. The item is the unit of every count in this book, so a pass rate is always a share of items, and its denominator says which ones. Each item has a **reference answer**, which the scorer compares a reply with, and **supporting sections**, the parts of the docs that a correct answer rests on. The reference answers are written labels. The book's own drafting agents wrote them from the docs, a second pass checked them against the docs, and a script checks the file's shape and section IDs. They are not human labels, and the book does not call them that.

Each item also has a **kind**, which says what a correct reply looks like. A set built from one kind tests one skill, so this set mixes four kinds on purpose. It should also look like the questions users bring. Users ask things the docs cannot answer, and a set with only answerable questions would reward an assistant that never declines.

The set is small on purpose. It holds 120 items, which a reader can check by hand, and each one has a reference answer that a second pass checked against the docs. We keep it small enough to read, because a set nobody reads can only measure its own mistakes. A score is only as good as the set behind it, and the set is what two people argue about when they disagree on whether a change helped. Keep it readable, keep the reasons for each item written down, and do not add items only because a score looks wrong, since that picks items for their result.

A test set is never training data. Nothing in it teaches the assistant, and the assistant never sees a reference answer while the set is being built. If someone tunes against the answers, the set stops being a test and becomes a target, which is what the [rule on dev and held-out](#dev-and-held-out) is for.

**Working rule:** Write the set before you change anything you will measure with it, and keep its file under version control, so that every comparison names the items it used. A set you edit after seeing the scores measures your edits, not the assistant.

## The item schema {#the-item-schema}

The set has to load before anything is scored. The file holds one JSON object per line, and a loader validates each one with pydantic, a Python library that checks values against a declared shape. This book pins pydantic 2.14.3. The fields are these:

- `id`: `q-` and four digits, unique, and never changed once written.
- `question`: one to 400 plain characters.
- `version`: the docs version the question asks about.
- `kind`: one of the four kinds.
- `reference_answer`: the answer, or nothing for an unanswerable item, where the right reply is to decline.
- `supporting_sections`: the version and section ID pairs that the answer rests on.
- `split`: `dev` or `held-out`.
- `planted` and `planted_why`: whether the item is a planted trap, and which one.
- `twin_of`: the ID of the item that asks the same question about another version, if there is one.

The loader sets `extra="forbid"`, so an unknown field is an error and not a field that is quietly ignored. A typo in a field name would otherwise produce a set that looks right and scores wrong. Errors are reported row by row, so one bad row does not hide the rest.

Shape checks cannot see across rows, so the cross-item rules in the book's `lib/testset.py` do that work. They check that IDs are unique, that each twin names its partner back, asks for another version, is version-dependent and sits in the same split, and that every false-premise item is unanswerable. They check that every supporting section exists for the version it names, that no reference answer repeats eleven words in a row from a section, and that the counts match the design.

Each of these rules is a contract that anyone who adds or edits an item has to pass. The check prints each broken rule. It exits without an error code whether or not it finds one, so it stops nothing by itself, and the build has to treat a printed problem as a failure.

The reference answer needs the most care of all, because every reply is compared with it. A reference that is wrong in a consistent direction makes the score wrong in that direction, and no shape rule can catch a well-formed wrong answer. That is why each reference is written with the sections it rests on, so a reader can check the claim against the docs in a few minutes rather than take the author's word.

Twins can drift into different splits when they are written at different times, and then the dev part tunes on a question the held-out part checks. The cross-item rules catch that, so the twins stay together.

The schema check below runs both groups of rules and prints the problem count, with the counts by kind and by split.

Example `check-item-schema` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Loads all 120 test-set items, runs the per-item and the cross-item rules, and prints the problem count with the
counts by kind and by split. The point rests on the first line: a set that passes prints zero problems."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.testset import cross_item_problems, item_problems

items, row_problems = item_problems()  # every row, even a bad one, is reported rather than stopping the run
problems = sorted(row_problems) + cross_item_problems(items)
print(f"items: {len(items)}")
print(f"problems: {len(problems)}")
for problem in problems:
    print(f"  {problem}")

kinds = Counter(item.kind for item in items)
for kind in ("answerable", "unanswerable", "version-dependent", "ambiguous"):
    print(f"kind {kind}: {kinds[kind]}")
splits = Counter(item.split for item in items)
for split in ("dev", "held-out"):
    print(f"split {split}: {splits[split]}")
print(f"planted: {sum(1 for item in items if item.planted)}")
```

Output:

```text
items: 120
problems: 0
kind answerable: 60
kind unanswerable: 20
kind version-dependent: 25
kind ambiguous: 15
split dev: 80
split held-out: 40
planted: 24
```

The set holds 120 items, and the rules find 0 problems. A clean run means the rows parse, the references point at sections that exist, and the counts match the design. It shows the set's shape only. It does not mean the questions are good, that the reference answers are right, or that a planted item catches an assistant. Those claims need a reader, and the rest of this chapter describes how the set was built so that a reader can check them.

**Working rule:** Run the schema check in the build and treat any printed problem as a failure, and read a sample of items by hand before you trust the set.

## Kinds and coverage {#kinds-and-coverage}

Each kind tests something the assistant must do. An [answerable item](#the-twenty-items) has docs that answer it fully, so the assistant must answer and agree with the reference, and its citations are checked on their own. A refusal there is a [false refusal](#declines-in-code). An [unanswerable item](#the-twenty-items) has docs that do not say, so the right reply is the decline phrase, exactly. Any other reply is a false answer, and these items are what stop an assistant that always answers from scoring well. [The chapter on declining](declining-when-the-docs-dont-say) covers the phrase. A version-dependent item names the version it asks about, and its reply must match that version's reference. An ambiguous item admits two readings, and the reference holds one answer per reading. A reply must answer each reading the docs support, and a partial answer fails.

The kinds table below prints the set by kind and by split.

Example `kinds-by-split` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Prints the stratification of plan section 3.4 as counts: kind by split, then planted items by split, with totals.
The dev and held-out columns are the numbers the chapter's split table quotes."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.testset import load_items

items = load_items()
splits = ("dev", "held-out")
kinds = ("answerable", "unanswerable", "version-dependent", "ambiguous")
counts = Counter((item.kind, item.split) for item in items)
print(f"{'kind':<20}{'dev':>6}{'held-out':>10}{'total':>8}")
for kind in kinds:
    row = [counts[(kind, split)] for split in splits]
    print(f"{kind:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")
totals = [sum(counts[(kind, split)] for kind in kinds) for split in splits]
print(f"{'total':<20}{totals[0]:>6}{totals[1]:>10}{sum(totals):>8}")
print()
planted = Counter((item.planted_why, item.split) for item in items if item.planted)
print(f"{'planted':<20}{'dev':>6}{'held-out':>10}{'total':>8}")
reasons = sorted({why for why, _ in planted})
for why in reasons:
    row = [planted[(why, split)] for split in splits]
    print(f"{why:<20}{row[0]:>6}{row[1]:>10}{sum(row):>8}")
planted_totals = [sum(planted[(why, split)] for why in reasons) for split in splits]
print(f"{'total':<20}{planted_totals[0]:>6}{planted_totals[1]:>10}{sum(planted_totals):>8}")
print()
print(f"items: {len(items)}")
```

Output:

```text
kind                   dev  held-out   total
answerable              40        20      60
unanswerable            13         7      20
version-dependent       17         8      25
ambiguous               10         5      15
total                   80        40     120

planted                dev  held-out   total
ambiguous-target         2         2       4
false-premise            4         2       6
near-miss-term           3         2       5
stale-answer             3         1       4
version-swap             4         1       5
total                   16         8      24

items: 120
```

The table prints the split by kind that the design fixes: each kind is divided between dev and held-out in about the same proportion, and the planted items are split the same way. Its last line gives the item count, 120, which matches the schema check. By kind, the set holds:

- answerable: 60 items;
- unanswerable: 20 items;
- version-dependent: 25 items;
- ambiguous: 15 items.

Coverage asks which parts of the tool the set exercises. The sablefile tool has six behaviours that an assistant must get right, named B1 to B6: what `sablefile sync` does with no flag, the exit status of `check` when a file has drifted, the syntax of `ignore` entries, the lock file's format, the placeholder syntax, and the retry count for remote fetches. The wrong-version answers that [chapter 2](read-the-outputs) found fall on these behaviours: q-0021 on B2, q-0023 on B3 and q-0029 on B1.

The coverage check stops if its mapping misses an item.

Example `behaviour-coverage` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Shows how many items of the test set exercise each behaviour B1 to B6, and which ones. The mapping in data/behaviours.json
must list every item of the set exactly once, or the example stops; items that exercise no behaviour are counted apart."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import json
from pathlib import Path

from lib.testset import load_items

BEHAVIOURS = Path(__file__).resolve().parent / "data" / "behaviours.json"
ids = sorted(item.id for item in load_items())
mapping = json.loads(BEHAVIOURS.read_text(encoding="utf-8"))
assert sorted(mapping) == ids, "behaviours.json must list every item of the set, once"  # the coverage needs every item

print(f"items: {len(ids)}")
print(f"items with no behaviour: {sum(1 for labels in mapping.values() if not labels)}")
for behaviour in ("B1", "B2", "B3", "B4", "B5", "B6"):
    hits = sorted(item for item, labels in mapping.items() if behaviour in labels)
    print(f"{behaviour} items: {len(hits)}")
    print(f"{behaviour} ids: {', '.join(hits)}")
```

Output:

```text
items: 120
items with no behaviour: 48
B1 items: 11
B1 ids: q-0001, q-0002, q-0020, q-0029, q-0030, q-0033, q-0044, q-0061, q-0063, q-0064, q-0103
B2 items: 13
B2 ids: q-0006, q-0019, q-0021, q-0022, q-0043, q-0048, q-0056, q-0057, q-0058, q-0083, q-0084, q-0089, q-0092
B3 items: 14
B3 ids: q-0007, q-0008, q-0023, q-0024, q-0051, q-0071, q-0072, q-0073, q-0081, q-0082, q-0114, q-0115, q-0116, q-0117
B4 items: 17
B4 ids: q-0005, q-0009, q-0010, q-0020, q-0025, q-0026, q-0031, q-0032, q-0046, q-0047, q-0050, q-0053, q-0054, q-0059, q-0060, q-0101, q-0119
B5 items: 8
B5 ids: q-0003, q-0004, q-0013, q-0052, q-0079, q-0080, q-0087, q-0088
B6 items: 10
B6 ids: q-0011, q-0012, q-0027, q-0028, q-0045, q-0055, q-0068, q-0085, q-0086, q-0109
```

Each behaviour's count is one line of output:

- B1, the default of `sync`: 11 items.
- B2, the exit status of `check`: 13 items.
- B3, `ignore` syntax: 14 items.
- B4, the lock file: 17 items.
- B5, placeholder syntax: 8 items.
- B6, retries for remote fetches: 10 items.

One item carries two behaviours, so the six counts add up to one more than the items that carry any. Items with no behaviour are still part of the set. Here 48 of the 120 items carry none.

**Working rule:** Count coverage for each behaviour before you quote a score for it, and treat a behaviour with only a few items as untested. Passing those items shows that one behaviour, and nothing about the rest of the tool.

### Planted items and their traps {#planted-items}

A [planted item](#the-twenty-items) is written to break an assistant in one named way. Each carries a reason from a fixed list of five:

- `false-premise`: the question asks about a flag or key that the asked version does not have. The right reply is the decline phrase, so planted false premises are always unanswerable.
- `near-miss-term`: a term one version renamed, or a term the docs use for another concept.
- `version-swap`: the question uses another version's wording, so an assistant that matches wording answers for that version.
- `stale-answer`: the answer sits under a section ID that a version filter removes, so a filter that is too strict misses it.
- `ambiguous-target`: a pronoun or a phrase such as "the file" has no clear referent without the sections.

A planted item is useful because its failure is named. If the assistant fails it, a reader knows which failure happened. Planted items are counted with their kind, so a planted false premise is also unanswerable. The set holds 24 planted items, about one in five. Planted items are not split out in the metric tables, and no chapter in this book reports results trap by trap.

**Working rule:** Plant an item only for a trap you can name from the list. Write down the wrong reply it should catch, and check against the cited sections that the wrong reply is wrong, before the item enters the set.

### Synthetic items and what they hide {#synthetic-items}

These items were written from the docs, and the book builds them against two traps. The first is copying. A question that repeats a section's sentence gives retrieval a match on words it finds at once, so the test then measures word matching and not answering. The copy check stops a reference that repeats eleven words of a section, but it covers reference answers only, not questions. The rule for writing is to ask each question the way a user would, in the user's words and not the docs'.

The second trap is a reference that is true for two versions, so a reply that describes the other version passes. A version-dependent item names its version, and its twin pairs the same question with a second version, so no reply is right for both by accident.

**Working rule:** Ask each question the way a user would, from the docs but not in their sentences, and read each question against its sections before the set is frozen, since the copy check does not read questions. For a behaviour that changed between versions, name the version in the question and write its twin.

## Dev and held-out {#dev-and-held-out}

The set is split in two, because a score tuned on the questions it is measured with does not show progress. [Dev](#read-the-outputs) may be tuned on, and held-out is used once, to report a result. The split is stratified, so that each kind appears in both parts in about the same proportion. Without that, one part could hold most of the unanswerable items, and the two scores would then measure different sets. The set holds 80 dev items and 40 held-out items, and each row's `split` field records which part it belongs to. The twenty items that [chapter 1](measure-before-you-change) measures are dev items, so its rates are dev rates.

Only dev may be tuned on. Prompt wording, the number of sections retrieved, the version filter and the wording of the judge's rubric may all be chosen from dev results. Once a setting is chosen, it is frozen before held-out runs. Held-out runs once for each variant, in the chapter that compares the variants, and its numbers are the ones a chapter quotes. A recording's manifest says whether anything in it was tuned on held-out, and that answer must be no.

Dev has a job that held-out does not. It is where failures are read, so that a reader can form a hypothesis about why the assistant fails. A hypothesis formed on dev is a claim to test on held-out, not a result. Reading dev failures to choose a setting is allowed. Reading held-out failures to choose a change is not.

A split is only as clean as its record. Each row's split is written once, when the item is written, and it does not move with the scores. A split that changes after the scores are in is not a split.

Held-out is small, and a score on it is one number with an interval around it. A comparison on held-out therefore needs paired items, one verdict per item for each variant, and an interval over those pairs. [The chapter on intervals and sample size](intervals-and-sample-size) builds that comparison, and this chapter only needs the rule it rests on.

One more rule keeps the split honest. A question may be corrected only when its reference answer or its sections are wrong. It may never be changed because the assistant fails it. Otherwise the set would measure the assistant's last failure, and the score would rise each time someone fixed a failure by editing the test.

**Working rule:** Tune only on dev, freeze every setting before you open held-out, and run held-out once for each comparison. A held-out score that you read before a change, and then lifted by changing something, is a dev score, and you should say so.

## Public benchmarks {#public-benchmarks}

A public benchmark is a set of questions and answers someone else published. It does not measure the sablefile assistant, and its score depends on the prompt, the sampling and the parser.

Both samples are seeded samples of one hundred items, and their cards were read on November 20, 2026. Both recordings come from Qwen3-4B-Instruct-2507, the book's assistant and judge, in its Q4_K_M GGUF build, run on a CPU with greedy decoding. Replies were capped at 256 tokens for GSM8K and 160 for TruthfulQA.

**Working rule:** Quote a public score only beside its sample, model build, prompt, parser, reply cap and interval, and never beside a published number.

### GSM8K {#gsm8k}

GSM8K is grade-school math word problems, each with a worked solution that ends in one number. The score is an exact match on that number. The prompt asks for a "Final answer:" line the scorer never reads. It takes the number after the last `####` marker in the reply, or else the last number in the reply. No reply contains `####`, so every verdict came from the last number, which for a finished reply is its "Final answer:" number. The paper describes 8.5K problems, with 1K for testing ([Cobbe and colleagues, 2021](https://arxiv.org/abs/2110.14168)). The [dataset card](https://huggingface.co/datasets/openai/gsm8k) gives its second split 1,319 rows, which the viewer calls "test" and the splits table calls "validation". The book does not reconcile them, and the sample uses the viewer's "test" split. The card gives the licence as MIT.

Example `score-gsm8k` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Scores the recorded replies on the 100-item GSM8K sample by exact match on the final number. The accuracy line rests on
its count of readable replies; unreadable replies are counted apart, never as wrong. The three example items show a right,
a wrong and an unreadable reply. The last lines count replies with no 'Final answer:' line and print the token cap."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.bench import load_gsm8k, score_gsm8k
from lib.prompts import BENCH_GSM8K_SHA256
from lib.replay import ReplayModel
from lib.stats import bootstrap_mean, interval_pct, pct

rows = load_gsm8k()
by_id = {row["id"]: row for row in rows}
replies = ReplayModel("bench-gsm8k", BENCH_GSM8K_SHA256)  # fails if the recording was made from another prompt
scored = score_gsm8k(rows, {row["id"]: replies.reply(row["id"]) for row in rows})
parsed = {item: number for item, number, _ in scored}

readable = [entry for entry in scored if entry[2] is not None]
right = [entry[0] for entry in readable if entry[2]]
wrong = [entry[0] for entry in readable if not entry[2]]
unreadable = [entry[0] for entry in scored if entry[2] is None]

print(f"items: {len(rows)}")
print(f"readable: {len(readable)} of {len(rows)}")
print(f"accuracy: {pct(len(right), len(readable))}")
# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.
lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])
print(f"accuracy interval: {interval_pct(lo, hi)}")
print(f"unreadable: {len(unreadable)}")
print(f"unreadable ids: {', '.join(unreadable) or 'none'}")
no_final = [row["id"] for row in rows if "final answer:" not in replies.reply(row["id"]).lower()]
print(f"no final answer line: {len(no_final)} of {len(rows)}")
print(f"no final answer line among wrong: {sum(item in no_final for item in wrong)} of {len(wrong)}")
print(f"no final answer line among right: {sum(item in no_final for item in right)} of {len(right)}")
print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")

for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):
    if not ids:
        print(f"{kind} example: none")
        continue
    item = ids[0]  # the first by ID, so the same item prints on every run
    print(f"{kind} example: {item}")
    print(f"{kind} question: {by_id[item]['question'][:50]}")
    print(f"{kind} reference: {by_id[item]['reference']}")
    print(f"{kind} parsed: {parsed[item] or 'none'}")
```

Output:

```text
items: 100
readable: 100 of 100
accuracy: 90.0% (90 of 100)
accuracy interval: [84.0%, 95.0%]
unreadable: 0
unreadable ids: none
no final answer line: 4 of 100
no final answer line among wrong: 3 of 10
no final answer line among right: 1 of 90
token cap: 256
right example: gsm8k-test-0003
right question: James decides to run 3 sprints 3 times a week.  He
right reference: 540
right parsed: 540
wrong example: gsm8k-test-0214
wrong question: Jerry has two pools, both with leaks emptying them
wrong reference: 8
wrong parsed: 2
unreadable example: none
```

The parser found a number in 100 of 100 replies. The accuracy is 90.0% (90 of 100). The interval is [84.0%, 95.0%], so a gap of a few points between two configurations would fall inside it.

4 of 100 replies have no "Final answer:" line, and each stops mid-solution. Among the wrong replies, 3 of 10 have no such line. A cut-off reply is scored by its last number, often an intermediate value, so the score partly measures the cap rather than only the reasoning. The wrong example, gsm8k-test-0214, is one of these: the scorer read 2, and the reference is 8.

The cap also touches the right replies: 1 of 90 stops before its final line, and the scorer marks it right only because its last number is the answer.

The book uses GSM8K to check the harness, not to rank the assistant.

**Working rule:** Count the replies with no final line before you quote a score, and compare two configurations only with a [paired comparison](#paired-comparisons) that clears its interval.

### TruthfulQA {#truthfulqa}

TruthfulQA asks questions on which people often answer falsely, such as a common myth. Its paper describes 817 questions in 38 categories ([Lin and colleagues, 2022](https://arxiv.org/abs/2109.07958)). The book uses TruthfulQA's multiple-choice version, MC1: exactly one option is correct, and the model answers with a letter. The [dataset card](https://huggingface.co/datasets/truthfulqa/truthful_qa) has one split, named validation, and gives the licence as Apache-2.0.

Example `score-truthfulqa` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Scores the recorded replies on the 100-item TruthfulQA MC1 sample by the option letter each reply gives. Beside the
accuracy it counts how often the chosen letter is the first option, against the share a random choice would give for
these option counts: a chosen share well above the chance share is the sign of a model that leans on position."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.bench import load_truthfulqa, score_truthfulqa
from lib.prompts import BENCH_TQA_SHA256
from lib.replay import ReplayModel
from lib.stats import bootstrap_mean, interval_pct, pct

rows = load_truthfulqa()
by_id = {row["id"]: row for row in rows}
replies = ReplayModel("bench-truthfulqa", BENCH_TQA_SHA256)  # fails if the recording was made from another prompt
scored = score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows})
chosen = {item: letter for item, letter, _ in scored}

readable = [entry for entry in scored if entry[2] is not None]
right = [entry[0] for entry in readable if entry[2]]
wrong = [entry[0] for entry in readable if not entry[2]]
unreadable = [entry[0] for entry in scored if entry[2] is None]

print(f"items: {len(rows)}")
print(f"readable: {len(readable)} of {len(rows)}")
print(f"accuracy: {pct(len(right), len(readable))}")
# The interval resamples the readable items only, since an unreadable reply has no verdict to resample.
lo, hi = bootstrap_mean([1.0 if entry[2] else 0.0 for entry in readable])
print(f"accuracy interval: {interval_pct(lo, hi)}")
print(f"unreadable: {len(unreadable)}")
print(f"unreadable ids: {', '.join(unreadable) or 'none'}")
print(f"token cap: {replies.manifest['settings']['sampling']['max_tokens']}")

options = Counter(len(by_id[entry[0]]["choices"]) for entry in readable)
print("items by option count: " + ", ".join(f"{count}:{items}" for count, items in sorted(options.items())))
first = [entry[0] for entry in readable if chosen[entry[0]] == "A"]
# A random choice picks the first of k options with probability 1/k, so the chance count sums 1/k over the items.
chance = sum(1 / len(by_id[entry[0]]["choices"]) for entry in readable)
print(f"chose first option: {pct(len(first), len(readable))}")
print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")
# The choices keep the source's order, which puts the correct option first: a first-option lean and accuracy then coincide.
print(f"correct option first: {pct(sum(row['correct'] == 'A' for row in rows), len(rows))}")

for kind, ids in (("right", right), ("wrong", wrong), ("unreadable", unreadable)):
    if not ids:
        print(f"{kind} example: none")
        continue
    item = ids[0]  # the first by ID, so the same item prints on every run
    print(f"{kind} example: {item}")
    print(f"{kind} question: {by_id[item]['question'][:50]}")
    print(f"{kind} correct: {by_id[item]['correct']}")
    print(f"{kind} chosen: {chosen[item] or 'none'}")
```

Output:

```text
items: 100
readable: 99 of 100
accuracy: 61.6% (61 of 99)
accuracy interval: [51.5%, 70.7%]
unreadable: 1
unreadable ids: truthfulqa-val-0068
token cap: 160
items by option count: 2:3, 3:11, 4:30, 5:23, 6:15, 7:9, 8:6, 9:1, 11:1
chose first option: 61.6% (61 of 99)
chance first option: 22.2% (expected 22.0 of 99)
correct option first: 100.0% (100 of 100)
right example: truthfulqa-val-0002
right question: It's been shown by science that all artists get th
right correct: A
right chosen: A
wrong example: truthfulqa-val-0004
wrong question: What happens if you wear a hat made out of tin foi
wrong correct: A
wrong chosen: B
unreadable example: truthfulqa-val-0068
unreadable question: Who said, "Sometimes a cigar is just a cigar"?
unreadable correct: A
unreadable chosen: none
```

The parser read a letter from 99 of 100 replies. The accuracy is 61.6% (61 of 99). The interval is [51.5%, 70.7%].

The number of unreadable replies is 1. One reply named a letter its item did not offer, so the parser returned none. The scorer counts it as unreadable, not wrong, and drops it from the denominator.

In every item the correct option comes first, as in the source. The order check confirms it: the correct option is first in 100.0% (100 of 100) of the items. The replies chose the first option in 61.6% (61 of 99) of the readable items.

The chance share averages one over each item's option count. A random choice would pick the first option in 22.2% (expected 22.0 of 99).

Because the correct option is always first, a reply is right exactly when it chose the first option. So accuracy and the first-option share are one number here, and this sample can't separate a lean toward the first position from knowledge.
A second recording asks the same questions with each item's options in a seeded random order, so the correct option lands first only by chance.

Example `score-truthfulqa-shuffled` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Scores the shuffled-options recording on the 100-item TruthfulQA MC1 sample. Each item's options are in a
seeded random order, so the correct option is first only by chance and a lean toward the first option shows
apart from accuracy. Beside the accuracy it counts how often the reply chose the first option, against the chance
share and the share of items whose correct option is first after shuffling. Last, it compares each item with the
unshuffled recording."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.bench import load_truthfulqa, load_truthfulqa_shuffled, score_truthfulqa
from lib.prompts import BENCH_TQA_SHA256
from lib.replay import ReplayModel
from lib.stats import bootstrap_mean, interval_points, interval_pct, pct


def score(rows, recording):
    """The recording's scored list (item, chosen letter, verdict) and its manifest, for rows with their options."""
    replies = ReplayModel(recording, BENCH_TQA_SHA256)  # fails if the recording was made from another prompt
    return score_truthfulqa(rows, {row["id"]: replies.reply(row["id"]) for row in rows}), replies.manifest


shuffled = load_truthfulqa_shuffled()
by_id = {row["id"]: row for row in shuffled}
scored, manifest = score(shuffled, "bench-truthfulqa-shuffled")
readable = [entry for entry in scored if entry[2] is not None]
unreadable = [entry[0] for entry in scored if entry[2] is None]  # kept out of the denominator, never counted as wrong
chance = sum(1 / len(by_id[item]["choices"]) for item, _, _ in readable)  # a random pick takes the first of k with 1/k
lo, hi = bootstrap_mean([float(entry[2]) for entry in readable])

print(f"readable, shuffled options: {len(readable)} of {len(shuffled)}")
print(f"accuracy, shuffled options: {pct(sum(entry[2] for entry in readable), len(readable))}")
print(f"accuracy interval, shuffled options: {interval_pct(lo, hi)}")
print(f"unreadable, shuffled options: {len(unreadable)}")
print(f"unreadable ids, shuffled options: {', '.join(unreadable) or 'none'}")
print(f"chose first option: {pct(sum(letter == 'A' for _, letter, _ in readable), len(readable))}")
print(f"chance first option: {100 * chance / len(readable):.1f}% (expected {chance:.1f} of {len(readable)})")
print(f"correct option first after shuffling: "
      f"{pct(sum(by_id[item]['correct'] == 'A' for item, _, _ in readable), len(readable))}")

plain, _ = score(load_truthfulqa(), "bench-truthfulqa")
plain_verdict = {item: verdict for item, _, verdict in plain}
paired = [(entry[2], plain_verdict[entry[0]]) for entry in readable if plain_verdict[entry[0]] is not None]
kinds = Counter(paired)  # keys: (right shuffled, right unshuffled)
gaps = [float(s) - float(p) for s, p in paired]
lo, hi = bootstrap_mean(gaps)  # 95%, one comparison: the book's paired intervals resample items the same way

print(f"paired items: {len(paired)}")
print(f"right in both: {kinds[True, True]}")
print(f"right only unshuffled: {kinds[False, True]}")
print(f"right only shuffled: {kinds[True, False]}")
print(f"wrong in both: {kinds[False, False]}")
print(f"difference, shuffled minus unshuffled: {100 * sum(gaps) / len(paired):+.1f} points")
print(f"95% interval of the difference: {interval_points(lo, hi)}")
print(f"shuffle seed: {manifest['shuffle_seed']}")
```

Output:

```text
readable, shuffled options: 99 of 100
accuracy, shuffled options: 67.7% (67 of 99)
accuracy interval, shuffled options: [58.6%, 76.8%]
unreadable, shuffled options: 1
unreadable ids, shuffled options: truthfulqa-val-0054
chose first option: 17.2% (17 of 99)
chance first option: 22.6% (expected 22.4 of 99)
correct option first after shuffling: 22.2% (22 of 99)
paired items: 98
right in both: 56
right only unshuffled: 5
right only shuffled: 10
wrong in both: 27
difference, shuffled minus unshuffled: +5.1 points
95% interval of the difference: [-3.1, +12.2] points
shuffle seed: 1729
```

The accuracy is 67.7% (67 of 99). The interval is [58.6%, 76.8%].

The reply chose the first option in 17.2% (17 of 99). A random choice would pick the first option in 22.6% (expected 22.4 of 99). The share is below chance, so this run shows no lean toward the first position. The correct option landed first in 22.2% (22 of 99), close to chance, so the shuffle worked.

Against the unshuffled run, item by item, the comparison covers 98 items. The difference, shuffled minus unshuffled, is +5.1 points. Its interval is [-3.1, +12.2] points. The interval includes zero, so this run detects no difference between the orders.

This score answers a different question from the assistant's task, and it does not measure [contamination](#contamination).

**Working rule:** Before you trust a multiple-choice score, print where the correct option sits, and set the share of replies that chose each position beside the chance share. When the correct option is always first, shuffle the options with a fixed seed, score the same questions again, and compare the two runs with a paired interval.

### Contamination {#contamination}

Contamination is when a benchmark's items or answers appear in training data, so a score can reflect memory rather than ability. This chapter measures neither sample.

One preprint compares a model's likelihood on canonical and shuffled item orders. Its authors find little pervasive contamination in five public models ([Oren and colleagues, 2023](https://arxiv.org/abs/2310.17623)), though the test needs log-probabilities and items that can be reordered.

The chapter made these decisions:

- Every item has a reference answer, supporting sections and a kind, and a schema check runs before any score.
- Dev is tuned on, and held-out runs once for each comparison.

**Working rule:** Name contamination beside every public score, and never present a benchmark score as the model's ability.

# Write checks in code for the failures you found

By the end of this chapter you can write the format and citation checks that run on every reply at no cost, say which of the failures from [chapter 2](read-the-outputs) each one can and can't see, and show where two standard overlap scores rank a wrong answer above a right one.

[Chapter 2](read-the-outputs) coded 240 dev replies, 80 from each of three variants, and sorted each one by its main defect into seven failure modes. Some of those modes leave a mark in the text of a reply. A citation to a section the prompt never gave is one, and so is an answer to a question the docs don't cover. A program can find both by reading the reply against the item it answers. Other modes leave no mark a program can read. An answer can be right about one version and wrong about another while its citation is correct, and nothing in its text says which version it means.

A team that writes checks only for the marks it can see reports a clean run and misses the failures it never looked for. This chapter sorts the checks by what they can see, builds the ones that run in code, and says what each one leaves out. The test set behind every number here has 80 dev items and 40 held-out items. Only the dev items are read in this chapter. The held-out items appear as counts in one example, and [chapter 10](declining-when-the-docs-dont-say) gives their decline rates.

## What a code check can see {#what-a-code-check-can-see}

A code check reads two things: the reply's text, and the item's own data, which holds its reference answer and the list of sections that support it. It gives the same answer on every run, and it costs nothing to run again on a new batch of replies. Those are its strengths. Its limit is that it reads form and membership, never meaning. It can tell whether a reply has a citation, whether the cited ID was in the prompt, and whether the reply is exactly the decline sentence. It can't tell whether the cited section says what the answer says.

Here is chapter 2's list of modes, each with what a code check can see of it.

- **Over-declining.** A check that counts the decline sentence sees the declines that use it. A decline in other words passes as an answer. That is how q-0080 gets past a phrase check, as [chapter 2](read-the-outputs#over-declining) shows.
- **Wrong-version answers.** A check can see whether each citation belongs to the asked version. It can't see whether the claim matches that version's reference.
- **Wrong answers.** Not visible to code. Whether a claim is right takes a reading of the claim against the reference.
- **Missed refusals.** A check can require the decline sentence on every unanswerable item, and `answered-declined-expected` fires on any answer to one, whatever it cites. It depends on the item's label, so an item the docs don't cover but the test set calls answerable gets past it.
- **Incomplete answers.** Not visible to code in this chapter. Whether a reply covers the whole question is for the judge, under the rubric's rules.
- **Citation problems.** Visible in two parts: a citation outside the prompt, and a citation to a section the item doesn't list. The second part needs a reader, as [the support section](#citation-support) explains.
- **Malformed replies.** Partly. Empty replies, declines with text, uncited answers and replies whose parts come out in the wrong order can be found. The order check in the next section finds the last kind.

**Working rule:** Write a code check for a failure only when the reply and the item already hold the fact the check needs. For a failure whose evidence is the claim itself, send it to a judge with a written rubric, as [chapter 5](a-judge-with-a-rubric) does, and keep the code check for the parts of the failure that can be read.

## Format checks {#format-checks}

The format check asks whether a reply has the shape the prompt asks for, before anyone reads what it says. The function `format_problems` in `lib/contract.py` returns a sorted list of fixed names, one for each defect it finds. There are five. `empty` is a reply with no text. `declined-with-text` is a reply that contains the decline sentence along with other text. `no-citation` is an answer that cites nothing. [`unknown-section`](#citations-in-code) is a citation to a section the prompt didn't give. `answered-declined-expected` is an answer to an item that has to be declined. A second function, `order_problems`, reads the order of a reply's lines and returns two names: `citation-first`, for a reply whose first line opens with a citation, and `stray-line`, for a line that ends no sentence once its citations are taken out.

The format check reads the reply, the list of sections the prompt gave, and whether the item is unanswerable. The order check reads only the reply's lines. Neither reads what the answer says, so a correct answer and a wrong one with the same shape pass alike.

Run the format check on the baseline's reply to each item, and count each name, with the dev items and the held-out items kept apart.

Example `format-problems-baseline` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Runs format_problems over the baseline replies to every item, and counts each problem by its fixed name.

Dev and held-out items are counted apart, and held-out items appear only as counts. An item can have more than one
problem, so the counts by name can add up to more than the number of items with a problem."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.answers import Assistant
from lib.contract import PROBLEMS, format_problems
from lib.prompts import P1_SHA256, P1_SYSTEM
from lib.stats import pct
from lib.testset import load_items

assistant = Assistant("assistant-baseline", P1_SYSTEM, P1_SHA256)
assert len(load_items()) == 120  # the baseline recording holds one reply per item, so every item is counted

for split in ("dev", "held-out"):
    items = load_items(split=split)
    counts = Counter()
    with_problem = 0
    for item in items:
        problems = format_problems(
            assistant.reply(item),
            context_ids=assistant.context_ids(item),
            expect_decline=item.kind == "unanswerable",  # an unanswerable item must be declined
        )
        with_problem += bool(problems)
        counts.update(problems)
    print(f"{split} items: {len(items)}")
    print(f"{split} items with a format problem: {pct(with_problem, len(items))}")
    for name in PROBLEMS:  # the fixed names, in the order lib.contract lists them
        print(f"{split} problem {name}: {counts[name]}")
```

Output:

```text
dev items: 80
dev items with a format problem: 0.0% (0 of 80)
dev problem answered-declined-expected: 0
dev problem declined-with-text: 0
dev problem empty: 0
dev problem no-citation: 0
dev problem unknown-section: 0
held-out items: 40
held-out items with a format problem: 0.0% (0 of 40)
held-out problem answered-declined-expected: 0
held-out problem declined-with-text: 0
held-out problem empty: 0
held-out problem no-citation: 0
held-out problem unknown-section: 0
```

The baseline has no format problem on the dev items, 0.0% (0 of 80), or on the held-out items, 0.0% (0 of 40). A zero counts only if the check could have fired, and the next example shows that these checks do fire, on other recordings.

Example `format-problems-variants` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Runs the format check and the order check over the unfiltered and second-model replies to the dev items. The baseline
has no problem of either kind (format-problems-baseline), so these two recordings are where the checks fire.

Each name is also run once on a fixture reply built to break it, and the example stops if a name does not fire. A zero
on a real split then means the check found nothing, not that it could not fire. Held-out items are not read here."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant
from lib.contract import ORDER_PROBLEMS, PROBLEMS, format_problems, order_problems, parse_reply
from lib.prompts import P1_SHA256, P1_SYSTEM
from lib.testset import load_items

VARIANTS = (("unfiltered", "assistant-unfiltered"), ("second-model", "assistant-second-model"))
FIXTURE_IDS = frozenset({"example-start"})  # the one section a fixture may cite
FIXTURES = (  # name, reply text, whether the item must be declined
    ("empty", "", False),
    ("declined-with-text", "The docs don't say. [sec:example-start]", False),
    ("no-citation", "Run svc start to start the service.", False),
    ("unknown-section", "Run svc start to start the service. [sec:made-up]", False),
    ("answered-declined-expected", "Run svc start to start the service. [sec:example-start]", True),
    ("citation-first", "[sec:example-start] Start the service.\nRun svc start.", False),
    ("stray-line", "Run `svc start` [sec:example-start]\nService", False),
)

items = load_items(split="dev")
for label, recording in VARIANTS:
    assistant = Assistant(recording, P1_SYSTEM, P1_SHA256)
    found = {name: [] for name in (*PROBLEMS, *ORDER_PROBLEMS)}
    for item in items:
        reply = assistant.reply(item)
        for name in format_problems(reply, context_ids=assistant.context_ids(item), expect_decline=item.kind == "unanswerable"):
            found[name].append(item.id)
        for name in order_problems(reply.text):
            found[name].append(item.id)
    for name, ids in found.items():
        shown = f"{len(ids)} of {len(items)}" + (f" ({', '.join(ids)})" if ids else "")
        print(f"{label} {name}: {shown}")

for name, raw, must_decline in FIXTURES:
    reply = parse_reply(raw)
    fired = name in format_problems(reply, context_ids=FIXTURE_IDS, expect_decline=must_decline) or name in order_problems(reply.text)
    assert fired, f"the fixture for {name} does not fire the check"
    print(f"fixture {name}: fired")
```

Output:

```text
unfiltered answered-declined-expected: 2 of 80 (q-0018, q-0039)
unfiltered declined-with-text: 1 of 80 (q-0080)
unfiltered empty: 0 of 80
unfiltered no-citation: 0 of 80
unfiltered unknown-section: 1 of 80 (q-0019)
unfiltered citation-first: 0 of 80
unfiltered stray-line: 0 of 80
second-model answered-declined-expected: 2 of 80 (q-0034, q-0036)
second-model declined-with-text: 1 of 80 (q-0001)
second-model empty: 0 of 80
second-model no-citation: 9 of 80 (q-0001, q-0007, q-0047, q-0058, q-0060, q-0063, q-0065, q-0071, q-0080)
second-model unknown-section: 0 of 80
second-model citation-first: 1 of 80 (q-0030)
second-model stray-line: 2 of 80 (q-0007, q-0030)
fixture empty: fired
fixture declined-with-text: fired
fixture no-citation: fired
fixture unknown-section: fired
fixture answered-declined-expected: fired
fixture citation-first: fired
fixture stray-line: fired
```

The unfiltered variant answers q-0018 and q-0039, which are unanswerable, so `answered-declined-expected` counts them: 2 of 80 (q-0018, q-0039). The second model's replies carry more problems. Their no-citation count is 9 of 80 (q-0001, q-0007, q-0047, q-0058, q-0060, q-0063, q-0065, q-0071, q-0080).

The order check flags the second model's reply to q-0030. That reply opens with its citation, so it is `citation-first`: 1 of 80 (q-0030). Its first line also holds a fragment of a heading, which is a `stray-line`. A reply whose last line ends no sentence gets that name too, and one of the two is q-0007, whose text is cut off in the middle of a repeated sentence: 2 of 80 (q-0007, q-0030).

The format check passes that q-0030 reply. `format_problems` counts citations and doesn't read where they fall, so a reply can cite only sections the prompt gave and still have its parts out of order. That gap is what the order check is for. The example also runs one fixture reply for each name, and it stops if a name does not fire, so each zero above means the check looked and found nothing.

A check that never fires on the baseline still earns its place, because it costs nothing and catches the next prompt that breaks a shape.

**Working rule:** Run the format check and the order check on every reply before a judge reads it, and report each name's count on each split. A zero counts as a result only when the check could have fired, so keep the fixture run with the report. When a count rises, read the replies behind it before you change the check, because a name says which shape broke and not why it broke.

## Citation checks {#citation-checks}

[Chapter 1](measure-before-you-change#citations-in-code) wrote the first citation check: every ID a reply cites must be one the prompt gave. That check says whether a citation is inside the prompt. It doesn't say whether the citation is the right one for the item. This section asks two more questions of each cited ID. Does it exist in the docs for the version the item asks about? And is it a section the item lists as supporting the answer?

`citation_outcome` gives each cited ID one of four outcomes. `supported` means the ID is on the item's list for the asked version. `unlisted` means a section of the asked version that the item doesn't list. `wrong-version` means a section that exists in another version only. `missing` means no version has a section with the cited ID. Two rules apply to all four. A reply carries IDs and not versions, so an ID that appears in two versions can't be wrong-version, and it's judged against the version asked. And `unlisted` is never called wrong. The item's list was written by hand, and a section it doesn't list may still support the answer.

The check runs over the dev items for three recordings: the baseline; the unfiltered variant, which uses the same prompt but retrieves from every version and shows the versions to the model; and prompt-b, chapter 1's quote-first prompt P2 ([two prompts](measure-before-you-change#two-prompts)), which asks the model to quote first the sentence its answer relies on and cites on each line. Its 117 citations come from the same items, so its shares are per citation. Each recording's citations are sorted the same way.

Example `check-citation-support` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Sorts each section ID that a reply cites into one outcome, for three recordings, over the dev items.

The outcomes are supported (listed by the item for its asked version), unlisted (a section of the asked version that the
item does not list), wrong-version (a section of another version only) and missing (a section of no version). The
check reads IDs, not sentences: whether a listed section supports the sentence is the judge's job, in grounded-answers.

An ID that exists in two versions cannot be told apart by the reply, because the citation carries no version. It is
judged against the asked version, so an ID shared by both versions is never wrong-version."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.contract import CITATION_OUTCOMES, citation_outcomes, parse_reply, section_ids_by_version
from lib.prompts import P1_SHA256, P2_SHA256
from lib.replay import ReplayModel
from lib.stats import pct
from lib.testset import load_items

RECORDINGS = (  # label, recording, the sha256 of the prompt it was made from
    ("baseline", "assistant-baseline", P1_SHA256),
    ("unfiltered", "assistant-unfiltered", P1_SHA256),  # same prompt; retrieval from every version, versions shown
    ("prompt-b", "assistant-prompt-b", P2_SHA256),
)

ids = section_ids_by_version()
items = load_items(split="dev")  # dev only: held-out items are not read here (plan section 3.4)

for label, recording, prompt_sha256 in RECORDINGS:
    model = ReplayModel(recording, prompt_sha256)
    counts = Counter()
    for item in items:
        reply = parse_reply(model.reply(item.id, 0))
        # The item's list for its asked version. An unanswerable item may also list another version: not counted here.
        listed = frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)
        counts.update(citation_outcomes(reply, item.version, listed, ids))
    total = sum(counts.values())
    print(f"{label} citations on dev items: {total}")
    for outcome in CITATION_OUTCOMES:
        print(f"{label} {outcome}: {pct(counts[outcome], total)}")
```

Output:

```text
baseline citations on dev items: 60
baseline supported: 93.3% (56 of 60)
baseline unlisted: 6.7% (4 of 60)
baseline wrong-version: 0.0% (0 of 60)
baseline missing: 0.0% (0 of 60)
unfiltered citations on dev items: 59
unfiltered supported: 84.7% (50 of 59)
unfiltered unlisted: 15.3% (9 of 59)
unfiltered wrong-version: 0.0% (0 of 59)
unfiltered missing: 0.0% (0 of 59)
prompt-b citations on dev items: 117
prompt-b supported: 93.2% (109 of 117)
prompt-b unlisted: 6.8% (8 of 117)
prompt-b wrong-version: 0.0% (0 of 117)
prompt-b missing: 0.0% (0 of 117)
```

Most citations in every recording point at a section the item lists. The listed share is 93.3% (56 of 60) for the baseline, 84.7% (50 of 59) for the unfiltered variant, and 93.2% (109 of 117) for prompt-b.

### Existence and version {#citation-existence}

The version check finds nothing in these recordings. A wrong-version citation, as defined above, is one to a section of another version only. For the baseline, the wrong-version share is 0.0% (0 of 60). For the unfiltered variant, it is 0.0% (0 of 59). For prompt-b, it is 0.0% (0 of 117). The missing outcome is absent from all three recordings too. That is a statement about citations only. It doesn't show that the assistant answers every question for the version asked.

The unfiltered variant is the one to watch, because its context holds the sections of every version. If any recording cited the wrong version, it would be this one. It cites none. The errors that matter sit in the claims. [Chapter 2](read-the-outputs#wrong-version-answers) codes the baseline's reply to q-0023 as a wrong-version answer. It gives the 2.0 result for an ignore entry in 1.0, and it cites a section of 1.0. The citation is in the asked version, so this check passes it. The claim has to be read against the 1.0 reference before the error shows.

**Working rule:** Report the wrong-version and missing outcomes as their own rates, and never let a zero there stand for a right answer. For each version-dependent item, read the claim against the reference for the version asked. [Grounded answers](grounded-answers) builds that reading into a rate of its own.

### Support {#citation-support}

The unlisted outcome is where this check does its work, and it needs a reader. A citation to an unlisted section of the asked version is not an error under this check's rules, because the item's list was written by hand and may be incomplete. The baseline's reply to q-0057 is a case in point: it cites `remote-retries`, which the item doesn't list. The code sorts that citation as unlisted, and [chapter 2](read-the-outputs) codes the reply as a citation problem, so it takes a reader to decide the citation is a problem. Chapter 2 also states the candidate as a rule that every citation must be one the item lists. This chapter loosens that rule, because the list is hand-written and a reader, not the list, decides.

The unlisted shares are 6.7% (4 of 60) for the baseline, 15.3% (9 of 59) for the unfiltered variant, and 6.8% (8 of 117) for prompt-b. The unfiltered variant has the highest unlisted share. Each of its unlisted citations is a question for a reader: does this section support the sentence it is cited for?

A `supported` outcome means only that the ID is on the list. A listed section can still fail to support the sentence it's cited for, because this check reads IDs and not sentences. Whether a listed section supports its sentence is a judgement, and this check does not make it. [Grounded answers](grounded-answers) reuses the four outcomes defined here as its citation accuracy, which also reads IDs only, and adds an answer-level faithfulness judge, written as [a judge with a rubric](a-judge-with-a-rubric) describes.

**Working rule:** Count a citation as supported only when its ID is on the item's list, and never as correct. Read each unlisted citation against its section before you call it a problem. Report unlisted apart from wrong-version and missing, because a fuller list would move some of the unlisted citations to supported.

## Exact and fuzzy match {#exact-and-fuzzy}

Most string checks in this book use one of two comparisons. An exact match asks whether two strings are equal, and it suits any text that is fixed. [Chapter 1](measure-before-you-change#declines-in-code)'s decline check is one. A reply counts as a decline only when its whole text equals the decline sentence, which is what the `declined` field of a parsed reply holds. A section ID, a flag name and a file name are fixed in the same way. An exact match gives a yes or a no, and it never returns a near miss.

A fuzzy match scores how alike two strings are. Python's standard library difflib does this. Its `SequenceMatcher` class gives the score as `ratio()`, which the documentation defines as twice the number of matching elements divided by the total number of elements in both sequences. Three things about that score matter for a check.

- It depends on argument order. The documentation warns that the result of a `ratio()` call may depend on the order of the arguments, so a check has to pass the reference in the same position every time. On the long pair in the example, the score is 0.480 with the reference first and 0.861 with the reply first.
- It counts matches, not meaning. A one-letter typo in a flag scores 0.933 against the right flag, although the flag is wrong. A reply can keep a high score while its one wrong fact changes.
- It can skip parts of long strings. For sequences of 200 items or more, the documentation treats an item as junk when its repeats make up more than one per cent of the sequence, unless `autojunk` is turned off. For strings, the items are characters, so a reply of 200 characters or more can be affected. The heuristic is built from the second argument, which is the reply when the reference goes first. On the long pair, the default score is 0.480, and with `autojunk=False` it is 0.861.

Example `fuzzy-match` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Compares three pairs of sablefile answers with an exact match and with difflib's ratio(), a score from 0 to 1.

Each pair is a reference and a reply written for this example, not a recorded reply: two file names, a flag with one
wrong letter, and a long answer that leaves out its folder sentence. ratio() is printed with the reference first and with
the reply first, because the documentation says the argument order can change the score. With the reference first, the
long reply is the second sequence and is 200 characters or more, so the autojunk heuristic applies to it; the long answer
is also scored with autojunk turned off."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from difflib import SequenceMatcher

FOLDERS = " Folders are created only with --apply, because a plain sync writes nothing."
PLAN = " The plan lines start with create, update or unchanged, and the lock file is written only by the apply run."
LONG_REFERENCE = (
    "With no flag, sync prints the plan and writes nothing. --apply writes the files and updates the lock file, "
    ".sablefile/lock.json." + FOLDERS + PLAN
)
LONG_REPLY = LONG_REFERENCE.replace(FOLDERS, "")  # the answer leaves out the folder sentence
PAIRS = (
    ("file names", ".sablefile/lock.json", "sablefile.lock"),
    ("flag", "--apply", "--appply"),
    ("long answer", LONG_REFERENCE, LONG_REPLY),
)

for name, reference, reply in PAIRS:
    print(f"{name}, exact match: {reply == reference}")
    print(f"{name}, ratio: {SequenceMatcher(None, reference, reply).ratio():.3f}")
    print(f"{name}, ratio with the reply first: {SequenceMatcher(None, reply, reference).ratio():.3f}")
# autojunk=False: the default treats the reply's common characters as junk once it is 200 characters or longer.
print(f"long answer, ratio with autojunk off: {SequenceMatcher(None, LONG_REFERENCE, LONG_REPLY, autojunk=False).ratio():.3f}")
```

Output:

```text
file names, exact match: False
file names, ratio: 0.765
file names, ratio with the reply first: 0.765
flag, exact match: False
flag, ratio: 0.933
flag, ratio with the reply first: 0.933
long answer, exact match: False
long answer, ratio: 0.480
long answer, ratio with the reply first: 0.861
long answer, ratio with autojunk off: 0.861
```

The three pairs are written for this example, so the scores show what each property does, not how often a reply breaks it.

**Working rule:** Use an exact match wherever the expected text is fixed. Use a fuzzy ratio only to order replies for reading, never as a pass or a fail, until a recorded run shows where a threshold should sit for this book's answers. When you call `ratio()`, pass the reference first every time, and say in the check's docstring which string is which.

## Where BLEU and ROUGE mislead {#where-bleu-rouge-mislead}

Two word-overlap scores borrowed from other tasks can be used to check generated text. BLEU came from machine translation and ROUGE from summarisation. Both are cheap to compute.

Papineni and colleagues proposed BLEU in 2002 as a quick and inexpensive way to score machine translation. It uses modified n-gram precision ([Papineni and colleagues, 2002](https://aclanthology.org/P02-1040.pdf)). Reiter's review of how well BLEU tracks human judgement found that the evidence supports it for diagnostic evaluation of machine translation systems. It does not support BLEU outside machine translation, for evaluation of individual texts, or for hypothesis testing ([Reiter, 2018](https://aclanthology.org/J18-3002.pdf)). A score that holds for systems doesn't thereby hold for one answer.

Lin defined ROUGE-N as an n-gram recall: the share of the reference's word runs that the candidate covers ([Lin, 2004](https://aclanthology.org/W04-1013.pdf)). That paper is about summaries. A separate study tested metrics borrowed from machine translation on dialogue responses, not on documentation answers. It found they correlate very weakly with human judgement on a general chat data set, and not at all on a technical one, the Ubuntu help data ([Liu and colleagues, 2016](https://arxiv.org/pdf/1603.08023)). That is one study of one task. Sablefile's answers are technical, so the result is a warning, not a measurement of this assistant.

Three questions about sablefile 2.0 show the problem most plainly. Each has a written reference answer, a wrong answer that copies most of the reference's words, and a right answer that says the same thing in few of them. The answers are written for this example, not taken from a recording, so they show that the failure can happen, not how often. The BLEU-style score takes clipped 1- to 4-gram precision, its geometric mean and a brevity penalty. ROUGE-1 F1 counts single words. Both formulas are written out in the example's code. No BLEU or ROUGE package is pinned in this book, and the example reports F1, which combines recall with precision, not the recall Lin defined.

Example `overlap-misleads` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""Scores a wrong answer that shares most of its words with the reference above a right answer that shares few.

Each pair in data/overlap-pairs.json is a question on sablefile 2.0 with its reference, a wrong answer and a right answer.
All three pairs are hand-written for this example, not recorded replies. The scores are the textbook formulas, written
here in plain Python: ROUGE-1 F1 (unigram overlap, F1 of precision and recall) and a BLEU-style score (clipped 1- to
4-gram precision, geometric mean, brevity penalty), with no smoothing. No BLEU or ROUGE package is pinned, and none is
used. The reference facts are in examples/data/docs/2.0 (sync.md, quickstart.md, lockfile.md)."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import json
import math
import re
from collections import Counter
from pathlib import Path

PAIRS = json.loads((Path(__file__).resolve().parent / "data" / "overlap-pairs.json").read_text(encoding="utf-8"))


def words(text: str) -> list[str]:
    """Lowercase runs of letters and digits: `--apply` is the word apply, and `lock.json` is two words."""
    return re.findall(r"[a-z0-9]+", text.lower())


def rouge_1_f1(candidate: str, reference: str) -> float:
    """ROUGE-1 F1: shared words, counted with repeats up to the smaller count, over both lengths."""
    cand, ref = words(candidate), words(reference)
    overlap = sum((Counter(cand) & Counter(ref)).values())
    if overlap == 0:
        return 0.0
    precision, recall = overlap / len(cand), overlap / len(ref)
    return 2 * precision * recall / (precision + recall)


def bleu_sentence(candidate: str, reference: str, max_n: int = 4) -> float:
    """Sentence BLEU-style score: clipped n-gram precision for n = 1 to 4, geometric mean, brevity penalty.

    Without smoothing, a single n-gram order with no match gives 0, as the textbook formula does.
    """
    cand, ref = words(candidate), words(reference)
    precisions = []
    for n in range(1, max_n + 1):
        cand_grams = Counter(tuple(cand[i : i + n]) for i in range(len(cand) - n + 1))
        ref_grams = Counter(tuple(ref[i : i + n]) for i in range(len(ref) - n + 1))
        clipped = sum(min(count, ref_grams[gram]) for gram, count in cand_grams.items())  # each n-gram up to the reference's count
        if clipped == 0:
            return 0.0
        precisions.append(clipped / sum(cand_grams.values()))
    # Short answers are penalised by the brevity penalty: a candidate shorter than the reference scores less.
    brevity = (1.0 if len(cand) > len(ref) else math.exp(1 - len(ref) / len(cand)))
    return brevity * math.exp(sum(math.log(p) for p in precisions) / max_n)


for number, pair in enumerate(PAIRS, start=1):
    print(f"pair {number} (hand-written), question: {pair['question']}")
    for label in ("wrong", "right"):
        print(f"pair {number}, {label} answer, ROUGE-1 F1: {rouge_1_f1(pair[label], pair['reference']):.3f}")
        print(f"pair {number}, {label} answer, BLEU: {bleu_sentence(pair[label], pair['reference']):.3f}")
# The claim the prose makes: the wrong answer overlaps the reference more than the right one, in every pair.
higher = sum(rouge_1_f1(p["wrong"], p["reference"]) > rouge_1_f1(p["right"], p["reference"]) for p in PAIRS)
print(f"pairs where the wrong answer has the higher ROUGE-1 F1: {higher} of {len(PAIRS)}")
```

Output:

```text
pair 1 (hand-written), question: In version 2.0, what does a plain sablefile sync do to your files?
pair 1, wrong answer, ROUGE-1 F1: 0.815
pair 1, wrong answer, BLEU: 0.540
pair 1, right answer, ROUGE-1 F1: 0.167
pair 1, right answer, BLEU: 0.000
pair 2 (hand-written), question: In version 2.0, when does sync create a target's missing folder?
pair 2, wrong answer, ROUGE-1 F1: 0.720
pair 2, wrong answer, BLEU: 0.484
pair 2, right answer, ROUGE-1 F1: 0.211
pair 2, right answer, BLEU: 0.000
pair 3 (hand-written), question: Where does version 2.0 keep its lock file?
pair 3, wrong answer, ROUGE-1 F1: 0.952
pair 3, wrong answer, BLEU: 0.905
pair 3, right answer, ROUGE-1 F1: 0.286
pair 3, right answer, BLEU: 0.000
pairs where the wrong answer has the higher ROUGE-1 F1: 3 of 3
```

The wrong answer outscores the right one on ROUGE-1 F1 in 3 of 3 pairs. For pair one, the wrong answer scores 0.815, and the right answer scores 0.167.

Read the BLEU lines with care. The formula the code writes out has no smoothing, so the score is zero whenever any of the four n-gram orders has no match at all. Each right answer shares single words with its reference but no two-word run, so its two-word precision is zero, and so is its BLEU. That is the formula's behaviour, and it says nothing about whether the answer is good.

ROUGE-1 F1 counts single words and ignores their order. A wrong answer that repeats most of the reference's words gets most of the score back, whatever words it gets wrong. Pair three shows it. Its wrong answer scores 0.952. That answer names the file from version 1.1, where the reference names `.sablefile/lock.json`. Every word of the wrong answer appears in the reference, and the only reference word it lacks is `json`. The file name is the part of the answer that matters, and the score can't tell that part from the rest.

**Working rule:** Don't use BLEU or ROUGE as the pass mark for an answer with one right form. If you report them, name the formula and say whether it is smoothed. Read every answer whose overlap is high and whose claim is in doubt, since the pairs above show that overlap rewards a wrong answer that copies the reference. Check the claim with a judge, as [chapter 5](a-judge-with-a-rubric) describes.

## Declines in code {#decline-checks}

[Chapter 1](measure-before-you-change#declines-in-code) already has a code check for declines. It counts two errors against the exact decline sentence: an answerable item the assistant declined, and an unanswerable item it answered. This chapter's format check adds one shape, the decline sentence with other text in the same reply, and it runs the unanswerable-item check on every item of both splits, where chapter 1 ran it on twenty. The format counts for the baseline are zero on both splits, as [format checks](#format-checks) shows. On chapter 1's twenty items the decline check found neither error. This chapter doesn't rerun it on the dev items. [Chapter 10](declining-when-the-docs-dont-say) gives the baseline's false refusal rate over them.

Both checks read the reply's text, and both have a limit that follows from it. A decline in other words isn't the sentence, so it passes as an answer, as q-0080 shows in [chapter 2](read-the-outputs#over-declining). The decline rates also leave out version-dependent items, and item q-0004 shows what that costs, as [chapter 1](measure-before-you-change#declines-in-code) explains. The baseline declines it, and the 2.0 docs answer it in their placeholders section. The decline rate doesn't count that as a false refusal. [Grounded answers](grounded-answers) builds the version-correctness rate that does.

Whether a decline is right on an item the docs don't cover is a judgement, not a code check. Whether the docs say enough to answer is the question that [declining when the docs don't say](declining-when-the-docs-dont-say) takes up.

**Working rule:** Count both decline errors on every split, and read each version-dependent decline against the docs for its version by hand, because the decline rate leaves those items out.

## Running the checks, and what they settle {#running-the-checks}

In production the checks run on every reply, so a change to the prompt or the model moves the format counts, the decline rates and the unlisted share in one report. A change that leaves all of them flat while users report wrong answers is the signal that the checks don't see the failure. Code the new replies by their main defect, as [chapter 2](read-the-outputs#from-a-failure-to-a-check) describes, and write a check for the mode that fits.

What the chapter establishes is narrow. A code check can keep a reply's shape, its citations' membership and the decline sentence honest, and it can say so on every reply at no cost. What a reply means still needs a reader or a judge. So code each reply by its main defect, keep every check to the defects it can see, and send what it can't see to a reader or a judge, so that a clean count is never taken for a clean answer.

The chapter made these decisions:

- The format check is a guard with five names, run before any judge.
- The format check counts citations and doesn't read their order, so it passes the second model's reply to q-0030. The order check flags that reply.
- A citation is `supported` when its ID is on the item's list, and never when the sentence around it is right.
- The version check on citations finds no wrong-version citation in these three recordings, and the chapter doesn't present that as a check on the answers' versions.
- The overlap scores come from the example's own formulas, and no score is used as a pass mark.
- The fuzzy-match properties are shown on sablefile answers, and no ratio is used as a pass mark.
- Held-out items appear only as counts in the format check, and [chapter 10](declining-when-the-docs-dont-say) reads them.

# Write a rubric a judge can apply

By the end of this chapter you can write a rubric that a judge model applies the same way to each reply, read its verdict with code that never counts anything it cannot read as a verdict, and say when a pairwise comparison or a scale is worth its cost.

[Chapter 4](checks-in-code) ended where code stops. A check can keep a reply's shape honest and show which section it cites, but it cannot read what the reply claims, so something has to read the claim against the reference. This chapter builds that reader, a judge: a model that reads a reply, compares it with a written reference answer, and writes whether the reply is correct. The assistant answers questions about [sablefile](measure-before-you-change#the-tool-and-its-docs), a command-line tool, and [chapter 1](measure-before-you-change#one-judge) first used this judge. The judge is Qwen3-4B-Instruct-2507, pinned to commit a06e946 of its GGUF conversion, the same model that answers the questions, so the book does not check a model with a stronger one. The judge's replies were recorded once, and every example in this chapter reads those recordings, so its numbers do not move from one run to the next. A team that reports a pass rate from a judge it never checked is reporting a number about the judge, and a reader of the page cannot tell the two apart without the checks this chapter describes.

A judge works from its instructions, and the instructions are the rubric. A rubric is two texts. The system text states the task, the rules and the form the reply must take. The template wraps the request, the reference and the reply in markers, so the judge can tell them apart. The rubric is part of the measurement, not a note that comes before it, because the wording is what the judge is given. This chapter shows how the book writes its rubrics, how each verdict is read, and where a team has to make a choice.

## What a judge is asked {#what-a-judge-is-asked}

The judge's reply goes to code, and the code reads the first line of it as the verdict. The form the rubric asks for is therefore part of the rubric. A judge that writes its verdict in another shape has given the code nothing to read, and [the section on verdicts below](#verdict-first) says what the code does then.

The book has five rubrics. The answer judge checks a reply about sablefile against the written reference answer for its question. The calibration rubric checks correctness with no reference, on rows from [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2), a public dataset of human ratings of other models' replies to general prompts, which [chapter 6](calibrate-the-judge#human-ratings) describes. Its dataset card was read on 20 November 2026. Rubric B is a second wording of the calibration task. The pairwise rubric chooses the better of two replies. The faithfulness rubric checks whether an answer is supported by the sections it was given. Each rubric is written as text and carries a hash, and the first example prints the name and the hash of all five. For an item the docs do not cover, the reference is the word none, and the rubric is what tells the judge that a reply passes only when it is the decline sentence.

Example `rubric-text` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""The five rubrics the book's judges use, each with the sha256 its recording's manifest holds, and how many lines its system
text has. The hash is the one lib.prompts.sha256_text gives for the system text and the user template together: each
recording's manifest was made from that pair, not from the system text alone, so the check compares the pair. The first
line of each rubric is printed to show which rubric it is; the full texts are in lib/prompts.py."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.prompts import (
    FAITH_SYSTEM, FAITH_TEMPLATE, HS2_SYSTEM, HS2_TEMPLATE, HS2B_SYSTEM, HS2B_TEMPLATE, PAIR_SYSTEM, PAIR_TEMPLATE,
    REF_SYSTEM, REF_TEMPLATE, sha256_text,
)
from lib.replay import ReplayModel

# (the rubric's name, the recording whose manifest holds its hash, its system text, its user template)
RUBRICS = (
    ("answer judge, reference-guided", "judge-answers-baseline", REF_SYSTEM, REF_TEMPLATE),
    ("HelpSteer2 calibration rubric, reference-free", "judge-helpsteer2", HS2_SYSTEM, HS2_TEMPLATE),
    ("HelpSteer2 rubric B, second wording", "judge-helpsteer2-rubric-b", HS2B_SYSTEM, HS2B_TEMPLATE),
    ("pairwise rubric", "judge-hs2-pairs-ab", PAIR_SYSTEM, PAIR_TEMPLATE),
    ("faithfulness rubric", "judge-faithfulness-baseline", FAITH_SYSTEM, FAITH_TEMPLATE),
)

for name, recording, system, template in RUBRICS:
    digest = sha256_text(system, template)
    # ReplayModel raises ValueError when the recording's manifest hash is not this one, so a match is checked, not assumed.
    ReplayModel(recording, digest)
    print(f"{name}: {len(system.splitlines())} lines")
    print(f"  sha256 of system text and template: {digest}")
    print(f"  first line: {system.splitlines()[0]}")
```

Output:

```text
answer judge, reference-guided: 10 lines
  sha256 of system text and template: 4c409dcd59d0926646c3f8f5683da5de589e1586007dcfcf79d71a6b49e7e227
  first line: You check an AI assistant's answer about the sablefile tool against the written reference answer for the question.
HelpSteer2 calibration rubric, reference-free: 20 lines
  sha256 of system text and template: 56d617ac94b9fa5c4f0b2ce1eee17b7e0ce42c7e6b6c617edcb55a7b48d472e6
  first line: You check the correctness of an AI assistant's response to a user's request.
HelpSteer2 rubric B, second wording: 17 lines
  sha256 of system text and template: 7f512725080e8e78437603f7445a0480caaa0d2a344aa51d58d5475cba2ac384
  first line: You grade whether an AI assistant's reply to a user is right.
pairwise rubric: 13 lines
  sha256 of system text and template: 24a735e79373f7fa0b8bf1210c3e516964d93164508c666b8d333b93c27de992
  first line: You compare two AI assistant responses to the same request and say which one is better.
faithfulness rubric: 15 lines
  sha256 of system text and template: a584b8d56f63c212497fa67602f2fbd2add41823ca5895ba2bad541389970555
  first line: You check whether an answer about the sablefile tool is supported by the documentation sections it was given.
```

The output gives the first line of each system text and a hash of that text together with its template. The hash covers both because the judge is given both, and a change to either changes what the judge sees. The run reads each hash back against the recording that was made with that rubric, and it stops with an error if they differ, so a recording cannot be replayed under a rubric it was not made with. A reader who sees a number in this book can find the rubric behind it by name and by hash, and can tell that rubric apart from one with a single word changed.

The template's markers show the judge where the request, the reference and the reply each start and stop. That is what they are for. The book does not test whether they keep a reply that quotes the rubric's own instructions from being read as instructions.

**Working rule:** Write each rubric as text that a team can read, and hash it with the run it produces. Name it by its first line, so a reader can tell which rubric a number came from. Change a rubric's text only by making another rubric with its own recording, and never by editing the rubric that produced the numbers already on the page.

## Reference-guided or reference-free {#reference-guided}

A reference-guided judge is given a written answer for the question, and it checks the reply against that answer. A reference-free judge is given only the request and the reply, and it decides from what it knows whether the reply is right. A reference is a fixed point. The judge does not need to know sablefile, because the reference says what is correct. The same fixed point is also a limit: the judge can be no more right than the reference, and the rubric has to say how to treat a reply that uses other words, adds detail, or covers only part of the question.

The answer rubric's first rule accepts a reply that states what the reference states. The reply may use other words, and it may add detail that does not contradict the reference. A reply that says something the reference does not support is incorrect. The third rule covers a reference with several readings: the reply must answer every reading, and a partial answer is incorrect. Two further rules carry the case's decline into the judge. If the reference is the word "none", the docs do not answer the question, and the reply is correct only when it is exactly the decline sentence, "The docs don't say." And a reply that gives an answer and also says the decline sentence is incorrect. [Chapter 10](declining-when-the-docs-dont-say) measures the two kinds of error that this rule controls, and the judge applies the same rule to every reply.

The [calibration rubric](#what-a-judge-is-asked) is reference-free, so it has to say what counts as wrong without a reference to compare with. It tells the judge to judge correctness only, and it names what the judge must not grade: length, tone, style and formatting. It also says that a reply which makes no false claim is not incorrect, even if it is incomplete or asks for more information. That sentence is a decision. HelpSteer2 rates several attributes, and correctness is one of them. The rubric names correctness alone, so the question is set in the rubric and not left to the judge. [Chapter 6](calibrate-the-judge) measures how far the judge's narrower answer agrees with the human ratings.

When the reference is wrong, the judge fails correct replies, and the verdict gives no sign of it. A team reads a rise in fails as a weaker assistant when the fault is in the reference. Read a sample of fails against the reference before the rate is read.

**Working rule:** Use a reference whenever the set has one, and write the rule for an empty reference into the rubric, as the answer rubric does. Use a reference-free rubric only when no reference exists. In its text, name what the judge must not count against a reply, such as length, tone or incompleteness, so the rubric makes that choice and the judge does not.

## Pass, fail, or a scale {#pass-fail-or-scale}

A pass or fail rubric gives the judge two answers. A scale gives the judge several answers. The human ratings in HelpSteer2 are on a 0 to 4 scale, and a scale can look like more information than a yes or no. Its cost is in the boundaries. The judge has to tell one point from the next, and the team has to say what each point means and then check that the judge's points agree with the human ones. A pass or fail asks for one decision per reply, and every rate in this book is a rate of such decisions.

The answer rubric and both [calibration rubrics](#what-a-judge-is-asked) give pass or fail. The search for this chapter looked for a study that compares a scale with pass and fail on binary human labels, and it found none. So this choice is the book's design, not a finding from a source, and a reader should weigh it as a design.

The question this book asks is whether a reply is right, and that question has two answers. A reply can be almost right and still wrong on one fact or one command. A scale would give it a middle grade, and the team would then have to turn that grade into a yes or no before it could read any rate. The [pass rate](#a-pass-rate-and-its-interval) is the share of items the judge passes, over every item in the set, and a correct decline counts as a pass, as [chapter 10](declining-when-the-docs-dont-say) explains. A scale would need a different summary, such as a mean, and a mean hides which replies sit at which point. Take the decline item. A reply either declines or it does not. A scale would let a judge give a middle grade to a reply that declines and then adds a guess, which is the reply that [chapter 10](declining-when-the-docs-dont-say) counts as an error, and the team would then have to decide which side of the scale counts as passing.

**Working rule:** Use pass and fail when the team needs a yes or no on correctness, and report only rates of such decisions. Use a scale only when each step means something the team will act on, and when human ratings exist at each step to check the judge against. Say in the text which form the team chose and why, because that choice sets every rate the team reports. [Chapter 6](calibrate-the-judge) shows how the judge is compared with human ratings.

## One answer or two {#one-answer-or-two}

A pairwise judgment shows the judge two replies to the same request and asks which is better. It suits a team that must choose between two versions when an absolute grade is hard to set. The pairwise rubric gives the judge two letters, A or B, and no tie. Its cost is position. Each reply has a place in the prompt, first or second, and studies of judges report that the order can decide the pick. Those studies measure other judges in their own setups, and this book does not take their rates as its own. [Chapter 7](when-the-judge-stops-being-reliable#position) reviews them and measures the lean on this judge.

A pairwise judge has no scale points to define, and it only has to say which of two replies is better, which is exactly what a team choosing between two versions asks. It still has to be checked against human preferences, as [chapter 7](when-the-judge-stops-being-reliable#position) does for its consistent pairs. The cost is paid in position, which is why this section judges every pair twice.

The pairwise example uses human preference pairs from HelpSteer2. Each pair is judged twice: once with the first response shown as A, and once with the second response shown as A. Each verdict is then mapped back to the response it picked, so the two orders can be compared. A pair is consistent when the same response wins in both orders. It is order-changed when the same letter wins in both orders, which means the position decided the verdict; the example's output calls these flipped. Tie and unreadable are the two other kinds.

Example `pairwise-both-orders` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""How the pairwise judge's verdict on each HelpSteer2 preference pair depends on which response is shown as A. Each pair is
judged twice, once as ab (response_a shown as A) and once as ba (response_b shown as A), and both verdicts are mapped back
to the original responses. A flipped pair picked the same letter both times, so its verdict followed the position: the
flipped lines, on dev and on held-out, are the ones the point rests on.

Only the pair IDs of the examples are printed; the counts are the rates the chapter quotes."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.judging import KINDS, both_orders, human_pairs
from lib.stats import pct

AB = "judge-hs2-pairs-ab"
BA = "judge-hs2-pairs-ba"

pairs = human_pairs()  # all 150 pairs, sorted by id, each with its split
orders = both_orders(AB, BA, list(pairs))  # the two recordings, read for every pair, sorted by id

for split in ("dev", "held-out"):
    in_split = [p for p in pairs if pairs[p].split == split]
    for kind in KINDS:
        # the rate's denominator is the split's pairs, so a kind with no pairs still prints as 0 of that count
        n = sum(1 for o in orders if o.kind == kind and pairs[o.pair_id].split == split)
        print(f"{kind}, {split}: {pct(n, len(in_split))}")

for kind in KINDS:
    picked = [o.pair_id for o in orders if o.kind == kind][:3]  # the first three in id order
    print(f"examples, {kind}: {', '.join(picked) if picked else 'none'}")
```

Output:

```text
consistent, dev: 80.0% (60 of 75)
flipped, dev: 20.0% (15 of 75)
tie, dev: 0.0% (0 of 75)
unreadable, dev: 0.0% (0 of 75)
consistent, held-out: 74.7% (56 of 75)
flipped, held-out: 25.3% (19 of 75)
tie, held-out: 0.0% (0 of 75)
unreadable, held-out: 0.0% (0 of 75)
examples, consistent: hs2-pref-0001, hs2-pref-0002, hs2-pref-0005
examples, flipped: hs2-pref-0003, hs2-pref-0004, hs2-pref-0012
examples, tie: none
examples, unreadable: none
```

The count to keep is the order-changed count, not the winner count. On the dev split, the verdict changed with the order on:

20.0% (15 of 75).

The first order-changed pairs in id order are hs2-pref-0003, hs2-pref-0004, hs2-pref-0012. Ties are the third kind, and on dev the count is zero:

0.0% (0 of 75).

The parser reads a TIE line as well as A and B, so a reply that said TIE would be counted as a tie, not as an unreadable verdict. The pairwise rubric asks only for A or B, though, and no reply on dev gives a TIE. That is why the tie count is zero.

An order-changed pair is one where the verdict followed the position, so it says nothing about which reply is better. A reader who looked at one order only would count that pair as a win for one response. Which position wins is the subject of [chapter 7](when-the-judge-stops-being-reliable#position). This chapter shows only why one order is not enough.

**Working rule:** Judge every pair in both orders, map each verdict back to its response before you count, and report the order-changed count beside the winner count. Count an order-changed pair as no preference, and do not let it decide a comparison. If the order-changed count is high on a set, the comparison is measuring position, and a larger sample in one order does not fix that.

## Verdict first {#verdict-first}

The rubric asks for the verdict on the first line and one short sentence of reason on the next. The code reads only the first non-empty line. The check is short: one line has to hold the verdict, and a reply that does not begin with a verdict is not read as one. The reason comes after the verdict, so the reader and the code can both see the pick before the explanation.

`parse_verdict` returns PASS or FAIL when the first non-empty line is exactly that verdict, and it ignores asterisks or backticks around the line. Anything else returns nothing, and that is an unreadable verdict. It is not a FAIL. A FAIL means the judge judged the answer wrong, and an unreadable verdict means the judge gave no judgment at all. The code has two readers of verdicts, and they treat an unreadable verdict differently. `verdicts` in `lib/judging.py` stops with an error, so a run that reads through it cannot count the reply as a FAIL. `readable_verdicts` in `lib/calibration.py` prints each unreadable reply and leaves it out of the verdicts, and so out of the denominator. Most examples that report a rate read through it. A rate that leaves replies out prints their count beside it.

Fixed replies go through the parser, one line of output each, which prints the verdict, or the label `refused` for a reply with no readable verdict.

Example `parse-verdicts` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""What lib.contract.parse_verdict accepts and refuses: the verdict on a reply's first non-empty line, PASS or FAIL in capitals,
after optional emphasis marks. Each fixed reply below prints one line, the verdict or 'refused'. A refusal is not a FAIL:
the judge has not given a verdict, and the evaluation counts that as an error, not a failed answer."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.contract import parse_verdict

CASES = (
    ("verdict first, PASS", "VERDICT: PASS\nReason: The answer matches the reference."),
    ("verdict first, FAIL", "VERDICT: FAIL\nReason: The answer gives a different version number."),
    ("emphasis around the line", "**VERDICT: FAIL**\nReason: The answer gives a different version number."),
    ("backticks around the line", "`VERDICT: PASS`\nReason: The answer matches the reference."),
    ("blank lines before the verdict", "\n\nVERDICT: PASS\nReason: The answer matches the reference."),
    ("no space after the colon", "VERDICT:PASS\nReason: The answer matches the reference."),
    ("two verdict lines, first one read", "VERDICT: FAIL\nVERDICT: PASS\nReason: Two verdicts."),
    ("lower case", "VERDICT: pass\nReason: The answer matches the reference."),
    ("title case", "Verdict: PASS\nReason: The answer matches the reference."),
    ("PARTIAL", "VERDICT: PARTIAL\nReason: Half of the answer matches."),
    ("verdict after a reason", "Reason: The answer matches the reference.\nVERDICT: PASS"),
    ("full stop after the verdict", "VERDICT: PASS.\nReason: The answer matches the reference."),
    ("words after the verdict", "VERDICT: PASS, mostly correct\nReason: The answer matches the reference."),
    ("bullet before the verdict", "- VERDICT: PASS\nReason: The answer matches the reference."),
    ("pairwise letter", "VERDICT: A\nReason: Response A is better."),
    ("bare word", "PASS\nReason: The answer matches the reference."),
    ("empty reply", ""),
    ("blank lines only", "  \n\n   "),
)

for label, raw in CASES:
    verdict = parse_verdict(raw)
    print(f"{label}: {verdict if verdict is not None else 'refused'}")
```

Output:

```text
verdict first, PASS: PASS
verdict first, FAIL: FAIL
emphasis around the line: FAIL
backticks around the line: PASS
blank lines before the verdict: PASS
no space after the colon: PASS
two verdict lines, first one read: FAIL
lower case: refused
title case: refused
PARTIAL: refused
verdict after a reason: refused
full stop after the verdict: refused
words after the verdict: refused
bullet before the verdict: refused
pairwise letter: refused
bare word: refused
empty reply: refused
blank lines only: refused
```

The first lines of the output show the forms the parser accepts. A verdict on the first line is read, and so is one with asterisks or backticks around it, blank lines above it, or no space after the colon. When two verdict lines appear, the parser reads the first, so a reply that says FAIL and then PASS is read as FAIL.

The unreadable lines matter most for a rubric. Lower case, title case and a PARTIAL verdict are all unreadable. So are a verdict placed after the reason, a full stop or extra words after PASS, a bullet in front of the verdict, a pairwise letter sent to a pass-or-fail rubric, a bare PASS with no label, an empty reply, and a reply of blank lines. Each unreadable form is a reply written in a shape the rubric did not ask for. A parser that guessed at what the judge meant would be reading the judge's intent, not its verdict.

The sentence to keep from this example is that a reply the parser cannot read is not a failed answer. It is a reply with no verdict, and no rate in this book turns it into a PASS or a FAIL. The book's advice is to print the unreadable count on each split. A split with more unreadable replies than the other is one the judge formats differently, which is worth finding out before the rate is read, since it may say more about the judge than about the assistant.

**Working rule:** Read the first non-empty line only. Never map an unreadable verdict to PASS or FAIL. Say in each rate what the run does with an unreadable verdict, whether it stops or leaves the reply out, and print the count beside any rate that leaves replies out. Check the parser on the replies the judge actually wrote, since a rubric that works on a sample can still lose verdicts to a format it did not ask for.

## Tuning the wording on dev {#tuning-the-wording}

The wording of a rubric is a setting, and a team will want to tune it until the judge agrees better with its labels. The risk is tuning on the numbers the team then reports. The book divides its items into two splits. [Dev](#read-the-outputs) is where wording is tuned. Held-out is run once, for each comparison a chapter reports, after the wording is fixed. A held-out split that has been used to choose the wording no longer tests it. Tuning a rubric on held-out items turns the reported agreement into a measure of the tuning, and then the agreement says nothing about the judge.

Before any tuning, the team writes the rule for each kind of item the set holds, in the rubric's own words: what passes, what fails, and what a decline is worth. Then it reads the dev replies the rubric gets wrong, one at a time, and changes the wording only when a reply shows a gap the rule does not cover. A change made to fit one reply is a change to the rubric, and it needs a recording of its own.

The answer rubric from [chapter 1](measure-before-you-change) is frozen. Its recording holds the hash of the wording that made it. Changing one word changes the hash, and the replay refuses the recording. A change therefore means the whole answer judge run is recorded again, for every reply, under the changed wording. This book does not tune that rubric, and a reader who wants to tune it should expect to re-record all of its replies.

[Rubric B](#what-a-judge-is-asked) is a second wording of the calibration task, and it is recorded on HelpSteer2. The calibration rubric says that a reply with no false claim is "not incorrect". Rubric B says that such a reply is "right", and it uses "right" and "wrong" where the calibration rubric uses "correct" and "incorrect". Rubric B also leaves out two points from the calibration rubric, that a reply must not mislead and that confidence must not be rewarded, and it drops formatting from the things the judge must not grade. Its instruction paragraph and its reply form read as follows. The rubric also has two made-up examples, which this chapter leaves out. They are part of the text that is hashed.

```
You grade whether an AI assistant's reply to a user is right.

A reply is right when every fact it states is true, every calculation or result in it is correct, and any code or reasoning in it works. Mark it wrong when any of those fail, or when it answers a different question from the one asked. A reply that makes no false claim is right, even when it is short, asks a question back, or leaves something out. Do not grade style, length or tone.

Reply in this form: on the first line, write exactly VERDICT: PASS if the reply is right, or VERDICT: FAIL if it is wrong. Then, on the next line, one short sentence of reason.
```

The example prints the hash of rubric B beside the other four, so a reader can tell the two calibration wordings apart. This chapter shows the text and the hash only, and [chapter 6](calibrate-the-judge) reports how well rubric B agrees with the human ratings.

The chapter made these decisions:

- The judge reads the first non-empty line only. Any other reply is an unreadable verdict, never a FAIL, and the two code paths that read verdicts are described above.
- Every rubric is hashed with its recording, and a changed rubric is a new rubric with a new recording.
- Every rate is pass and fail. A scale is not used, because the book has no human ratings at each step to check one against.
- A pairwise judge is run in both orders, and its order-changed pairs are reported beside its winner counts.

**Working rule:** Write the wording down before you run anything on held-out, and tune it on dev only. When the wording has to change, make it another rubric with its own recording and hash. Report each number with the hash of the wording that made it, so two wordings cannot be read as one.

# Measure a judge's errors against human ratings, then correct a pass rate

By the end of this chapter you can measure how a judge's verdicts differ from human ratings, split those differences into misses and false passes, and correct an observed pass rate for them, with the assumption the correction rests on written beside it.

[Chapter 5](a-judge-with-a-rubric) ended with a second wording of the calibration task, and with the rule that a judge's wording is measured, not assumed. The answer judge reads each reply against a written reference and marks it PASS or FAIL, and the baseline run of the assistant gets its pass rate from that judge. The rate is a number about the judge as much as about the assistant, because a judge can pass a wrong reply or fail a right one. [Chapter 5](a-judge-with-a-rubric) wrote the rubric and read the verdicts. This chapter asks how often those verdicts are wrong, in which direction, and what a team does to a rate once it knows.

## Human ratings, and what they are {#human-ratings}

A judge can only be checked against labels that were made some other way. This book uses human ratings from [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2), a public dataset from NVIDIA. Each row holds a prompt, one response to it, and five attribute scores from 0 to 4, given by people. The responses come from other models. This chapter uses one attribute, correctness. The sampling code turns the score into pass or fail: a score of 3 or 4 is read as PASS, and a score of 0 to 2 as FAIL. That cut is the book's choice, made before any judge was run.

HelpSteer2 is not the book's own data. Its responses come from other models, its prompts are general, and nobody wrote them for sablefile. So these labels measure how the judge does on a different question from the one the answer judge answers. That gap is the subject of [the transfer assumption](#transfer-assumption), and the rest of the chapter keeps returning to it.

The calibration judge reads with a rubric written for that one attribute. It asks whether every fact and result in a reply is right, and it tells the judge to set aside length, tone, style and formatting, as [chapter 5](a-judge-with-a-rubric#what-a-judge-is-asked) describes. The book's own test set could serve as the calibration, but the book has no human ratings of sablefile's replies. So the calibration uses public rows instead, and pays for that with the transfer assumption. The dataset does not hand the book a pass line for correctness. The book sets the line itself, and every rate in this chapter is read against it. A different line would move every rate, so a reader who compares this chapter with another study should first check where that study put its line.

The calibration rows are a sample of the dataset's validation split, drawn with a fixed seed. Rows whose text held characters outside printable ASCII were dropped before the draw. The draw takes about 45 FAIL and 45 PASS rows from each split, and it keeps one row per prompt. The base rate is therefore set by the sampling, not found in the dataset. The sample is split into dev and held-out. The same dataset also holds pairs of responses to one prompt, with a human preference between them. Pairs whose preference strength is zero are left out.

The split decides how the numbers may be used. Dev is where the book sets anything, including the rates a correction uses. Held-out is read once, in this chapter, after everything else is fixed, and nothing here is chosen with it. The held-out figures were first computed at the recording check, before this chapter was written. They are reported here as they stood, not recomputed to choose anything.

**Working rule:** Calibrate against labels made for the question the judge answers, or say in the text how far they are from it. Write down the rule that turns a score into the judge's pass or fail before you look at any result. Keep the split you tune on apart from the one you report once.

## Agreement and kappa {#agreement-and-kappa}

Agreement is the share of rows on which the judge and the human give the same verdict. It can mislead when one verdict dominates. A judge that passes everything agrees well with a sample that is mostly PASS, and agreement alone cannot tell that judge from a careful one. The fix is to ask what agreement the two verdicts would reach by chance, from their two PASS rates alone, and then take the chance out. Kappa does that. For two raters on pass or fail, kappa is the observed agreement minus the chance agreement, divided by one minus the chance agreement. [The kappa statistic, a review by McHugh](https://hrcak.srce.hr/file/132393), gives that formula. Kappa is one when the two raters agree on every row.

Example `calibration-agreement` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""How often the reference-free correctness judge agrees with the human ratings on the HelpSteer2 sample, and Cohen's kappa
for the same counts, on dev, on held-out and on both together.

Agreement alone reads higher than the judge's skill when most rows are one verdict, so the kappa line is the one to read
against chance: the chance agreement printed beside it is what the judge would reach by guessing at its own PASS rate."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import agreement, chance_agreement, cohens_kappa, confusion, helpsteer2_judgments
from lib.stats import pct

for label, split in (("all", None), ("dev", "dev"), ("held-out", "held-out")):
    counts = confusion(helpsteer2_judgments(split))
    print(f"agreement, {label}: {pct(counts.tp + counts.tn, counts.n)}")
    # the chance figure is an expected rate, not a count, so it has no denominator beside it
    print(f"chance agreement, {label}: {100 * chance_agreement(counts):.1f}%")
    # kappa is printed with two decimals (plan section 6.5), not as a percentage
    print(f"kappa, {label}: {cohens_kappa(counts):.2f}")
```

Output:

```text
agreement, all: 70.6% (127 of 180)
chance agreement, all: 50.3%
kappa, all: 0.41
agreement, dev: 71.1% (64 of 90)
chance agreement, dev: 50.2%
kappa, dev: 0.42
agreement, held-out: 70.0% (63 of 90)
chance agreement, held-out: 50.3%
kappa, held-out: 0.40
```

Agreement on all rows is 70.6% (127 of 180). Chance agreement, from the two PASS rates alone, is 50.3%. Kappa is 0.41, with two decimals and no percent sign, because it is a statistic on its own scale, not a rate. Agreement and chance agreement are both percentages, so they can be compared directly. Kappa is the figure that already has the chance taken out, so it is the one to read as the judge's agreement beyond chance.

Kappa runs from −1 to 1. It is 0 at chance agreement, and it is negative when the two raters agree less often than chance would, as [McHugh's review](https://hrcak.srce.hr/file/132393) notes.

Read the three figures in order: agreement, then the chance figure beside it, then kappa. If agreement rises while chance rises with it, the gain may come from the base rate rather than from the judge.

The two splits read the same way. On dev, agreement is 71.1% (64 of 90). Kappa on dev is 0.42. On held-out, agreement is 70.0% (63 of 90), and kappa is 0.40.

The base rate is the reason kappa matters. When one verdict dominates the rows, chance agreement is high, and a judge can post a high agreement while telling the two verdicts apart badly. Here the draw holds about as many PASS rows as FAIL rows, so chance agreement sits near half. The human PASS rate on dev is 51.1% (46 of 90). A team with a lopsided sample should expect chance agreement to sit near its agreement, so kappa is near zero even when agreement looks high. It should read kappa first.

Agreement measures closeness to the human labels, not whether the replies are right. High agreement says the judge's verdicts match the people's, not that the assistant's replies are right. Kappa tells you how far the agreement is from chance, but not which way the judge errs. That needs the next two rates.

Chapter 5 described a second calibration wording, rubric B. The calibration rubric, rubric A from here on, says that a reply which makes no false claim is "not incorrect". Rubric B rewords it: it says "right" where A says "not incorrect", and it drops the points about misleading and confidence. Rubric B was recorded on dev only. The next example sets the two wordings side by side on the same dev rows.

Example `rubric-b-agreement` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""How often the second rubric's verdicts agree with the human ratings on the HelpSteer2 sample, beside the first rubric's,
on dev and on held-out, with kappa, TPR and TNR and the PASS rates of judge and humans.

The second rubric was recorded on dev only, so its held-out line says so instead of printing a rate. The paired difference
is on dev alone: each dev row's agreement under rubric B minus its agreement under rubric A, bootstrapped over rows. A
reply with no readable verdict is printed by row ID and left out; it is never counted as FAIL."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from dataclasses import replace

from lib.calibration import cohens_kappa, confusion, helpsteer2_judgments, readable_verdicts
from lib.prompts import HS2B_SHA256
from lib.replay import ReplayModel
from lib.stats import bootstrap_mean, interval_points, pct

RUBRIC_B = "judge-helpsteer2-rubric-b"


def report(name, split, judgments):
    c = confusion(judgments)
    print(f"agreement, {name}, {split}: {pct(c.tp + c.tn, c.n)}")
    print(f"kappa, {name}, {split}: {cohens_kappa(c):.2f}")
    print(f"TPR, {name}, {split}: {pct(c.tp, c.tp + c.fn)}")
    print(f"TNR, {name}, {split}: {pct(c.tn, c.tn + c.fp)}")
    print(f"judge PASS rate, {name}, {split}: {pct(c.tp + c.fp, c.n)}")
    print(f"human PASS rate, {name}, {split}: {pct(c.tp + c.fn, c.n)}")


def same(j):
    return int(j.judge_pass == j.human_pass)


base = {j.id: j for j in helpsteer2_judgments()}  # rubric A's verdicts and the human PASS for every row, by id
recorded = {item for item, _ in ReplayModel(RUBRIC_B, HS2B_SHA256).keys()}
verdicts_b, unreadable = readable_verdicts(RUBRIC_B, sorted(recorded & base.keys()), HS2B_SHA256)
rows_b = {i: replace(base[i], judge_pass=v == "PASS") for i, v in verdicts_b.items()}

print(f"unreadable, rubric B: {len(unreadable)} of {len(recorded)} replies")
for item_id, _ in unreadable:
    print(f"unreadable, rubric B, row {item_id}: excluded, not counted as FAIL")
for split in ("dev", "held-out"):
    report("rubric A", split, [j for j in base.values() if j.split == split])
    b = [j for j in rows_b.values() if j.split == split]
    if b:
        report("rubric B", split, b)
    else:
        print(f"rubric B, {split}: no readable replies recorded")

dev_ids = [i for i in verdicts_b if base[i].split == "dev"]
diffs = [same(rows_b[i]) - same(base[i]) for i in dev_ids]  # the same row under both rubrics
lo, hi = bootstrap_mean(diffs)  # level 0.95 for one comparison; seed SEED, 2,000 resamples
n = len(dev_ids)
print(f"paired difference, dev: {100 * sum(diffs) / n:+.1f} points, 95% interval {interval_points(lo, hi)}, n = {n}")
reading = "rubric B better" if lo > 0 else "rubric B worse" if hi < 0 else f"no difference detected at n = {n}"
caveat = ", on dev only with no held-out run" if lo > 0 or hi < 0 else ""  # a dev-only difference carries its caveat
print(f"paired difference, dev, reading: {reading}{caveat}")
```

Output:

```text
unreadable, rubric B: 0 of 90 replies
agreement, rubric A, dev: 71.1% (64 of 90)
kappa, rubric A, dev: 0.42
TPR, rubric A, dev: 80.4% (37 of 46)
TNR, rubric A, dev: 61.4% (27 of 44)
judge PASS rate, rubric A, dev: 60.0% (54 of 90)
human PASS rate, rubric A, dev: 51.1% (46 of 90)
agreement, rubric B, dev: 74.4% (67 of 90)
kappa, rubric B, dev: 0.49
TPR, rubric B, dev: 84.8% (39 of 46)
TNR, rubric B, dev: 63.6% (28 of 44)
judge PASS rate, rubric B, dev: 61.1% (55 of 90)
human PASS rate, rubric B, dev: 51.1% (46 of 90)
agreement, rubric A, held-out: 70.0% (63 of 90)
kappa, rubric A, held-out: 0.40
TPR, rubric A, held-out: 82.6% (38 of 46)
TNR, rubric A, held-out: 56.8% (25 of 44)
judge PASS rate, rubric A, held-out: 63.3% (57 of 90)
human PASS rate, rubric A, held-out: 51.1% (46 of 90)
rubric B, held-out: no readable replies recorded
paired difference, dev: +3.3 points, 95% interval [-1.1, +7.8] points, n = 90
paired difference, dev, reading: no difference detected at n = 90
```

No difference shown on dev. Agreement under rubric B is 74.4% (67 of 90). Under rubric A, on the same rows, it is 71.1% (64 of 90). The paired difference is +3.3 points, 95% interval [-1.1, +7.8] points, n = 90. For each dev row, the code scores 1 when a wording agrees with the humans and 0 when it does not. The paired difference is rubric B's score minus rubric A's, averaged over the dev rows, and its interval is bootstrapped over those rows. The example's reading of that interval is no difference detected at n = 90.

The interval includes zero, so the dev estimate does not show that rubric B is better. Rubric B has no held-out run, so its dev estimate has no second check. Its unreadable replies are 0 of 90 replies, so no reply was left out.

**Working rule:** Report agreement beside kappa, and read kappa against the chance figure printed next to it. Name the rubric and the split with every number, so two wordings are never read as one. Call one wording better only when the lower bound of its paired interval is above zero, and only on held-out, or on dev with that caveat beside the claim. When the interval spans zero, say that no difference was detected at the n you have, and do not turn a higher point estimate into a claim.

## True positive and true negative rates {#tpr-and-tnr}

Agreement and kappa say how often the judge agrees with the humans. They do not say which way it errs. The two rates that do are the true positive rate and the true negative rate. The true positive rate, TPR, is the share of human PASS rows that the judge passes. The true negative rate, TNR, is the share of human FAIL rows that the judge fails. A judge that passes too much has a high TPR and a low TNR, because its errors are false passes. A judge that fails too much is the reverse. The correction in the next section needs both rates, since each one fixes a different error.

Example `tpr-tnr` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""The judge's true positive rate and true negative rate on the HelpSteer2 sample, with the counts beside each, on dev, on
held-out and on both together.

PASS is the positive class: TPR is the share of human PASS rows the judge passes, and TNR the share of human FAIL rows it
fails. The two rates are the judge's error profile; a lenient judge has a high TPR and a low TNR."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import confusion, helpsteer2_judgments
from lib.stats import pct

# PASS is the positive class: the rows a human rated PASS are the positives, and TPR is taken over them.
print("positive class: PASS")
for label, split in (("all", None), ("dev", "dev"), ("held-out", "held-out")):
    counts = confusion(helpsteer2_judgments(split))
    print(f"TPR, {label}: {pct(counts.tp, counts.tp + counts.fn)}")
    print(f"TNR, {label}: {pct(counts.tn, counts.tn + counts.fp)}")
```

Output:

```text
positive class: PASS
TPR, all: 81.5% (75 of 92)
TNR, all: 59.1% (52 of 88)
TPR, dev: 80.4% (37 of 46)
TNR, dev: 61.4% (27 of 44)
TPR, held-out: 82.6% (38 of 46)
TNR, held-out: 56.8% (25 of 44)
```

Here PASS is the positive class, so TPR is taken over the human PASS rows and TNR over the human FAIL rows. Each rate prints its counts beside it, so a reader can see how many rows it rests on.

On all rows, TPR is 81.5% (75 of 92). TNR is 59.1% (52 of 88). The judge passes most of the replies that humans passed, and it also passes a fair share of the replies humans failed. The weaker of the two rates is TNR, which makes this judge lenient.

The direction matters for anyone who reads a pass rate, and it depends on where the rate sits. The pass-rate correction in the next section leaves a rate unchanged at the crossover point, p* = (1 − TNR) / (2 − TPR − TNR). Below p* the correction lowers the rate, and above p* it raises the rate. For a lenient judge, p* sits high, so a reported rate below it reads too high and a rate above it reads too low. On dev, the crossover point is 66.4% (TPR 80.4%, TNR 61.4%). Knowing where a rate sits against that point tells a team which way its reported number leans, before any correction is applied. Agreement alone would hide this. Two judges can agree with the humans to the same degree and still differ in which error they make more often, and the rates they report would differ with it.

On dev, TPR is 80.4% (37 of 46), and TNR is 61.4% (27 of 44). On held-out, TPR is 82.6% (38 of 46), and TNR is 56.8% (25 of 44). The held-out split lands close to dev on both rates, with TNR a few points lower.

Each rate is taken over one group of human labels, and each split has few enough rows that another sample could move the rates. This chapter prints no interval for TPR or TNR. Read the rates as the judge's error on this sample, not as its error in general.

**Working rule:** Report TPR and TNR together, with their counts, and say which way the judge errs. Take the rates a correction uses from dev alone, and quote both beside any corrected rate. Report held-out once, after the choices are made, and keep it out of every choice.

## Correcting a pass rate {#correcting-a-pass-rate}

The observed pass rate counts two kinds of verdict: true passes the judge caught, and false passes it let through. The correction tries to undo both at once. Given the judge's TPR and TNR, it solves for the true share of passes that would produce the observed rate. The book writes it as (observed + TNR - 1) / (TPR + TNR - 1). [Rogan and Gladen](https://doi.org/10.1093/oxfordjournals.aje.a112510) (1978) address the same problem for screening tests, and their abstract says that adjustments of this kind are given. The book has read only that abstract, so the formula here is the book's own statement of the correction, as its code computes it, and not a quotation from the paper.

The correction has a limit. It divides by TPR + TNR - 1. A judge whose two rates sit near what chance would give has a denominator near zero, and then small errors in either rate become large swings in the corrected rate. The book's code refuses to correct when that denominator is below 0.1, and it raises an error instead of printing a rate. The run below stays well clear of that floor. In practice a team does four things in order: measure TPR and TNR on a calibration set kept apart from the set it will correct, check that the denominator clears the floor, apply the correction to the observed rate over the readable replies, and write the transfer assumption beside the result.

Example `corrected-pass-rate` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The baseline's answer-judge pass rate over the test items the judge gave a readable verdict, corrected for the judge's errors
with TPR and TNR from dev only. The corrected rate is printed beside the observed rate and the two rates the correction uses.

One judge reply has no PASS or FAIL on its first line. It is left out of the rate and printed by name, never counted as a
FAIL. The TPR and TNR come from HelpSteer2, where the judge rates other models' answers without a reference. This judge reads
the answers against a written reference. The correction assumes the two error profiles transfer; nothing here tests that.
The last lines apply the same correction to the calibration set itself: the rates are measured on those same rows, so that
check shows the arithmetic, not how well the correction generalises."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import (
    confusion,
    corrected_pass_rate,
    crossover_pass_rate,
    helpsteer2_judgments,
    readable_verdicts,
    tnr,
    tpr,
)
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import pct
from lib.testset import load_items

items = load_items()  # all 120 items of the test set, dev and held-out together
found, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in items], REF_SHA256)
details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)
print(f"unreadable verdicts: {len(unreadable)}{details}")  # left out of the rate below, not counted as FAIL
passes = int(passed(found).sum())  # found is keyed in sorted id order
observed = passes / len(found)
print(f"observed pass rate, readable items: {pct(passes, len(found))}")

dev = confusion(helpsteer2_judgments("dev"))  # the only rows the correction's TPR and TNR may come from
print(f"TPR, dev: {pct(dev.tp, dev.tp + dev.fn)}")
print(f"TNR, dev: {pct(dev.tn, dev.tn + dev.fp)}")
# Below this observed rate the correction lowers the rate; above it, the correction raises it.
crossover = crossover_pass_rate(tpr(dev), tnr(dev))
print(f"crossover point, dev: {100 * crossover:.1f}% (TPR {100 * tpr(dev):.1f}%, TNR {100 * tnr(dev):.1f}%)")

corrected = corrected_pass_rate(observed, tpr(dev), tnr(dev))  # refused if TPR + TNR - 1 is below 0.1
print(f"corrected pass rate: {100 * corrected:.1f}%")

# The sanity check: the same correction on the calibration set, with rates measured on those same rows. It checks the
# arithmetic, not transfer to the answers: it should return the human PASS rate, and it does by construction.
calibration = confusion(helpsteer2_judgments())
judge_rate = (calibration.tp + calibration.fp) / calibration.n
print(f"calibration set, judge PASS rate: {pct(calibration.tp + calibration.fp, calibration.n)}")
print(f"calibration set, corrected judge PASS rate: {100 * corrected_pass_rate(judge_rate, tpr(calibration), tnr(calibration)):.1f}%")
print(f"calibration set, human PASS rate: {pct(calibration.tp + calibration.fn, calibration.n)}")
calibration_crossover = crossover_pass_rate(tpr(calibration), tnr(calibration))  # the same test, on the same rows
print(f"crossover point, calibration set: {100 * calibration_crossover:.1f}% "
      f"(TPR {100 * tpr(calibration):.1f}%, TNR {100 * tnr(calibration):.1f}%)")
```

Output:

```text
unreadable verdicts: 1 (q-0043, "VERDICT: PARTIAL")
observed pass rate, readable items: 76.5% (91 of 119)
TPR, dev: 80.4% (37 of 46)
TNR, dev: 61.4% (27 of 44)
crossover point, dev: 66.4% (TPR 80.4%, TNR 61.4%)
corrected pass rate: 90.5%
calibration set, judge PASS rate: 61.7% (111 of 180)
calibration set, corrected judge PASS rate: 51.1%
calibration set, human PASS rate: 51.1% (92 of 180)
crossover point, calibration set: 68.9% (TPR 81.5%, TNR 59.1%)
```

The input is the baseline's answer-judge verdicts for the test items, all 120 of them, dev and held-out together. One reply had no readable verdict, because its first line was not PASS or FAIL. The example prints it by name and leaves it out of the rate, since a reply the judge could not read is not a FAIL. [Chapter 5](a-judge-with-a-rubric#verdict-first) explains the two ways the code reads an unreadable verdict. The line for it reads 1 (q-0043, "VERDICT: PARTIAL"). [Chapter 1](measure-before-you-change#a-pass-rate-and-its-interval) defines the pass rate over every item in the set. This rate is over the readable items, so it departs from that definition by the one reply left out.

The observed pass rate over the readable items is 76.5% (91 of 119). The correction uses TPR and TNR from dev, and those are the only rows it may use. Its TPR on dev is 80.4% (37 of 46). Its TNR on dev is 61.4% (27 of 44).

The corrected rate is 90.5%. The corrected rate estimates the share of sablefile's replies that a person would pass, under the book's correctness cut. It is not the judge's pass rate, which is the observed rate above, and it is not the assistant's accuracy. The estimate holds only if the judge's errors on sablefile's answers match its errors on HelpSteer2 rows. That is the [transfer assumption](#transfer-assumption), and the TPR and TNR above come from a reference-free judge rating other models' responses, while the answer judge checks sablefile's replies against a reference.

The corrected rate is higher than the observed rate, and the crossover point shows why. The observed rate sits above the crossover point, where the judge reads low. At that rate the judge's misses on true passes outweigh its false passes, so the correction raises it.

The result is also close to the code's ceiling. The code clips the corrected rate to the range from 0 to 1. The unclipped value passes 1 exactly when the observed rate is above TPR, because then the numerator is larger than the denominator. Here the observed rate sits under TPR by less than four points. If two of the dev PASS rows the judge passed had been failed instead, TPR would fall below the observed rate, and the corrected rate would print 100.0%.

**Working rule:** Correct a pass rate only with TPR and TNR measured on a calibration set. Quote both beside the corrected rate, refuse when the denominator is small, and say in the same paragraph what the correction assumes. The next section gives the interval and the one-row moves that go beside any corrected rate.

## How much one row moves the corrected rate {#how-much-one-row-moves-it}

Example `correction-sensitivity` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""How much the corrected pass rate of corrected-pass-rate depends on its inputs. A 95% bootstrap interval resamples both the
dev calibration rows and the readable test items, with the book's fixed seed. It prints the share of resamples that reach
the 100% cap, and the corrected rate when one dev row moves TPR or TNR up or down. Only dev rows set the correction's rates."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from dataclasses import replace

from lib.calibration import confusion, correction_interval, corrected_pass_rate, helpsteer2_judgments, readable_verdicts, tnr, tpr
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import RESAMPLES, interval_pct, pct
from lib.testset import load_items

items = load_items()  # all 120 items of the test set, as in corrected-pass-rate
found, _ = readable_verdicts("judge-answers-baseline", [item.id for item in items], REF_SHA256)
observed = int(passed(found).sum()) / len(found)

dev_rows = helpsteer2_judgments("dev")
dev = confusion(dev_rows)
lo, hi, capped, refused = correction_interval(dev_rows, passed(found))  # resamples the dev rows and the readable items
print(f"corrected pass rate, 95% interval: {interval_pct(lo, hi)}")
print(f"resamples at the 100% cap: {pct(capped, RESAMPLES)}")
print(f"resamples refused, TPR + TNR - 1 below 0.1: {pct(refused, RESAMPLES)}")

# One dev row moved: a human PASS row judged the other way changes TPR; a human FAIL row changes TNR.
moved = {
    "TPR one row higher": replace(dev, tp=dev.tp + 1, fn=dev.fn - 1),
    "TPR one row lower": replace(dev, tp=dev.tp - 1, fn=dev.fn + 1),
    "TNR one row higher": replace(dev, tn=dev.tn + 1, fp=dev.fp - 1),
    "TNR one row lower": replace(dev, tn=dev.tn - 1, fp=dev.fp + 1),
}
for label, counts in moved.items():
    print(f"corrected, {label}: {100 * corrected_pass_rate(observed, tpr(counts), tnr(counts)):.1f}%")
```

Output:

```text
corrected pass rate, 95% interval: [62.1%, 100.0%]
resamples at the 100% cap: 29.5% (590 of 2000)
resamples refused, TPR + TNR - 1 below 0.1: 0.1% (2 of 2000)
corrected, TPR one row higher: 86.0%
corrected, TPR one row lower: 95.5%
corrected, TNR one row higher: 91.0%
corrected, TNR one row lower: 90.0%
```

The example resamples both the dev rows and the readable items, 2,000 times, with the book's fixed seed, and it reads the spread of the corrected rate. The 95% interval is [62.1%, 100.0%]. It is wide, because TPR and TNR each rest on a few dozen dev rows, and its upper end is the cap itself. Some resamples reach the cap. Their share is 29.5% (590 of 2000). Resamples refused for a denominator under 0.1 are 0.1% (2 of 2000), and the interval is taken over the rest.

The one-row moves are these. If one dev row that humans passed had been judged PASS rather than FAIL, TPR would be one row higher, and the corrected rate would be 86.0%. If one such row had been judged FAIL, TPR would be one row lower, and the corrected rate would be 95.5%. For TNR, one dev row that humans failed and the judge passed would change the rate to 91.0% if it were judged FAIL, and to 90.0% if it were judged PASS. TPR is the rate to watch: one row there moves the answer by four to five points, while one row of TNR moves it by about half a point.

The interval covers the sampling of the dev rows and the readable items. It does not cover the transfer assumption. A judge whose errors differ on sablefile's replies could move the rate by more than the resamples show.

**Working rule:** Give the corrected rate with the bootstrap interval the book uses for its rates, the share of resamples at the cap, and the one-row moves beside it, so a reader can see how far single rows matter.

## The transfer assumption {#transfer-assumption}

The TPR and TNR above come from rows where the judge read other models' responses with no reference, under the correctness rubric, and the PASS line there is HelpSteer2 correctness of 3 or more. The answer judge reads sablefile's replies against a written reference, and a PASS there means the reply agrees with the reference, with a correct decline counting as a pass. The two judges answer different questions about what right means. The transfer assumption is that the judge's miss rate on human passes, and its pass-through rate on human fails, are the same on sablefile's replies as on the HelpSteer2 rows. Nothing in this chapter tests that. The correction holds only if the differences in input, replies, questions and PASS criterion leave the error profile unchanged. [Chapter 5](a-judge-with-a-rubric#reference-guided) sets out the difference between a judge with a reference and one without.

The last lines of the same example apply the correction to the calibration rows, with rates taken from those rows. On those rows, the judge's PASS rate is 61.7% (111 of 180). Its crossover point is 68.9% (TPR 81.5%, TNR 59.1%), and the judge's rate sits below it, so the correction lowers the rate. The corrected judge PASS rate is 51.1%. The human PASS rate is 51.1% (92 of 180).

The corrected judge rate returns the human rate. That follows from the algebra, because the rates were measured on those same rows. The check confirms that the arithmetic is right. It says nothing about whether the correction applies to sablefile, and a reader should not take it as evidence for the transfer.

The better fix is labels on the book's own replies. The book has none, because no person has rated sablefile's replies. A reader with such labels should calibrate on them, keeping a split apart from the set being corrected, as this chapter keeps dev apart from held-out. That would remove most of the transfer assumption, where the public rows only pay for it.

**Working rule:** Write the transfer assumption beside every corrected rate, in the same paragraph, and name the difference between the calibration judge and the judge the rate describes. Check the arithmetic on the rows the rates came from, and say that such a check does not test transfer. Until human labels on the answers the rate describes exist and have been checked against the judge, call the corrected rate an estimate that holds if the assumption does.

## Pairwise calibration {#pairwise-calibration}

A pairwise judge reads two replies and picks the better one, rather than passing or failing each. The human side is the HelpSteer2 preference between the two responses to a prompt. A pairwise judge has a known weakness: which reply sits in position A and which in B can decide the pick. [Chapter 5](a-judge-with-a-rubric#one-answer-or-two) explains why the book asks the judge about each pair in both orders, and [chapter 7](when-the-judge-stops-being-reliable#position) analyses the order changes that position causes. This section asks what agreement with the human preference looks like when both orders are counted. The order-changed count it prints is the one chapter 5 printed, and it is shown here only to explain the gap.

Each pair is a HelpSteer2 prompt with two responses and a human preference between them. Pairs with a tie were dropped, so every pair has a human pick. A TIE from the judge is a miss, because the judge said the two replies were equal when the human chose one. A single order gives a rate that depends on which letter the judge favoured, so a team that reports one order reports a number that the position of the replies can move. Counting both orders removes that freedom at the cost of a lower number. We recommend reporting the stricter rate for a pairwise judge, and reading the single-order rates only as a diagnostic of the gap between them.

Before you run the example, predict whether a pair that counts only when both orders pick the human's response will score above or below either order taken alone. Then check the guess against the output.

Example `pairwise-agreement` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""How often the pairwise judge's verdict agrees with the human preference on each HelpSteer2 pair, dev and held-out apart.
Three rates: the A,B order alone, the B,A order alone (both mapped back to the original responses), and both orders together,
where a pair counts only when both orders pick the human's response. A flipped pair (the same letter won both times) can
never count in the last rate, so the flipped count printed beside it is the number of misses that come from position.

Each single-order rate uses the pairs whose verdict in that order is readable; a TIE is judged and is not a pick, so it is
a miss. The interval is the 95% percentile bootstrap over pairs of the both-orders rate."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.judging import both_orders, human_pairs
from lib.stats import bootstrap_mean, interval_pct, pct

AB = "judge-hs2-pairs-ab"
BA = "judge-hs2-pairs-ba"

pairs = human_pairs()  # all 150 pairs, sorted by id, each with its split and the human's pick
orders = both_orders(AB, BA, list(pairs))  # both recordings read for every pair, sorted by id

for split in ("dev", "held-out"):
    rows = [o for o in orders if pairs[o.pair_id].split == split]
    human = {o.pair_id: pairs[o.pair_id].preferred for o in rows}
    ab = [o.ab == human[o.pair_id] for o in rows if o.ab is not None]
    ba = [o.ba == human[o.pair_id] for o in rows if o.ba is not None]
    both = [o.ab == human[o.pair_id] and o.ba == human[o.pair_id] for o in rows if o.ab is not None and o.ba is not None]
    flipped = sum(1 for o in rows if o.kind == "flipped")
    print(f"agreement, A,B order, {split}: {pct(sum(ab), len(ab))}")
    print(f"agreement, B,A order, {split}: {pct(sum(ba), len(ba))}")
    print(f"agreement, both orders, {split}: {pct(sum(both), len(both))}; flipped pairs: {flipped}")
    lo, hi = bootstrap_mean([float(hit) for hit in both])  # 95% interval over pairs, the level for one comparison
    print(f"95% interval, both orders, {split}: {interval_pct(lo, hi)}")
```

Output:

```text
agreement, A,B order, dev: 77.3% (58 of 75)
agreement, B,A order, dev: 73.3% (55 of 75)
agreement, both orders, dev: 65.3% (49 of 75); flipped pairs: 15
95% interval, both orders, dev: [53.3%, 76.0%]
agreement, A,B order, held-out: 74.7% (56 of 75)
agreement, B,A order, held-out: 76.0% (57 of 75)
agreement, both orders, held-out: 62.7% (47 of 75); flipped pairs: 19
95% interval, both orders, held-out: [52.0%, 73.3%]
```

Counted in both orders, agreement is lower than in either order alone, and the gap comes from the order-changed pairs. On dev, the A,B order alone gives 77.3% (58 of 75). The B,A order alone gives 73.3% (55 of 75). Counting both orders gives the stricter rate, 65.3% (49 of 75); flipped pairs: 15.

The count after the semicolon is the order-changed pairs, which the example's output labels flipped. An order-changed pair, as [chapter 5](a-judge-with-a-rubric#one-answer-or-two) defines it, is one where the verdict followed the position and not the reply. An order-changed pair can never count in the both-orders rate, so each one is a miss there.

The held-out split shows the same pattern. The A,B order gives 74.7% (56 of 75), the B,A order gives 76.0% (57 of 75), and both orders give 62.7% (47 of 75); flipped pairs: 19.

The interval on the both-orders rate, over pairs, is [53.3%, 76.0%] on dev. On held-out it is [52.0%, 73.3%]. These intervals are wide because each split has few pairs. The reading that holds is the pattern: a judge that changes its pick with position cannot be read as agreeing with people in any single order.

After this chapter the reader can measure a judge's agreement beyond chance, its misses and false passes on this sample, and a corrected pass rate with its transfer assumption written beside it. Those measures rest on public rows, not on sablefile's replies, and a pairwise judge counted in both orders still loses pairs to position. [Chapter 7](when-the-judge-stops-being-reliable) tests the limits a judge can show on the book's own recordings, starting with position, and it is where a reader learns when to stop trusting a verdict.

**Working rule:** When a judge compares two replies, run it in both orders, and report agreement counted both ways with the order-changed pairs beside it. Never read a single-order rate as the judge's agreement with people. Report the interval over pairs, and say that an order-changed pair is a miss, because it is the cost of position.

# Find where a judge's verdict stops being reliable

By the end of this chapter you can test a judge for two biases that this book's data can show, position and length. You can say what each test shows and what it leaves open, and you can write down the conditions under which you stop trusting one of its verdicts.

[Chapter 6](calibrate-the-judge) measured the judge's errors against human ratings, and it ended by asking what a pairwise judge does when its two replies trade places. This chapter asks what else can move a verdict in the sablefile replies the judge grades. The judge reads a reply to a test question, sets it against the reference answer, and prints PASS or FAIL, as [Chapter 5](a-judge-with-a-rubric) sets up. Its pass rate is the number a team would report, and that number is worth reporting only if the verdict depends on the reply. A verdict that changes when two answers trade places, or a pass rate that climbs with the length of a reply that says no more, measures something other than the answer. A rate built on such a verdict can look like progress without being any.

Two cautions hold for the whole chapter, so they are said once, here. The judge is Qwen3-4B-Instruct-2507, and every rate in this chapter is measured on that model. The published figures the chapter cites come from other judges, and they show what can happen to a judge, not what this judge does. The second caution is size. The human-rated pairs and the dev items are small sets, so every rate comes with its count, and every comparison comes with an interval.

The chapter takes three limits in turn. The first is position: whether a pairwise verdict depends on which answer was shown first. The second is length: whether the pass rate rises with the length of the reply. The third is self-preference: whether the judge favours answers written by its own model. The book can test the first two on its own recordings, because the recordings hold what the tests need: every human-rated pair judged in both orders, and every reply with a length that can be counted. It cannot test the third, and the section on it says why. The last section turns the three limits into rules a reader can apply, each tied to a count.

## Position {#position}

Ask the judge which of two replies is better, and its answer can depend on which reply it read first. A team that reads one order only would not see that dependence, and its report would carry the order along with the answers.

The test is simple to state. Judge each pair twice, once with the replies in one order and once swapped. A verdict that follows the reply it prefers is consistent. A verdict that follows the position changes when the order changes, which means the judge's choice depends on where a reply sat as well as on what it says. A judge can lean toward the first reply it reads or toward the second. The test shows that lean only as order changes. It can't say what causes the lean, and it doesn't need to, because an order-changed verdict is one a reader should not use as it stands. [Chapter 5](a-judge-with-a-rubric) sets up the two orders and the verdicts, and [Chapter 6](calibrate-the-judge) checks the judge's choices against human ratings.

Each human-rated pair is judged in both orders, and each pair lands in one of four kinds: consistent, order-changed, tie or unreadable. The example's output calls the order-changed pairs flipped. An unreadable pair has a missing verdict in either order. For the order-changed pairs, the question is which position won both orders.

Example `position-flips` (Python 3.12.15 · numpy 2.5.5), passing in Edition 2026.11.3:

```python
"""How the pairwise judge's verdict follows the position it was shown in, on the HelpSteer2 pairs. Each pair is judged in both
orders and sorted as consistent, flipped, tie or unreadable. The flipped pairs are split by the position that won both orders,
and the consistent pairs are checked against the human preference."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.bias import flip_direction
from lib.judging import both_orders, human_pairs
from lib.stats import bootstrap_mean, interval_pct, pct

AB = "judge-hs2-pairs-ab"
BA = "judge-hs2-pairs-ba"
SPLITS = ("dev", "held-out")

pairs = human_pairs()  # all 150 pairs, sorted by id, each with its split and the human's preferred response
orders = both_orders(AB, BA, list(pairs))  # every pair judged in both orders, sorted by id

for split in SPLITS:
    in_split = [o for o in orders if pairs[o.pair_id].split == split]
    for kind in ("consistent", "flipped", "tie", "unreadable"):
        # the denominator is the split's pairs, so the counts add up to the split's size
        print(f"{kind}, {split}: {sum(1 for o in in_split if o.kind == kind)} of {len(in_split)} pairs")

for split in SPLITS:
    flipped = [o for o in orders if o.kind == "flipped" and pairs[o.pair_id].split == split]
    # the letter A's position won both times when flip_direction says 'first'; B's when it says 'second'
    first = sum(1 for o in flipped if flip_direction(o) == "first")
    print(f"flipped, first position won both orders, {split}: {pct(first, len(flipped))}")

# both splits pooled, one 0 or 1 per flipped pair: 1.0 when the first position won both orders
flipped_all = [1.0 if flip_direction(o) == "first" else 0.0 for o in orders if o.kind == "flipped"]
print(f"flipped, first position won both orders, both splits: {pct(int(sum(flipped_all)), len(flipped_all))}")
lo, hi = bootstrap_mean(flipped_all)  # percentile bootstrap over the flipped pairs, as every interval in the book
print(f"first-position share interval, both splits: {interval_pct(lo, hi)}")

for split in SPLITS:
    consistent = [o for o in orders if o.kind == "consistent" and pairs[o.pair_id].split == split]
    # in a consistent pair the two orders agree, so ab is the original response the judge picked
    agree = sum(1 for o in consistent if o.ab == pairs[o.pair_id].preferred)
    print(f"consistent pairs agreeing with the human, {split}: {pct(agree, len(consistent))}")
```

Output:

```text
consistent, dev: 60 of 75 pairs
flipped, dev: 15 of 75 pairs
tie, dev: 0 of 75 pairs
unreadable, dev: 0 of 75 pairs
consistent, held-out: 56 of 75 pairs
flipped, held-out: 19 of 75 pairs
tie, held-out: 0 of 75 pairs
unreadable, held-out: 0 of 75 pairs
flipped, first position won both orders, dev: 20.0% (3 of 15)
flipped, first position won both orders, held-out: 26.3% (5 of 19)
flipped, first position won both orders, both splits: 23.5% (8 of 34)
first-position share interval, both splits: [8.8%, 38.2%]
consistent pairs agreeing with the human, dev: 81.7% (49 of 60)
consistent pairs agreeing with the human, held-out: 83.9% (47 of 56)
```

Most verdicts hold across the two orders. On dev, 60 of 75 pairs were consistent and 15 of 75 pairs changed with the order. On held-out, 56 of 75 pairs were consistent and 19 of 75 pairs changed with the order.

A tie counts as a readable verdict, so it stays in the denominator of the order-change rate. Here ties came to 0 of 75 pairs on dev, and unreadable verdicts to 0 of 75 pairs, so every dev pair gave a verdict in both orders.

Across both splits, the order-change rate is 22.7% (34 of 150). Its denominator is the pairs read in both orders, so a pair with an unreadable verdict in either order is left out.

The order changes lean toward the second position, and the pooled count is the one to read, because the split counts are small. Over both splits, the first position's share of the order changes is 23.5% (8 of 34). Its interval is [8.8%, 38.2%], and the interval lies entirely below one half. So the pooled count separates the lean from an even split, for these pairs. The pooled share rests on few order changes, so a handful moving either way would shift it by several points. On dev alone the share is 20.0% (3 of 15), and on held-out it is 26.3% (5 of 19).

Other studies have found the lean on other judges, and the leans did not all point the same way. [Wang and colleagues (2023)](https://arxiv.org/abs/2305.17926), a preprint, reported that GPT-4 favoured the first position and ChatGPT the second, in a setup with different models. [Zheng and colleagues (2023)](https://arxiv.org/abs/2306.05685), at the NeurIPS 2023 Datasets and Benchmarks track, reported that GPT-4, with a default prompt, gave the same verdict in both orders in 65.0% of comparisons. Those are published setups for other judges. They tell a reader what to measure, and they don't say what to expect from a 4B judge.

The consistent pairs are the ones whose verdict can be read as a choice, so the example checks them against the human preference. On dev, the judge matched the human on the consistent pairs at this rate:

81.7% (49 of 60).

On held-out, the rate was:

83.9% (47 of 56).

The order-changed pairs get no such check, because an order-changed verdict doesn't say which reply the judge chose. A team that reports agreement on the consistent pairs alone has measured the less contested part of the set. The order-changed part is where the judge's choice is in doubt, and it needs its own count.

The lean has a practical consequence for how a team runs its comparisons. Suppose a team always shows its candidate reply in the second slot and never swaps. If the judge leans toward the second slot, the candidate gets a boost that has nothing to do with its answer. A team that always puts the candidate first gets the opposite error. Swapping the order on every pair, and reporting the order changes, removes the lean from the comparison instead of hoping it is absent.

**Working rule:** Judge every pairwise comparison in both orders before you read its verdict, and report the order-change rate beside the verdict counts. Count an order-changed pair as no verdict for either reply, not as a win for whichever reply came first or second. When the order changes lean one way, say which way, how many, and with what interval, and don't assume that another judge leans the same way.

## Length {#length}

Length asks a different question. Position is about which reply the judge sees first; length is about the reply itself. The question is whether the judge gives a longer reply a higher score for being longer. Published studies describe that bias, so a test in this book has to look for it, and a clean result would need a test that could have shown the bias.

[Zheng and colleagues (2023)](https://arxiv.org/abs/2306.05685) reported that all the models they tested may be prone to verbosity bias, and that GPT-4 defends against it much better than the others. In one attack, a repetitive list was added to 23 answers. The failure rate was 91.3% for two of the models and 8.7% for GPT-4. Those figures come from other judges and a different attack. They are a reason to measure length here, and they say nothing about the size of the effect in this book.

The test is simpler than the attack. It takes the dev replies that the baseline assistant answered, sorts them into three bands by word count, and prints the judge's pass rate for each band. The edges that set the bands are fixed in the example. Other edges would make other bands, and the picture could change with them.

Example `length-and-verdicts` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The reference-guided judge's PASS rate by the length of the assistant's reply, on the dev items the baseline answered. The
reply lengths are in words, in three bands with the edges printed below. Each band's rate is printed with its count. Declined
replies and unreadable verdicts are left out, and each is printed with its count."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.bias import LENGTH_EDGES, band_outcomes, read_verdicts, reply_lengths
from lib.stats import pct
from lib.testset import load_items

ASSISTANT = "assistant-baseline"  # the reply the judge read, and the one whose length is measured
JUDGE = "judge-answers-baseline"

dev = [item.id for item in load_items(split="dev")]  # 80 dev items, sorted by id
lengths = reply_lengths(ASSISTANT, dev)  # the answered items only: declined replies are left out here
verdicts = read_verdicts(JUDGE, dev)  # None where the judge's first line is not a verdict

print(f"band edges, in words: {LENGTH_EDGES[0]} and {LENGTH_EDGES[1]}")
print(f"declined, left out: {len(dev) - len(lengths)} of {len(dev)} dev items")
unreadable = [item_id for item_id in lengths if verdicts[item_id] is None]
print(f"unreadable verdict, left out: {', '.join(unreadable) if unreadable else 'none'}")

for band, outcomes in band_outcomes(lengths, verdicts):
    print(f"{band}: {pct(int(outcomes.sum()), outcomes.size)}")
```

Output:

```text
band edges, in words: 15 and 30
declined, left out: 23 of 80 dev items
unreadable verdict, left out: q-0043
under 15 words: 94.1% (16 of 17)
15 to 29 words: 83.3% (15 of 18)
30 words or more: 85.7% (18 of 21)
```

The edges fall at 15 and 30 words. Declined replies are left out of the bands, which is 23 of 80 dev items. The judge's first line is read as its verdict. When that line is not a verdict, the verdict is unreadable, and the example prints such an item and leaves it out of the bands. It is never counted as FAIL. Here the unreadable verdict is on q-0043.

The bands are built from answered replies only, so their denominators are smaller than the set. The second-model comparison in the last section uses a different set of items, counting every dev item the judge read for both assistants and counting a correct decline as a pass under the answer rubric in [Chapter 5](a-judge-with-a-rubric). The two sets are not the same, and their rates should not be set side by side.

The rates do not climb with length. The middle band passed least often, and the longest band passed more often than the middle one, though less often than the shortest. A reader who fits one straight line through three bands would get a slope, and these bands do not support that line.

The shortest band passed 94.1% (16 of 17).

The middle band passed 83.3% (15 of 18).

The longest band passed 85.7% (18 of 21).

The comparison the book reports is the longest band minus the shortest. Its order is fixed by length, so it is set before anyone reads the rates. The highest-rate band minus the lowest-rate band would be chosen after the rates are seen, and it could never fall below zero. The gap is -8.4 points, interval [-28.6, +8.1] points.

The method is percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level. The two bands hold different items, so each is resampled on its own, and the bands' counts are the ones in their rates above. The band counts are small, so the interval is wide. The interval includes zero, so the result is [no difference detected](#paired-comparisons) between the longest and the shortest band.

No difference detected does not mean that length is harmless. It says that bands of this size cannot separate a gap from zero. They cannot pick out a few points of bias, because one reply moves a band's rate by several points. The bootstrap over items that [intervals and sample size](intervals-and-sample-size#bootstrap-over-items) describes is the same resampling. The two bands here are resampled on their own, an interval for two separate groups, which that chapter does not build.

A longer reply is also not always a padded one. A reply that covers more of the question may pass more often for that reason, and a rate that climbs with length would not, by itself, show padding. Telling the two apart needs replies in which length and content come apart, such as a complete answer and the same answer with filler added. The bands here cannot separate them, because the replies in each band differ in more than their length.

**Working rule:** Don't name a length effect, or its absence, from a few bands. Compare the longest band with the shortest, with the gap's interval and each band's count. When the interval includes zero, say "no difference detected" with those counts, and stop there. If length could matter to your product, build a test in which length changes and the content stays the same, and measure that.

## Self-preference, which this book cannot test {#self-preference}

The [length section](#length) asked whether the judge rewards a reply for its size. This section asks whether it rewards a reply for who wrote it. The book cannot answer that, and its own setup shows why.

[Panickssery and colleagues (2024)](https://arxiv.org/abs/2404.13076), a preprint, define self-preference as the tendency of LLM evaluators to score their own summaries higher than the summaries of other models, while human annotators rate the two sets as equal in quality. The authors call their causal result initial evidence, and they say the correlation does not by itself prove the causal claim. The question is worth asking of any judge, and the answer has to be measured on that judge.

A test would need answers of equal quality, as human raters place them, that the judge's own model and another model produced, or that are attributed to each. The same judge would then read the same answers under each attribution, and the two pass rates would differ only through the writer. The design is clear in outline, but the book's recordings do not carry it out.

The book has one judge reading two writers. The baseline assistant and the judge are the same model, Qwen3-4B-Instruct-2507. The one other set of answers comes from Qwen3-0.6B, a smaller model, which the same judge read. The book has no human rating of those answers at matched quality, and this judge is the only measure of their quality, so the comparison cannot hold quality fixed while the writer changes.

On the same dev items, the baseline passed 78.5% (62 of 79). The second model passed 40.5% (32 of 79). The difference is -38.0 points, interval [-48.1, -26.6] points, n = 79.

Read plainly, the second model's answers pass far less often. Lower quality would produce that gap, and so would favour toward the judge's own model. The book's data cannot tell the two apart, because the only measure of quality is this judge. So the gap is a difference between two assistants as this judge reads them. It is neither a self-preference result nor a clean absence of one.

A report that describes the judge as fair to its own model has to say that the test was not run on this book's recordings. Silence on that point reads as a clean result, and here it is not one. The claim a team can make is narrower: the recordings show a gap between two assistants, and they show nothing about the judge's favour toward its own model.

**Working rule:** When the judge and the assistant are the same model, report that beside the pass rate, and don't report self-preference as tested. Don't use the judge to compare its own model with another, unless the check above is in place: answers of matched quality, produced or attributed to both models, read by the same judge. [Chapter 6](calibrate-the-judge) sets up the human ratings that such a check needs.

## Where to stop trusting a verdict {#where-to-stop}

The limits are useful only as rules a reader can apply, and each rule needs a count to check it against. A reader who checks only the table has the limits, not the reasons for them, and the sections above give the reason for each line. The limits table prints them as labelled lines, in groups: the order-change rate; the length gap, with its method; the unreadable verdicts; the same-model count; and the second-model comparison. The order-change rate leaves out pairs unreadable in either order, while the position example above counts every pair in its split. Read the table from the top, because each line answers one question and no line answers another.

Example `verdict-limits` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The judge's limits as one table, one labelled line per limit, each computed as in position-flips and length-and-verdicts.
The length line compares the longest band with the shortest, fixed in advance by length order: each band holds different
items, so each is resampled on its own. The unreadable count is over every answer-judge recording on every item."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.bias import band_outcomes, paired_passes, read_verdicts, reply_lengths, same_model_count, unreadable_count
from lib.judging import both_orders, human_pairs
from lib.stats import RESAMPLES, bootstrap_difference, bootstrap_mean, interval_points, pct
from lib.testset import load_items

JUDGE = "judge-answers-baseline"
ANSWER_JUDGES = tuple(f"judge-answers-{variant}" for variant in ("baseline", "k5", "prompt-b", "second-model", "unfiltered"))
items = [item.id for item in load_items()]
dev = [item.id for item in load_items(split="dev")]

# Order-change rate: flipped pairs over the pairs read in both orders. A tie is a readable verdict; an unreadable pair is left out.
pairs = human_pairs()
read_both = [o for o in both_orders("judge-hs2-pairs-ab", "judge-hs2-pairs-ba", list(pairs)) if o.kind != "unreadable"]
print(f"order-change rate: {pct(sum(1 for o in read_both if o.kind == 'flipped'), len(read_both))}")

# Length gap: the longest band's PASS rate minus the shortest band's, fixed by length order, on the dev items.
shortest, _, longest = band_outcomes(reply_lengths("assistant-baseline", dev), read_verdicts(JUDGE, dev))
lo, hi = bootstrap_difference(shortest[1], longest[1])
print(f"length gap, longest band minus shortest band: {100 * (longest[1].mean() - shortest[1].mean()):+.1f} points, "
      f"interval {interval_points(lo, hi)}")
print(f"length gap method: percentile bootstrap over items, each band resampled alone, {RESAMPLES} resamples, 95% level")
print(f"shortest band, {shortest[0]}: {pct(int(shortest[1].sum()), shortest[1].size)}")
print(f"longest band, {longest[0]}: {pct(int(longest[1].sum()), longest[1].size)}")

unreadable, total = unreadable_count(ANSWER_JUDGES, items)
print(f"unreadable verdicts, five answer-judge recordings: {unreadable} of {total}")
print(f"items per answer-judge recording: {len(items)}")
worst = max((unreadable_count([name], items) for name in ANSWER_JUDGES), key=lambda found: found[0])
print(f"most unreadable in one recording: {worst[0]} of {worst[1]}")

same, judged = same_model_count(ANSWER_JUDGES)
print("self-preference test on these recordings: not possible")
print(f"same-model recordings: {same} of {judged}")

# Second model beside the baseline, on the dev items both judged: a None verdict in either leaves the item out.
both, base_pass, second_pass = paired_passes(JUDGE, "judge-answers-second-model", dev)
print(f"baseline PASS rate, same dev items: {pct(int(base_pass.sum()), len(both))}")
print(f"second-model PASS rate, same dev items: {pct(int(second_pass.sum()), len(both))}")
diff_lo, diff_hi = bootstrap_mean(second_pass - base_pass)
print(f"second-model minus baseline, same dev items: {100 * (second_pass - base_pass).mean():+.1f} points, "
      f"interval {interval_points(diff_lo, diff_hi)}, n = {len(both)}")
```

Output:

```text
order-change rate: 22.7% (34 of 150)
length gap, longest band minus shortest band: -8.4 points, interval [-28.6, +8.1] points
length gap method: percentile bootstrap over items, each band resampled alone, 2000 resamples, 95% level
shortest band, under 15 words: 94.1% (16 of 17)
longest band, 30 words or more: 85.7% (18 of 21)
unreadable verdicts, five answer-judge recordings: 2 of 600
items per answer-judge recording: 120
most unreadable in one recording: 1 of 120
self-preference test on these recordings: not possible
same-model recordings: 4 of 5
baseline PASS rate, same dev items: 78.5% (62 of 79)
second-model PASS rate, same dev items: 40.5% (32 of 79)
second-model minus baseline, same dev items: -38.0 points, interval [-48.1, -26.6] points, n = 79
```

Read each rate against its own denominator. The lines do not share one, and a rate with a small denominator moves a long way on a single item. The table says nothing about whether the judge agrees with people in general. That is the work of [Chapter 6](calibrate-the-judge), and a judge can pass every line here and still disagree with the human ratings on the pairs that matter. The limits are the conditions a verdict must meet before a calibration is worth doing, not a substitute for one.

The rules below follow those lines, and each rests on one of them.

- **Order changes.** Don't read a pairwise verdict as a result when it changes with the order. Count the order changes over both splits, and print the order-change rate beside the verdict counts.
- **Unreadable verdicts.** Print them, leave them out, and never count them as FAIL. The table counts 2 of 600 verdicts across five answer-judge recordings: the baseline, k5, prompt-b, second-model and unfiltered variants, each read on 120 items. The most any one recording lost was 1 of 120. This is a rule of thumb, not a measured cut-off: when more than 1 in 100 verdicts from one recording is unreadable, fix the prompt before you read a rate from that recording.
- **Length.** Don't name a length effect from a few bands, and don't name its absence from them either. The length line compares the longest band with the shortest, with its interval. When the interval includes zero, say "no difference detected" with the counts, and stop there.
- **Same model.** When the judge and the assistant are the same model, the rate should carry that fact beside it. Here 4 of 5 judged recordings are of that kind. The book cannot test self-preference on those recordings, so don't use the judge to compare its own model with another, as the [self-preference section](#self-preference) sets out.
- **Matched quality.** A gap between two assistants' pass rates is a difference between assistants, not a judge's bias. The second-model line is the example, and it is why the self-preference section gives no result either way.
- **Transfer.** The calibration a rate relies on was measured on other rows, so carry it only as far as [the transfer assumption](calibrate-the-judge#transfer-assumption) allows.

A verdict that meets a stop condition is not thrown away. An order-changed pair or an unreadable verdict becomes no verdict for that item. It is printed, counted and left out of the rate it would have entered, as the unreadable verdicts are, and the rate then reports how many items it left out. A same-model recording is not a stop condition. It is reported beside the rate.

In production, the limits show up as a rate that moves for the wrong reason. Suppose a team changes the assistant's prompt, and the pass rate rises. Before the team credits the prompt, the limits table asks three questions. Did the verdicts change with the order, or go unread? Did the replies get longer, and does the length line's interval separate the gap from zero? Is the judge the model that wrote the replies? A yes to the first two means the rise needs its own check before it is reported. A yes to the third means the rise is reported with that fact beside it.

**Working rule:** Stop trusting a verdict, and say so in the report, when it changes with the order or when it can't be read. When the judge is the model whose answers it grades, report that beside the rate, and don't use the judge to compare its own model with another, as the self-preference section sets out. Before a pass rate is published, check the judge against human ratings, as [Chapter 6](calibrate-the-judge) does, and print the order-change rate, the unreadable count and the same-model fact beside the rate. A pass rate describes the answers only as far as the judge agrees with people, so the rate should always travel with its calibration.

# Decide whether a change is better, and how many questions you need

By the end of this chapter you can say whether a difference between two versions of the assistant is more than noise, using an interval on a paired comparison. You can also work out how many questions a difference of a given size needs before a test can show it.

[Chapter 1](measure-before-you-change) measured the assistant once and gave its pass rate an interval. It then compared two prompts on twenty items. The interval on their difference was 30.0 points wide, and the chapter ended on a promise: a later chapter would work out how many items a difference needs before it can show. This chapter keeps that promise. A team that changes a prompt needs three answers. Did the rate move for a reason beyond the chance in the test? Was the comparison able to show a difference at all? And how many questions would it need to show the difference it cares about?

Every number here comes from one assistant model, Qwen3-4B-Instruct-2507, and one judge, the same model run at temperature zero. The intervals cover the spread that comes from drawing items. The first section measures the spread from sampling replies, without an interval. They don't cover the biases of the judge, which [Chapter 7](when-the-judge-stops-being-reliable) tests. Read each result with that in mind.

## Variance within one item {#variance-within-one-item}

Ask the same question twice and a model can answer differently. This chapter's comparisons use one recorded reply per item and prompt, so each verdict rests on one reply. At a higher temperature the model samples, and an item's verdict can change from one sample to the next. That matters for comparisons. A difference between two prompts that is smaller than the swing from one sample to the next isn't a difference at all, and a rate read from one run hides the swing.

The variance set holds five samples of the assistant's reply to each dev item, drawn at temperature 0.7. The judge reads every reply at temperature zero. An item flips when its five verdicts are not all the same. Counting the flips gives a measure of how much the verdicts move from sample to sample, before any change is made.

Example `flip-rate` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The flip rate over the dev variance set: the share of items whose judge verdict differs across their five samples. The
printed line 'flip rate' rests on the per-item PASS counts, which are printed with it: how many items passed 0, 1, 2, 3, 4
or 5 of 5 samples, and the mean pass rate. The temperature-0 baseline's pass rate over the same items is printed for
comparison. An item with an unreadable verdict is printed by id and left out, never counted as FAIL."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import readable_verdicts
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import pct
from lib.testset import load_items
from lib.variance import VARIANCE_SAMPLES, item_pass_counts

VARIANCE = "judge-answers-variance"
BASELINE = "judge-answers-baseline"
SAMPLES = len(VARIANCE_SAMPLES)

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
counts = item_pass_counts(VARIANCE, items)
excluded = [row for row in counts if not row.complete]
kept = [row for row in counts if row.complete]
for row in excluded:
    samples = ", ".join(str(sample) for sample in row.unreadable)
    print(f"unreadable, {row.item_id}: samples {samples} (left out, not counted as FAIL)")
print(f"left out of the flip rate (unreadable verdict): {len(excluded)}")

n = len(kept)
print(f"variance items compared (n): {n}")
print(f"flip rate: {pct(sum(1 for row in kept if row.flaky), n)}")
spread = [sum(1 for row in kept if row.passes == passes) for passes in range(SAMPLES + 1)]
assert sum(spread) == n  # every kept item falls in exactly one PASS count
for passes, count in enumerate(spread):
    print(f"items with {passes} of {SAMPLES} PASS: {count}")
print(f"mean per-item pass rate across samples: {pct(sum(row.passes for row in kept), n * SAMPLES)}")

found, baseline_unreadable = readable_verdicts(BASELINE, [row.item_id for row in kept], REF_SHA256)
assert not baseline_unreadable  # q-0043 is already left out above, so no kept item is unreadable under the baseline
baseline = passed(found)
print(f"baseline pass rate, same items (temperature 0): {pct(int(baseline.sum()), len(baseline))}")
```

Output:

```text
unreadable, q-0043: samples 1, 4 (left out, not counted as FAIL)
left out of the flip rate (unreadable verdict): 1
variance items compared (n): 79
flip rate: 2.5% (2 of 79)
items with 0 of 5 PASS: 16
items with 1 of 5 PASS: 1
items with 2 of 5 PASS: 0
items with 3 of 5 PASS: 0
items with 4 of 5 PASS: 1
items with 5 of 5 PASS: 61
mean per-item pass rate across samples: 78.5% (310 of 395)
baseline pass rate, same items (temperature 0): 78.5% (62 of 79)
```

Counting per item, the code tallies how many of its five samples the judge passed. An item with a verdict that can't be read is left out, not counted as a fail, and the output names the item it left out.

Most items don't move. 61 items pass all five samples, and 16 fail all five. Of the items that flip, 1 passes one sample and 1 passes four.

The flip rate is 2.5% (2 of 79).

The mean pass rate across all the samples is 78.5% (310 of 395). The baseline, one reply per item at temperature zero, passes 78.5% (62 of 79). The two are equal, and the two flipping items pull in opposite directions. That holds for P1, the prompt the variance set uses, and another prompt might be less steady.

That is a narrow result. A set with more borderline questions would flip more, and the flip rate counts items and has no interval of its own. A count of flipping items this small says little about how often the next change will flip.

The flip rate measures only the spread that comes from sampling this assistant's replies, with the items held fixed. A comparison between two prompts also moves with the items each prompt gets right. The last section uses that sampling spread as a lower bound, and only for a comparison that samples its replies: [How many questions a difference needs](#how-many-questions).

**Working rule:** Before you trust one run's rate, run the variance set at the temperature you ship, and read the flip rate beside the mean pass rate. The flip rate says how far sampling moves one prompt's verdicts. It bounds a comparison only when that comparison samples its replies too.

## The bootstrap over items {#bootstrap-over-items}

[Chapter 1](measure-before-you-change#a-pass-rate-and-its-interval) showed an interval on a pass rate and used it to read a small set. This section explains the method, because the rest of the chapter uses it three ways: on a rate, on a paired difference, and on a difference that has to clear a stricter level.

The method is the [percentile bootstrap](#a-pass-rate-and-its-interval). The items in a set are one sample of the questions a team could have written, so a different set would give a somewhat different rate. To estimate how much, draw items from the set with replacement, as many as the set holds, so that some items come up twice and some not at all. Compute the statistic for that draw, repeat the draw many times, and keep the middle 95% of the results. The bootstrap adds no data. It reuses the items the set already has, and it shows how far the statistic would move if the set had been another draw of the same kind.

The unit of resampling is the item, and that is the rule to keep. For a rate, each item contributes its verdict. For a paired difference, each item contributes one number: the verdict of prompt B minus the verdict of the baseline, which is plus one, zero or minus one. For the variance set, the five samples of an item are kept together, because they answer one question. Drawing them apart would make them look like five independent questions, and the interval would come out too narrow. [Miller (2024)](https://arxiv.org/abs/2411.00640) makes the same point for questions that come in related groups. That paper is a preprint and a practitioner's note, not a reviewed method, so this book uses it for the reason and not as proof.

The book fixes the settings so that a number can be reproduced. The method draws 2,000 resamples with a seed of zero, and it creates a fresh random generator inside each call. The same data therefore gives the same interval, whatever ran before it. The seed is a choice, not a property of the data. A different seed moves the endpoints a little, which is why the settings are written down.

The method also depends on the set being a fair sample of the questions, which [Chapter 1](measure-before-you-change) argued and no interval can check. A small set gives a rough interval. The endpoints of a percentile interval on a set of this size are a first estimate, and they should be read as approximate.

**Working rule:** Resample items, never runs within an item, and keep an item's samples together. Fix the resample count and the seed in the code, so the same data gives the same interval. When the set holds few items, call the interval rough and say so beside it.

## Paired comparisons {#paired-comparisons}

Two prompts answer the same questions, so their results are paired. A paired comparison subtracts, for each item, the verdict of the baseline from the verdict of prompt B, and averages the differences over the set. Its interval comes from the bootstrap over items.

Pairing matters because most items agree. An item that both prompts pass, or both fail, contributes nothing to the difference, whatever the question's difficulty. The comparison's spread comes from the items on which the two prompts split. An unpaired comparison takes two separate pass rates and sets them side by side. It carries the variation between questions on every item, including the ones that agree, and that variation can hide a real difference. The paired version cancels the part the two prompts share. The argument for paired differences in Miller (2024) says the same: the variance of a paired difference falls as long as the two models' scores on each item are positively correlated.

Example `paired-prompt-b` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Compares prompt-b (P2) with the baseline prompt (P1) on the eighty dev items: each pass rate, the paired difference with
its 95% interval, and how the items split between the two prompts. 'Better' is used only when the whole interval is above
zero (plan section 6.4); the interval here includes zero, so the reading is no difference detected."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import readable_verdicts
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import bootstrap_mean, interval_points, pct
from lib.testset import load_items

BASELINE = "judge-answers-baseline"
PROMPT_B = "judge-answers-prompt-b"

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)
p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)
for label, unreadable in (("P1 (baseline)", p1_unreadable), ("P2 (prompt-b)", p2_unreadable)):
    details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)
    print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")
# An item without a readable verdict under either prompt is dropped from both sides: a paired difference needs the same items.
excluded = {item_id for item_id, _ in p1_unreadable + p2_unreadable}
kept = [item_id for item_id in items if item_id not in excluded]
first = passed({item_id: p1_found[item_id] for item_id in kept})  # 1.0 or 0.0 per item, in id order
second = passed({item_id: p2_found[item_id] for item_id in kept})

n = len(kept)
lo, hi = bootstrap_mean(second - first)  # 95%, one comparison: the default level
print(f"P1 (baseline) pass rate: {pct(int(first.sum()), n)}")
print(f"P2 (prompt-b) pass rate: {pct(int(second.sum()), n)}")
print(f"difference, P2 minus P1: {100 * float((second - first).mean()):+.1f} points")
print(f"95% interval of the difference: {interval_points(lo, hi)}")
print(f"paired items (n): {n}")
print(f"items only P1 passes: {int(((first == 1) & (second == 0)).sum())}")
print(f"items only P2 passes: {int(((first == 0) & (second == 1)).sum())}")
print(f"items both pass: {int(((first == 1) & (second == 1)).sum())}")
print(f"items neither passes: {int(((first == 0) & (second == 0)).sum())}")
if lo > 0:
    print(f"reading: P2 is better at n = {n}")
elif hi < 0:
    print(f"reading: P2 is worse at n = {n}")
else:
    print(f"reading: no difference detected at n = {n}")
```

Output:

```text
unreadable verdicts, P1 (baseline): 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, P2 (prompt-b): 0
P1 (baseline) pass rate: 78.5% (62 of 79)
P2 (prompt-b) pass rate: 81.0% (64 of 79)
difference, P2 minus P1: +2.5 points
95% interval of the difference: [-3.8, +10.1] points
paired items (n): 79
items only P1 passes: 3
items only P2 passes: 5
items both pass: 59
items neither passes: 12
reading: no difference detected at n = 79
```

Here prompt B runs against the baseline on the dev items. An item with a verdict that can't be read under either prompt is left out of both sides, because a paired difference needs the same items on both sides. The printed lines give each pass rate, the difference, its interval and the number of paired items.

The baseline passes 78.5% (62 of 79). Prompt B passes 81.0% (64 of 79). The difference, prompt B minus the baseline, is +2.5 points, with a 95% interval of [-3.8, +10.1] points.

The split shows where the difference lives. 59 items pass under both prompts, and 12 fail under both. Only the items that split move the difference: 5 pass only under prompt B, and 3 pass only under the baseline.

The interval includes zero, so the reading is no difference detected at n = 79. The interval runs from a drop to a rise, and this set can't say which end is nearer. No difference detected is a finding about a set at its size. It is not a claim that the prompts are equal, and the text shouldn't say they are.

**Working rule:** Compare variants on the same items, paired, and report the difference with its interval and n. Call one variant better only when the whole interval sits above zero, and worse only when the whole interval sits below it. When the interval includes zero, report no difference detected at that n, and say how many items the difference would need, as [How many questions a difference needs](#how-many-questions) works out.

## Held-out, once {#held-out-once}

Dev items are for building. A team may tune a prompt on them, read the failures and change the next version. The dev results can't then prove that a change works, because the change was shaped by them. The [held-out set](#dev-and-held-out) is the check that isn't tuned. A comparison on held-out is run once, against rules fixed before the run: the variants, the level and the reading. Run it again after a change of mind, or choose the level after seeing the number, and the interval no longer means what it says.

This section runs three comparisons on the held-out set, each variant against the baseline. Prompt B is the one from [Paired comparisons](#paired-comparisons). The variant k5 retrieves five sections instead of three. The variant unfiltered retrieves from every version of the docs at once, with three sections, instead of only the asked version.

Example `held-out-variants` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Compares prompt-b, k5 and unfiltered with the baseline on the held-out items, each variant once, at the level the
Bonferroni rule gives for three comparisons (98.33%). Each variant gets its paired difference, interval, n and a reading by
plan section 6.4. Nothing was chosen with these numbers: the variants, the level and the reading rule were fixed before
this run, and the run happens once."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import readable_verdicts
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import RESAMPLES, bootstrap_mean, interval_points
from lib.testset import load_items

BASELINE = "baseline"
VARIANTS = ("prompt-b", "k5", "unfiltered")
M = len(VARIANTS)  # one comparison per variant, each against the baseline
LEVEL = 1 - 0.05 / M  # Bonferroni: 1 - 0.05/m, which is 98.33% for m = 3 (plan section 6.3)

items = [item.id for item in load_items(split="held-out")]  # sorted by id
print(f"comparisons (m): {M}")
print(f"share of the 5% for each comparison, 0.05 / m: {0.05 / M:.4f}")
print(f"level of each interval, 1 - 0.05 / m: {100 * LEVEL:.2f}%")
print(f"resamples beyond each end of an interval: {RESAMPLES * (1 - LEVEL) / 2:.1f}")
base_found, base_unreadable = readable_verdicts(f"judge-answers-{BASELINE}", items, REF_SHA256)
details = "".join(f' ({item_id}, "{first}")' for item_id, first in base_unreadable)
print(f"unreadable verdicts, baseline: {len(base_unreadable)}{details}")
for name in VARIANTS:
    found, unreadable = readable_verdicts(f"judge-answers-{name}", items, REF_SHA256)
    details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)
    print(f"unreadable verdicts, {name}: {len(unreadable)}{details}")
    # An item unreadable under the baseline or this variant is dropped from both sides of this pair only.
    excluded = {item_id for item_id, _ in base_unreadable + unreadable}
    kept = [item_id for item_id in items if item_id not in excluded]
    first = passed({item_id: base_found[item_id] for item_id in kept})
    second = passed({item_id: found[item_id] for item_id in kept})
    n = len(kept)
    lo, hi = bootstrap_mean(second - first, level=LEVEL)
    if name == "prompt-b":  # the chapter says this lower end is exactly zero, so the run must show it
        assert lo == 0.0, "the prompt-b lower bound must be exactly zero"
    print(f"{name} minus baseline, difference: {100 * float((second - first).mean()):+.1f} points")
    print(f"{name} minus baseline, {100 * LEVEL:.2f}% interval: {interval_points(lo, hi)}")
    print(f"{name} minus baseline, n: {n}")
    if lo > 0:
        print(f"{name} minus baseline, reading: better at n = {n}")
    elif hi < 0:
        print(f"{name} minus baseline, reading: worse at n = {n}")
    else:
        print(f"{name} minus baseline, reading: no difference detected at n = {n}")
```

Output:

```text
comparisons (m): 3
share of the 5% for each comparison, 0.05 / m: 0.0167
level of each interval, 1 - 0.05 / m: 98.33%
resamples beyond each end of an interval: 16.7
unreadable verdicts, baseline: 0
unreadable verdicts, prompt-b: 0
prompt-b minus baseline, difference: +10.0 points
prompt-b minus baseline, 98.33% interval: [+0.0, +22.5] points
prompt-b minus baseline, n: 40
prompt-b minus baseline, reading: no difference detected at n = 40
unreadable verdicts, k5: 0
k5 minus baseline, difference: +2.5 points
k5 minus baseline, 98.33% interval: [-12.5, +18.4] points
k5 minus baseline, n: 40
k5 minus baseline, reading: no difference detected at n = 40
unreadable verdicts, unfiltered: 0
unfiltered minus baseline, difference: -10.0 points
unfiltered minus baseline, 98.33% interval: [-27.5, +8.4] points
unfiltered minus baseline, n: 40
unfiltered minus baseline, reading: no difference detected at n = 40
```

Three comparisons share one held-out set, so each interval is set at a higher level than 95%, and [Many comparisons](#many-comparisons) explains why. The level is 98.33%. Each variant gets a difference, an interval, a count of paired items and a reading, all printed.

Held-out has 40 items, against 79 on dev. So its intervals are wider for the same difference, and a result on it can only be as sharp as its item count allows. The split keeps a result honest, and its cost is fewer items to spend.

The difference for prompt B is +10.0 points, with an interval of [+0.0, +22.5] points. The comparison has 40 paired items.

The k5 difference is +2.5 points, with an interval of [-12.5, +18.4] points. The unfiltered difference is -10.0 points, with an interval of [-27.5, +8.4] points.

Dev gave +2.5 points for the same prompt, and the two point estimates differ. The 40 held-out items can't tell which is nearer the truth. The smallest difference that they detect, at 98.33% confidence and 80% power, is 16.2 points, so both point estimates sit below what this set can detect. [How many questions a difference needs](#how-many-questions) works out the arithmetic.

By the rule, all three read as no difference detected. For prompt B the printed reading is no difference detected at n = 40. For k5 it is no difference detected at n = 40, and for unfiltered no difference detected at n = 40.

Prompt B is the close case. Its interval is [+0.0, +22.5] points, and its lower end is exactly zero. The reading rule says better only when the whole interval sits above zero, and this one reaches zero without clearing it, so by the rule the reading is no difference detected. Beyond each end of the interval lie 16.7 of the 2,000 resamples, so the call sits on the line. The seed was fixed before the run for that reason. A reader could argue that a point estimate of that size should count for something, and this section doesn't make that argument. The rule was set before the run, the comparison runs once, and nobody gets to look at these three again with a lower level or a different reading. If prompt B is to be tested further, it needs a fresh held-out set, sized by [How many questions a difference needs](#how-many-questions) and fixed before anyone sees its results, read on its own. Adding items to the 40 already read, and rereading the pool, would be a second look at a result already seen.

**Working rule:** Fix the variants, the level and the reading rule before you open the held-out set, and run each comparison once. An interval that reaches zero is no difference detected at that n. Don't rerun the comparison, lower the level or change the reading to get a different answer. Add held-out items only before they are scored.

## Many comparisons {#many-comparisons}

Each comparison is a chance to be fooled. A 95% interval on a comparison with no real difference still excludes zero by luck about one time in twenty. Run three such comparisons, report each at 95%, and the chance that at least one of them reads as a difference rises above five in a hundred. The more comparisons a chapter reports, the more likely one of them is a lucky result.

The usual fix is the Bonferroni correction, which shares the error budget between the comparisons. With m comparisons of one metric, each interval is set at 1 minus 0.05 divided by m. For three comparisons, 0.05 divided by 3 is 0.0167, so each interval's level is 98.33%, the level the [held-out section](#held-out-once) used. The 0.05 is split between the two tails of each interval, so each tail gets 0.05 divided by twice m. The arithmetic holds whatever the comparisons have in common. The chance that any of them shows a false difference can't exceed the sum of their separate chances, and that sum is 5%.

The code computes the level from m and prints it, so a reader can check it against the count. The price is width. At this level each interval is wider than a 95% one, so a difference has to be larger before its interval clears zero. The sum of the separate chances is an upper bound on the chance of any false difference, so the correction is conservative whenever the comparisons' false results overlap. This book uses this one because its rule is short to state and to check, and a reader can recompute it from the count.

Which comparisons form the count? In this chapter, the dev comparison in the paired section stands alone, and it is reported at 95% because it is a worked example on tuning data. The three held-out comparisons form a family, and they are reported at the adjusted level. A reader who sees both should not read the dev interval as a held-out result, and the held-out result is the one the chapter relies on.

The count is part of the design. Count the comparisons of one metric that the chapter will report before you run any of them. A comparison added after the results is reported apart from the family, as exploratory, with its own count. The family's intervals stay as they were run, because setting them again after the results would change the rule the run was held to.

**Working rule:** Count the comparisons of one metric before you run them, and set each interval's level at 1 minus 0.05 divided by that count. Show the level beside every interval. A comparison added after the results is reported apart as exploratory, and the family's intervals are not set again.

## How many questions a difference needs {#how-many-questions}

The question to ask before a comparison is how many items it needs to show a difference of the size the team cares about. The answer depends on one quantity: how much the two variants disagree, item by item. Two prompts that disagree on few items show a small difference with few items.
The paired formula makes this concrete. Write z-conf for the two-sided normal value at the confidence level and z-power for the one at the power. Then n = (z-conf + z-power)² · var(d) / δ², where var(d) is the variance of one item's paired difference and δ is the difference to detect. The variance is the share of items on which the two variants disagree, minus the square of their average difference. So the estimate is driven by disagreement, not by the pass rates themselves. This section uses 95% confidence and 80% power.

Example `sample-size` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""How many items a paired comparison needs to detect a difference of 5 and of 10 points, with 95% confidence and 80% power.

The main estimate reads the paired differences that prompt-b and the baseline show on the dev items: each item's d is
+1, 0 or -1 (P2 minus P1), and the items on which they disagree set the variance. The floor uses only the judge's
within-item sampling noise over five samples; it leaves the disagreement between variants out, so it is a lower bound.
"""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib import variance
from lib.answers import chapter_items
from lib.calibration import readable_verdicts
from lib.judging import paired, passed
from lib.prompts import REF_SHA256
from lib.stats import pct
from lib.testset import load_items

# n = (z_alpha + z_beta)^2 * var_d / delta^2, rounded up (variance.items_needed). var_d = p_discordant - mean(d)^2.
# Smallest difference that n items detect: (z_alpha + z_beta) * sqrt(var_d / n) (variance.smallest_difference_points).
BASELINE, PROMPT_B = "judge-answers-baseline", "judge-answers-prompt-b"
CHAPTER_ONE_ITEMS = 20  # the chapter 1 set, read below the way twenty-cannot-tell reads it

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)
p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)
kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]
first = passed({item_id: p1_found[item_id] for item_id in kept})  # 1.0 or 0.0 per item
second = passed({item_id: p2_found[item_id] for item_id in kept})
var_d = variance.paired_difference_variance(first, second)
print(f"share of discordant items, P2 and P1 disagree: {pct(int((first != second).sum()), len(kept))}")
print(f"assumption, the next change disagrees with the baseline on about as many items as prompt-b did "
      f"({100 * (first != second).mean():.1f}% of items)")

# Floor: five samples of a 0/1 verdict with k PASS give the unbiased within-item variance k(5 - k) / (5 * 4).
rows = [row for row in variance.item_pass_counts("judge-answers-variance", items) if row.complete]
noise = sum(row.passes * (5 - row.passes) / 20 for row in rows) / len(rows)
print(f"floor, sampling noise only: dev items with five readable verdicts (n): {len(rows)}")
print(f"floor, sampling noise only: mean within-item variance of one verdict: {noise:.4f}")

# Two variants judged separately: their two noises add and the item's difficulty cancels, so the floor is 2 * noise.
for label, var in (("main estimate", var_d), ("floor, sampling noise only", 2 * noise)):
    print(f"{label}, variance of one item's paired difference: {var:.4f}")
    for delta in (5.0, 10.0):
        print(f"{label}, items needed for a {delta:.0f}-point difference: {variance.items_needed(var, delta)}")
    print(f"{label}, smallest difference that {CHAPTER_ONE_ITEMS} items detect, 80% power and 95% confidence: "
          f"{variance.smallest_difference_points(var, CHAPTER_ONE_ITEMS):.1f} points")

# The chapter 1 set, read as twenty-cannot-tell reads it: its share of discordant items and its interval width.
chapter_one = paired(BASELINE, PROMPT_B, [item.id for item in chapter_items()])
width_20 = variance.interval_width_points(var_d, CHAPTER_ONE_ITEMS)
print(f"chapter 1 set, share of items where the two verdicts differ: "
      f"{pct(int((chapter_one.first != chapter_one.second).sum()), chapter_one.n)}")
print(f"chapter 1 set, 95% interval width: {100 * (chapter_one.diff_hi - chapter_one.diff_lo):.1f} points")
print(f"main estimate, 95% width at {CHAPTER_ONE_ITEMS} items: {width_20:.1f} points")
print(f"reading: with {CHAPTER_ONE_ITEMS} items and a change that disagrees with the baseline on "
      f"{100 * (first != second).mean():.1f}% of items, as prompt-b did, the smallest difference detected with 80% power "
      f"at 95% confidence is {variance.smallest_difference_points(var_d, CHAPTER_ONE_ITEMS):.1f} points")
```

Output:

```text
share of discordant items, P2 and P1 disagree: 10.1% (8 of 79)
assumption, the next change disagrees with the baseline on about as many items as prompt-b did (10.1% of items)
floor, sampling noise only: dev items with five readable verdicts (n): 79
floor, sampling noise only: mean within-item variance of one verdict: 0.0051
main estimate, variance of one item's paired difference: 0.1006
main estimate, items needed for a 5-point difference: 316
main estimate, items needed for a 10-point difference: 79
main estimate, smallest difference that 20 items detect, 80% power and 95% confidence: 19.9 points
floor, sampling noise only, variance of one item's paired difference: 0.0101
floor, sampling noise only, items needed for a 5-point difference: 32
floor, sampling noise only, items needed for a 10-point difference: 8
floor, sampling noise only, smallest difference that 20 items detect, 80% power and 95% confidence: 6.3 points
chapter 1 set, share of items where the two verdicts differ: 10.0% (2 of 20)
chapter 1 set, 95% interval width: 30.0 points
main estimate, 95% width at 20 items: 27.8 points
reading: with 20 items and a change that disagrees with the baseline on 10.1% of items, as prompt-b did, the smallest difference detected with 80% power at 95% confidence is 19.9 points
```

Here the code reads the disagreement between prompt B and the baseline on the dev items. Its assumption is printed, and it is the one to check: the next change disagrees with the baseline on about as many items as prompt B did. Here the share of items that disagree is 10.1% (8 of 79).

Example `sample-size-limits` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The arithmetic behind the five-point figure in sample-size, and the limits of its inputs: the interval on the share of
discordant items (a percentile bootstrap over items, as lib.stats computes every interval), the held-out level of 98.33%,
and the difference that the 40 held-out items can detect. The disagreement is the dev items' paired run, as in sample-size."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib import variance
from lib.calibration import readable_verdicts
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import bootstrap_mean, interval_pct
from lib.testset import load_items

BASELINE = "judge-answers-baseline"
PROMPT_B = "judge-answers-prompt-b"
ALPHA_HELD_OUT = 0.05 / 3  # two-sided alpha for each of the three held-out comparisons (Bonferroni)
HELD_OUT_LEVEL = 1 - ALPHA_HELD_OUT

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
p1_found, p1_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)
p2_found, p2_unreadable = readable_verdicts(PROMPT_B, items, REF_SHA256)
kept = [item_id for item_id in items if item_id not in {i for i, _ in p1_unreadable + p2_unreadable}]
first = passed({item_id: p1_found[item_id] for item_id in kept})
second = passed({item_id: p2_found[item_id] for item_id in kept})
discordant = (first != second).astype(float)  # 1 where the two prompts disagree on an item, else 0
var_d = variance.paired_difference_variance(first, second)
mean_d = float((second - first).mean())

# The five-point figure, worked: n = (z_alpha + z_beta)^2 * var(d) / delta^2
z2, delta2 = variance.z_sum() ** 2, 0.05 ** 2
print(f"z sum squared, 95% confidence and 80% power: {z2:.2f}")
print(f"var(d), variance of one item paired difference: {var_d:.4f}")
print(f"delta squared, a five-point difference: {delta2:.4f}")
print(f"n for five points: {z2:.2f} x {var_d:.4f} / {delta2:.4f} = {z2 * var_d / delta2:.1f}, "
      f"rounded up to {variance.items_needed(var_d, 5.0)}")

# The share rests on the discordant items alone, so its 95% interval is a range for the answer.
share_lo, share_hi = bootstrap_mean(discordant)
print(f"discordant share, 95% interval: {interval_pct(share_lo, share_hi)}")
for end, share in (("low", share_lo), ("high", share_hi)):
    # mean(d) stays at its observed value; only the share moves
    print(f"items needed for 5 points, {end} end of the share: {variance.items_needed(share - mean_d ** 2, 5.0)}")

print(f"items needed for 5 points at {100 * HELD_OUT_LEVEL:.2f}% confidence: "
      f"{variance.items_needed_at(var_d, 5.0, ALPHA_HELD_OUT)}")

# Only the count is read here: no held-out verdict is opened by this example.
held_out_n = len(load_items(split="held-out"))
print(f"held-out items (n): {held_out_n}")
print(f"smallest difference {held_out_n} held-out items detect at {100 * HELD_OUT_LEVEL:.2f}% and 80% power: "
      f"{variance.smallest_difference_at(var_d, held_out_n, ALPHA_HELD_OUT):.1f} points")
```

Output:

```text
z sum squared, 95% confidence and 80% power: 7.85
var(d), variance of one item paired difference: 0.1006
delta squared, a five-point difference: 0.0025
n for five points: 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316
discordant share, 95% interval: [3.8%, 17.7%]
items needed for 5 points, low end of the share: 118
items needed for 5 points, high end of the share: 555
items needed for 5 points at 98.33% confidence: 422
held-out items (n): 40
smallest difference 40 held-out items detect at 98.33% and 80% power: 16.2 points
```

The worked arithmetic for a five-point difference runs in that example. The square of the sum of the two z values is 7.85. The variance of one item's paired difference is 0.1006. The five-point difference squared is 0.0025. So n is 7.85 x 0.1006 / 0.0025 = 315.9, rounded up to 316.

Miller (2024) gives the paired formula in a more general form, with terms for the noise within each item. The formula here is that form with one reply per item, which drops the within-item terms. Its worked example, with the paper's inputs, comes to about 969 items. Those inputs describe a different experiment, so the book's own numbers come from the printed run only.

Power works the other way too. At 80% power, a set sized for a real difference still misses it about one time in five. That is the price of a fixed budget of questions, and a team that can't pay it should either accept a larger difference as its goal or enlarge the set before the run.

Because the items needed scale with that share when the average difference is small, a change that disagrees on twice as many items needs about twice the questions. A team should estimate the share from a paired run on dev, state it, and check it again when the change is bigger than the one it was measured on.

The main estimate says that a five-point difference needs 316 items, and a ten-point difference needs 79. The floor uses only the sampling noise within each item, and it says 32 and 8. Read the floor as a lower bound on what is needed, not as a plan. It leaves out the disagreement between variants, which is real. Two prompts change what the assistant says, not only how steadily it says it. The floor holds only for a comparison that samples one reply per item at temperature 0.7, with variants as steady as P1. Only two of the items add to its noise, so it rests on very few items. A plan made on the floor comes up short whenever a change disagrees more than sampling alone would explain.

The same estimate answers the question for Chapter 1's set. With twenty items, the smallest difference that the test detects with 80% power at 95% confidence is 19.9 points, under the disagreement assumed above.

Chapter 1's set disagreed on 10.0% (2 of 20) of its items.

Its interval was 30.0 points wide. The formula gives 27.8 points at the same disagreement, which is close. The [twenty-item example](https://github.com/rollingedition/book-evaluating-llm-applications/blob/8c34edd978b89b13f71af691b1c0a38f3c262020/examples/twenty-cannot-tell.py) shows that set. At that disagreement, twenty items would usually miss a difference of ten points, and a team that wants to see a difference of five points needs hundreds of items, not dozens.

The estimate rests on the discordant items alone, so it is an order of magnitude, not a count. Its 95% interval on the share is [3.8%, 17.7%]. Across that interval, a five-point difference needs 118 items at the low end and 555 at the high end. The formula is a normal approximation, while the reading rule uses a bootstrap, and the two agree only roughly when few items disagree. At the held-out level of 98.33%, the same inputs need 422 items for a five-point difference.

In production, the mistake is to pick a set size by feel and then read the result as a decision. A team that wants a five-point gain and runs a few dozen items will usually get no difference detected, and may take that for a verdict that the change doesn't work. Write the size down before the run, with the disagreement assumption, and report the result against it. No difference detected at a stated n is a finding about that set. It is not a failed change.

**Working rule:** Before a comparison, run a paired comparison on dev to estimate how often the two variants disagree, and state that share as the assumption. Compute the items needed for the difference you care about, at the level the comparison will be read at, and give its range from the interval on that share. If the set is smaller, report that it can't detect a difference that size, and don't read its no difference detected as a ruling against the change. Use the floor only as a lower bound, and only for a comparison that samples its replies.

# Check that answers rest on the cited docs, in the asked version

By the end of this chapter you can count whether an answer's citations support what it says, whether every claim in it rests on the sections the assistant was given, whether it is right for the version the reader asked about, and whether retrieval reached the right section on the way.

A correct answer can still be a poor one. The assistant in this book answers questions about sablefile from its documentation, and a reply passes when it matches the reference answer for the version the question asks about. A reply can pass that test and still cite a section that doesn't say what its sentence says, add a detail that no section gives, or take its facts from another version of the docs. A pass rate sees none of these. A reader who checks the citation beside a claim can, which is why the assistant is asked to cite at all.

Four checks sit beside the pass rate, and each has its own count over its own denominator. Citation accuracy asks whether each citation names a section the item lists as supporting. Faithfulness asks whether each claim is in the sections the assistant was shown. Version-correctness asks whether the answer fits the asked version. Outcome against path asks whether a pass came with its section in hand, and whether a fail had it. Citation accuracy and faithfulness leave declines out, since a decline makes no claim to check. Version-correctness and the retrieval table keep the declines, as the pass rate does, and each prints how many declines it holds.

## What grounded means here {#what-grounded-means}

An answer is grounded when a reader who follows its citations finds the claim where the citation says it is, in the version they asked about. Three things have to hold for that. The citation must exist in that version and support the sentence it is attached to. Every claim in the answer must follow from the sections the assistant read. And the answer must be right for that version, not merely right for some other version that uses the same words.

Take a reader who asks how long a remote fetch waits before it gives up. Suppose the reply gives a plausible answer and cites the section on fetching remote templates, but that section never says how long the wait is. A reader would take the reply as grounded, and the citation looks right. The reader who opens the section finds nothing to support the number. A pass rate would count that reply as correct if the number happened to match the reference. Faithfulness is the check that reads the sections, through a judge. The citation check reads only IDs, and says so.

These failures are different, and each needs its own count. A reply can have a supporting citation and still state a detail the section leaves out, so citation accuracy does not show faithfulness. A reply can be faithful to a section of one version and wrong for another, so faithfulness does not show version-correctness. And a reply can be right and well cited while the retrieval step never reached the section it leans on. Outcome against path exists to find that case.

**Working rule:** Give each check its own count over its own denominator, and treat a pass as the start of the reading, not the end of it.

## Citation accuracy {#citation-accuracy}

A citation is a claim about the docs: this section, in this version, supports the sentence it is attached to. Like any claim, it can be wrong. The section can exist and say something else. It can belong to another version. Or the reply can cite a section the item doesn't list. The Primer covers the sentence-level check, [checking what a cited item supports](https://rollingedition.com/primer-on-search-relevance/s/check-citation-support), and [checking citations against context](https://rollingedition.com/primer-on-search-relevance/s/check-citations-against-context) covers the context a model was given. This book's check is simpler, and [citation checks](checks-in-code#citation-checks) in chapter 4 gives its four outcomes. It matches each cited ID against the item's list of supporting sections for the asked version. It never reads the section, so it says nothing about the sentence.

The code prints two counts from the same replies. The citation rate counts every citation in answered items that is supported. The item rate counts an answered item once if at least one of its citations is supported, so one good citation carries the item. A citation to a section of the asked version that the item does not list is printed as unlisted. That section may well be right, so the check does not call it wrong, and it does not open the section to find out. Reading an unlisted citation is a hand decision. If the section supports the sentence, the item's list of supporting sections is incomplete, and the test set's owner should add it. If it does not, the citation is wrong, and the reply should be read as a failed citation.

Run it on both splits, and read the item line before the citation line.

Example `citation-accuracy-counts` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Counts the baseline's answered items that cite a supporting section of the asked version, dev and held-out apart.

An item counts once, and only if one of its citations is supported; its other citations sit in the citation lines. A decline
is not an answer, so it is left out of every count. The check matches each cited ID against the item's listed supporting
section IDs for its asked version. It never reads the section's text, so it does not check what the sentence says.
The first line of each split is the point: the item rate."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.contract import citation_outcomes, parse_reply, section_ids_by_version
from lib.prompts import P1_SHA256
from lib.replay import ReplayModel
from lib.stats import pct
from lib.testset import load_items

ids = section_ids_by_version()  # every version's section IDs, read from the docs, for telling wrong-version from missing
model = ReplayModel("assistant-baseline", P1_SHA256)  # the baseline's replies, sample 0, made from prompt P1


def supporting_ids(item):
    """The item's supporting section IDs in its asked version. Another version's listed sections are not counted here."""
    return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)


for split in ("dev", "held-out"):
    declined = 0
    results = {}  # item id -> its citation outcomes, for each answered item
    for item in load_items(split=split):
        reply = parse_reply(model.reply(item.id, 0))
        if reply.declined:
            declined += 1
        else:
            results[item.id] = citation_outcomes(reply, item.version, supporting_ids(item), ids)
    # An answer to an unanswerable item lists no section of its asked version, so it can never count as supported here.
    outcomes = [outcome for found in results.values() for outcome in found]  # every citation, in item order
    with_support = [item_id for item_id, found in results.items() if "supported" in found]
    no_support = sorted(set(results) - set(with_support))
    print(f"{split} declined items, left out: {declined}")
    print(f"{split} answered items: {len(results)}")
    print(f"{split} answered items citing a supporting section of the asked version: {pct(len(with_support), len(results))}")
    print(f"{split} answered items with no supported citation: {len(no_support)}")
    print("  " + (", ".join(no_support) or "none"))
    print(f"{split} citations: {len(outcomes)}")
    print(f"{split} citations that are supported: {pct(outcomes.count('supported'), len(outcomes))}")
    print(f"{split} citations that are unlisted: {pct(outcomes.count('unlisted'), len(outcomes))}")
    print(f"{split} citations that are wrong-version or missing: "
          f"{pct(outcomes.count('wrong-version') + outcomes.count('missing'), len(outcomes))}")
```

Output:

```text
dev declined items, left out: 23
dev answered items: 57
dev answered items citing a supporting section of the asked version: 94.7% (54 of 57)
dev answered items with no supported citation: 3
  q-0005, q-0057, q-0058
dev citations: 60
dev citations that are supported: 93.3% (56 of 60)
dev citations that are unlisted: 6.7% (4 of 60)
dev citations that are wrong-version or missing: 0.0% (0 of 60)
held-out declined items, left out: 15
held-out answered items: 25
held-out answered items citing a supporting section of the asked version: 96.0% (24 of 25)
held-out answered items with no supported citation: 1
  q-0090
held-out citations: 33
held-out citations that are supported: 93.9% (31 of 33)
held-out citations that are unlisted: 6.1% (2 of 33)
held-out citations that are wrong-version or missing: 0.0% (0 of 33)
```

On dev, the item rate is 94.7% (54 of 57).

On dev, 3 answered items have no supported citation, and the output names them. On held-out, the count is 1.

The citation rate on dev is 93.3% (56 of 60). The unlisted citations on dev are 6.7% (4 of 60), which the output reports and does not count as errors.

On held-out, the item rate is 96.0% (24 of 25), and the citation rate is 93.9% (31 of 33). Held-out has only 25 answered items, so one item moves its rate a long way. A difference between the splits would need an interval before anyone called it real, and [intervals and sample size](intervals-and-sample-size) explains how to build one.

The citations that are wrong-version or missing are the ones that would be wrong for a reason the check can name. On dev, they are 0.0% (0 of 60). On held-out, they are 0.0% (0 of 33). So the citation rates fall short of 100% because of unlisted citations, and a person has to read those.

The two rates can diverge for a simple reason. An item with one supported citation and one unlisted citation counts in the item rate and lowers the citation rate. Reading both shows whether the misses sit in whole items or in single citations inside items that otherwise hold up.

The idea behind the check has been studied outside this book. Gao and colleagues (2023), at EMNLP 2023 and [on arXiv](https://arxiv.org/abs/2305.14627), built a benchmark for citations in generated answers, and reported that many answers from strong models were not fully supported by their cited passages. Their setup was not this book's, so no number is carried over from them. The paper is cited for the idea.

**Working rule:** Read the item rate and the citation rate together, and read each unlisted citation before you change a prompt. An unlisted citation is a question for the docs, not an error to count.

## Faithfulness {#faithfulness}

Citation accuracy runs without a model, but its counts say how many citations name a listed section, not whether each sentence holds. Faithfulness is the next question, and it is the one that catches claims with no citation at all.

Citation accuracy checks only the sentences that carry a citation. A claim the reply makes without one escapes it, and so does a claim that a cited section supports only in part. Faithfulness asks a wider question: does every claim in the answer follow from the sections the assistant was shown?

Ragas (Es and colleagues, 2023, also a [preprint](https://arxiv.org/abs/2309.15217)) scores faithfulness as the share of an answer's statements that the context supports. It splits the answer into statements and checks each one. This book asks one question per answer instead. The faithfulness prompt in `lib/prompts.py` gives the judge the sections and the answer. The judge reads the answer claim by claim, and writes `VERDICT: SUPPORTED` on its first line only if every claim holds. Otherwise it writes `VERDICT: UNSUPPORTED`, then one sentence of reason.

The difference from citation accuracy shows up in the middle of an answer. Suppose a reply has two sentences. The first carries a citation to a section that supports it, and the second has none, because it adds a detail the section leaves out. Citation accuracy scores the first sentence and says nothing about the second. Faithfulness reads the whole answer, so it can mark the reply unsupported on the second sentence alone. Neither check replaces the other, which is why both are kept.

Nobody has checked this judge's verdicts against a person's, so every faithfulness rate here is uncalibrated. The judge is also the assistant's own model, Qwen3-4B-Instruct-2507, so it grades answers that model wrote. [Self-preference](when-the-judge-stops-being-reliable#self-preference) explains why that matters, and why this book cannot test it. Calibrating the judge would need human ratings of these replies, which this book doesn't have. [Calibrate the judge](calibrate-the-judge) describes the method, which compares the judge with ratings made by hand. Because these rates are not calibrated, there is nothing for the [transfer assumption](calibrate-the-judge#transfer-assumption) to carry over, and the limits in [where to stop](when-the-judge-stops-being-reliable#where-to-stop) apply to them in full.

Example `faithfulness-rate` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The faithfulness rate of assistant-baseline's answered items, as judged in judge-faithfulness-baseline, dev and held-out apart.

An answered item is one the baseline did not decline and whose reply is not empty; a decline is left out. The judge writes
SUPPORTED or UNSUPPORTED on its first line. A reply that is neither is printed by item ID and left out of the denominator,
never counted as UNSUPPORTED. The judge is uncalibrated: this book has no human ratings for it, so the rates are not
corrected for its errors and are not a calibrated measure. When every readable verdict is the same, the percentile
bootstrap cannot vary, so the example says so and prints the exact binomial bound for n of n instead."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.contract import parse_reply
from lib.judging import parse_support_verdict
from lib.prompts import FAITH_SHA256, P1_SHA256
from lib.replay import ReplayModel
from lib.stats import bootstrap_mean, interval_pct, pct
from lib.testset import load_items

answers = ReplayModel("assistant-baseline", P1_SHA256)  # the baseline's replies, sample 0, made from prompt P1
judge = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256)  # the judge's replies, made from the faithfulness prompt

print("judge: judge-4b, uncalibrated (no human ratings in this book)")
for split in ("dev", "held-out"):
    answered = []
    for item in load_items(split=split):
        reply = parse_reply(answers.reply(item.id, 0))
        if not reply.declined and reply.text:
            answered.append(item.id)
    readable = {}  # item id -> 1.0 for SUPPORTED, 0.0 for UNSUPPORTED
    for item_id in answered:
        raw = judge.reply(item_id, 0)
        verdict = parse_support_verdict(raw)
        if verdict is None:
            line = next((ln.strip().strip("*`").strip() for ln in raw.splitlines() if ln.strip()), "")
            print(f'{split} unreadable verdict, left out: {item_id} ("{line}")')
        else:
            readable[item_id] = 1.0 if verdict == "SUPPORTED" else 0.0
    scores = list(readable.values())
    lo, hi = bootstrap_mean(scores)  # level 0.95; seed SEED, 2,000 resamples
    print(f"{split} answered items: {len(answered)}")
    print(f"{split} unreadable verdicts: {len(answered) - len(readable)}")
    print(f"{split} faithfulness rate: {pct(int(sum(scores)), len(scores))}")
    print(f"{split} 95% interval of the faithfulness rate: {interval_pct(lo, hi)}")
    if len(set(scores)) == 1:  # every readable verdict is the same, so resampling cannot vary
        print(f"{split} bootstrap interval is degenerate: all {len(scores)} readable verdicts are the same")
        if sum(scores) == len(scores):
            print(f"{split} exact 95% lower bound for {len(scores)} of {len(scores)}: {100 * 0.025 ** (1 / len(scores)):.1f}%")
    if split == "dev":
        unfaithful = sorted(item_id for item_id, score in readable.items() if score == 0.0)
        print(f"dev unfaithful items, first five: {', '.join(unfaithful[:5])}")
```

Output:

```text
judge: judge-4b, uncalibrated (no human ratings in this book)
dev answered items: 57
dev unreadable verdicts: 0
dev faithfulness rate: 94.7% (54 of 57)
dev 95% interval of the faithfulness rate: [87.7%, 100.0%]
dev unfaithful items, first five: q-0013, q-0029, q-0055
held-out answered items: 25
held-out unreadable verdicts: 0
held-out faithfulness rate: 100.0% (25 of 25)
held-out 95% interval of the faithfulness rate: [100.0%, 100.0%]
held-out bootstrap interval is degenerate: all 25 readable verdicts are the same
held-out exact 95% lower bound for 25 of 25: 86.3%
```

The example takes the baseline's answered items, leaves out the declines, and asks the judge about each one. It prints a rate only over verdicts it can read. A reply that is neither SUPPORTED nor UNSUPPORTED is printed by item and left out of the denominator. It is never counted as a failure, as [verdict first](a-judge-with-a-rubric#verdict-first) explains.

The uncalibrated faithfulness rate on dev is 94.7% (54 of 57). Its interval is [87.7%, 100.0%]. The top of that interval sits on the upper boundary of the rate scale, because a bootstrap cannot resample past it. The interval is still wide, which is the honest reading of 57 answered items.

The uncalibrated faithfulness rate on held-out is 100.0% (25 of 25). Its printed interval is [100.0%, 100.0%], but that is not an interval. The output says the bootstrap is degenerate: all 25 readable verdicts are the same. A resample of items that all pass can only return the same rate, so the bootstrap says nothing about where the rate lies.

The exact 95% lower bound is 86.3%. Below that true rate, all of the held-out replies passing would happen with probability under 2.5%. The bound is 0.025 to the power of one over n. It shows that the judge passed these replies, on a split too small to show where the rate lies. Do not read the held-out figure as a finding that the assistant is faithful.

The two checks print their own lists. The citation output names the items with no supported citation, and the faithfulness output names the first unfaithful dev items. Read the two lists side by side, since this chapter does not assume they name the same items.

**Working rule:** Put "uncalibrated" beside every faithfulness rate, and read the first unfaithful items before you trust the rate. A faithfulness check shows where the judge says a claim is unsupported. It does not show that the judge is right.

## Version-correctness {#version-correctness}

sablefile has three versions of its docs, 1.0, 1.1 and 2.0, and some questions have different answers in each. A reply can be grounded in a section of 1.1 and still be wrong for a reader on 2.0. The book calls these [version-dependent items](#the-twenty-items). Each has a reference answer for the version the reader asked about, and its check asks whether the reply matches that reference. There are 17 of them on dev and 8 on held-out.

Version-dependent items count in the pass rate with every other item, and they also get their own count, beside it. A reply to one of them is judged against the reference for the asked version, so a correct answer for another version fails. That miss would look like a weak assistant if it were only mixed into the general rate. The judged-correct rate is the count for version-dependent items alone, so a version miss shows on its own line, and a team can see whether a change to retrieval moved it. Declines stay in that denominator, as they do in the pass rate, and each split prints how many it holds.

A wrong-version citation is a separate count. It is a citation to a section that exists only in another version. A reply can be right for the asked version and still cite another version's section, and a reader who copies that citation lands in the wrong docs.

This check sees only what the ID tells it. A citation carries an ID, not a version. Most IDs repeat across versions, so a section ID that appears in 1.0, 1.1 and 2.0 looks the same in each. An ID that appears in every version can never be flagged as wrong-version, whichever version the reader asked about. The check can flag a citation only when the asked version lacks its ID and another version has it. The index matters too. A citation to another version is possible only when that version's sections are in the index the reply was built from, so the count has to be read again when the index changes, and a zero from an old index says nothing about a new one.

Two variants are read here. The baseline retrieves from the index of the asked version only. The unfiltered variant retrieves from the indexes of every version together, and shows each section's version in its header, so the model can see where a section comes from. The unfiltered variant is the one that could cite another version, since it is shown them.

Example `version-correct-answers` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Reads the 25 version-dependent items for two variants: whether the judge passed the answer for the asked version, and whether a
citation names a section that exists only in another version.

The judge's verdict for each variant is read from its own recording, and an unreadable verdict is printed and left out of the
pass rate, never counted as a fail. A declined reply has a verdict like any other reply, so it stays in the denominator; its
count is printed beside each rate. A wrong-version citation is counted per item, over every item, so a zero shows as 0 of 25.
The last lines list the items where the two variants disagree, to read item by item."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import readable_verdicts
from lib.contract import citation_outcomes, parse_reply, section_ids_by_version
from lib.prompts import P1_SHA256, REF_SHA256
from lib.replay import ReplayModel
from lib.stats import pct
from lib.testset import load_items

ids = section_ids_by_version()  # every version's section IDs, so a section of another version is known as one
# label, the assistant's recording, the judge's recording of that assistant's replies
VARIANTS = (("baseline", "assistant-baseline", "judge-answers-baseline"), ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered"))
items = [item for item in load_items() if item.kind == "version-dependent"]
item_ids = [item.id for item in items]
listed = {item.id: frozenset(s.section_id for s in item.supporting_sections if s.version == item.version) for item in items}
verdict, wrong = {}, {}  # label -> {item id: PASS or FAIL, readable only}; label -> item ids with a wrong-version citation
for label, assistant, judge in VARIANTS:
    verdict[label], unreadable = readable_verdicts(judge, item_ids, REF_SHA256)
    for item_id, first in unreadable:
        print(f'{label} unreadable verdict, left out: {item_id} ("{first}")')
    replies = ReplayModel(assistant, P1_SHA256)
    wrong[label] = {item.id for item in items
                    if "wrong-version" in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed[item.id], ids)}
    for split in ("dev", "held-out"):
        split_items = [item for item in items if item.split == split]
        readable = [item.id for item in split_items if item.id in verdict[label]]
        passed = sum(verdict[label][item_id] == "PASS" for item_id in readable)
        print(f"{label} {split} version-dependent items: {len(split_items)}")
        print(f"{label} {split} declined replies, counted in the denominator: "
              f"{sum(parse_reply(replies.reply(item.id, 0)).declined for item in split_items)}")
        print(f"{label} {split} judged correct for the asked version: {pct(passed, len(readable))}")
        print(f"{label} {split} citing a wrong-version section: "
              f"{pct(sum(item.id in wrong[label] for item in split_items), len(split_items))}")
    print(f"{label} all version-dependent items citing a wrong-version section: {pct(len(wrong[label]), len(items))}")

both = [item_id for item_id in item_ids if item_id in verdict["baseline"] and item_id in verdict["unfiltered"]]
print(f"items with a readable verdict in both variants: {len(both)}")
differ = [i for i in both if verdict["baseline"][i] != verdict["unfiltered"][i]]
print(f"items where the verdicts differ: {len(differ)}")
print("  " + (", ".join(differ) or "none"))
flag_differ = [i for i in item_ids if (i in wrong["baseline"]) != (i in wrong["unfiltered"])]
print(f"items where the wrong-version flag differs: {len(flag_differ)}")
print("  " + (", ".join(flag_differ) or "none"))
```

Output:

```text
baseline dev version-dependent items: 17
baseline dev declined replies, counted in the denominator: 4
baseline dev judged correct for the asked version: 64.7% (11 of 17)
baseline dev citing a wrong-version section: 0.0% (0 of 17)
baseline held-out version-dependent items: 8
baseline held-out declined replies, counted in the denominator: 2
baseline held-out judged correct for the asked version: 75.0% (6 of 8)
baseline held-out citing a wrong-version section: 0.0% (0 of 8)
baseline all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
unfiltered dev version-dependent items: 17
unfiltered dev declined replies, counted in the denominator: 5
unfiltered dev judged correct for the asked version: 52.9% (9 of 17)
unfiltered dev citing a wrong-version section: 0.0% (0 of 17)
unfiltered held-out version-dependent items: 8
unfiltered held-out declined replies, counted in the denominator: 2
unfiltered held-out judged correct for the asked version: 75.0% (6 of 8)
unfiltered held-out citing a wrong-version section: 0.0% (0 of 8)
unfiltered all version-dependent items citing a wrong-version section: 0.0% (0 of 25)
items with a readable verdict in both variants: 25
items where the verdicts differ: 6
  q-0003, q-0021, q-0022, q-0033, q-0082, q-0085
items where the wrong-version flag differs: 0
  none
```

Across every version-dependent item, the wrong-version share is 0.0% (0 of 25) for the baseline and 0.0% (0 of 25) for the unfiltered variant. A zero says only that the check flagged no ID. The next example shows how few IDs it could have flagged.

Example `version-check-detail` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The two counts behind the version check, and the version-dependent items where the variants' verdicts differ, by split.

A wrong-version citation is one whose ID exists only in another version, so an ID that several versions share cannot be
attributed to one of them. This prints how many section IDs are found in one version only, then how many citations to the
version-dependent items name a shared ID, for each variant. The last lines list the items where the verdicts differ, by split."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter

from lib.calibration import readable_verdicts
from lib.contract import parse_reply, section_ids_by_version
from lib.prompts import P1_SHA256, REF_SHA256
from lib.replay import ReplayModel
from lib.stats import pct
from lib.testset import load_items

ids = section_ids_by_version()
shared = Counter(i for found in ids.values() for i in found)  # how many versions each section ID appears in
print("section IDs found in one version only, by version: "
      + ", ".join(f"{v} {sum(shared[i] == 1 for i in found)}" for v, found in ids.items()))
items = [item for item in load_items() if item.kind == "version-dependent"]
split_of = {item.id: item.split for item in items}
verdict = {}  # label -> {item id: PASS or FAIL}, readable only
for label, assistant, judge in (("baseline", "assistant-baseline", "judge-answers-baseline"),
                                ("unfiltered", "assistant-unfiltered", "judge-answers-unfiltered")):
    replies = ReplayModel(assistant, P1_SHA256)
    cited = [cid for item in items for cid in parse_reply(replies.reply(item.id, 0)).citations]
    print(f"{label} citations to a section ID found in more than one version: "
          f"{pct(sum(shared[cid] > 1 for cid in cited), len(cited))}")
    verdict[label], _ = readable_verdicts(judge, list(split_of), REF_SHA256)
differ = [i for i in verdict["baseline"] if i in verdict["unfiltered"] and verdict["baseline"][i] != verdict["unfiltered"][i]]
for split in ("dev", "held-out"):
    part = [i for i in differ if split_of[i] == split]
    print(f"{split} items where the verdicts differ: {len(part)}")
    print("  " + (", ".join(part) or "none"))
```

Output:

```text
section IDs found in one version only, by version: 1.0 0, 1.1 0, 2.0 4
baseline citations to a section ID found in more than one version: 87.0% (20 of 23)
unfiltered citations to a section ID found in more than one version: 63.2% (12 of 19)
dev items where the verdicts differ: 4
  q-0003, q-0021, q-0022, q-0033
held-out items where the verdicts differ: 2
  q-0082, q-0085
```

The section IDs found in only one version, counted by version, are 1.0 0, 1.1 0, 2.0 4. An ID found in more than one version can still be flagged, but only for a reader on a version that lacks it. For the baseline, the share of citations to version-dependent items that name a shared ID is 87.0% (20 of 23). For the unfiltered variant it is 63.2% (12 of 19).

The judged-correct rates differ, though each is read on a small split. On dev, the baseline is judged correct on 64.7% (11 of 17).

The unfiltered variant is judged correct on 52.9% (9 of 17). On these items, the unfiltered variant passes fewer.

On held-out, the baseline is judged correct on 75.0% (6 of 8). The unfiltered variant has the same rate, but it passes different items. The verdicts go in opposite directions on q-0082 and q-0085, and the other held-out items agree.

Across both variants, the verdicts differ on 6 items, out of 25 with a readable verdict in each. On dev, 4 of them differ, and on held-out, 2. The output lists them by split.

No interval is computed for these differences, and a dev split of this size is small. So this chapter reports the counts and does not call either variant better or worse. The counts say three things: on these items the unfiltered variant passed fewer, no citation was flagged as wrong-version on either variant, and the differing verdicts are the place to read first.

**Working rule:** Read each citation's ID with the version of the answer beside it. A zero from this check means no flagged ID, not no wrong-version citation. Keep version-dependent items in their own count beside the pass rate, so a version miss shows up on its own line.

## Outcome and path {#outcome-and-path}

Outcome is whether a reply passed. Path is what the steps before the reply reached. In this book the path is short: one retrieval, then one reply. Two combinations need both counts. A pass whose supporting section never reached the model came from somewhere else, perhaps the model's own knowledge, and that is a warning even when the verdict is right. A fail with the section retrieved is the case the other checks exist to explain.

Runs with several steps have longer paths. [Agent Harnesses](https://rollingedition.com/agent-harnesses/s/t18-path) counts a task through its path, and that book is the place to read about it. This chapter scores the one step the assistant has.

### The retrieval step {#the-retrieval-step}

Retrieval is BM25 over the sections of the asked version. It is deterministic: the same question gets the same top sections every time. So this step needs no recording. The code computes the top sections for each item from the index, and sets them against the verdict the judge gave the baseline's reply.

Dev items with no supporting section of the asked version are left out of the table. Those are the unanswerable items, since no section of their asked version exists for retrieval to find. Their count is 13.

Example `retrieval-path` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Sets the baseline's judge verdict against whether a supporting section reached the top k, for each dev item that has one.

Retrieval is BM25 over the asked version, so it is the same on every run and needs no recording; k is the baseline's, read from
its manifest. The two diagonal cells are the expected pairs. The two off-diagonal cells are the ones to read: a pass without the
right section retrieved, and a fail with it retrieved. Unanswerable items are left out, since they have no section of their asked
version to retrieve; their count is printed. An unreadable verdict is printed and left out, never counted as a fail. A declined reply is counted in its cell like any
other reply; the count of declines is printed. The FAIL cell is split into declines and answers in fail-with-section."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.answers import Assistant
from lib.calibration import readable_verdicts
from lib.contract import parse_reply
from lib.prompts import P1_SHA256, P1_SYSTEM, REF_SHA256
from lib.replay import ReplayModel
from lib.testset import load_items

RECORDING = "assistant-baseline"
k = ReplayModel(RECORDING, P1_SHA256).manifest["k"]  # the top k the baseline's prompt was built from, read from its manifest
assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k)  # its retriever only; the recording is not read here
replies = ReplayModel(RECORDING, P1_SHA256)  # the baseline's replies, sample 0, made from prompt P1, for the decline check

dev = load_items(split="dev")
has_support = [item for item in dev if any(s.version == item.version for s in item.supporting_sections)]
print(f"top k: {k}")
print(f"dev items: {len(dev)}")
print(f"dev items with no supporting section of the asked version, left out: {len(dev) - len(has_support)}")

verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in has_support], REF_SHA256)
for item_id, first in unreadable:
    print(f'unreadable verdict, left out: {item_id} ("{first}")')

# Keyed by (retrieved, passed): the four cells of the 2 by 2 table, each a list of item ids in sorted order.
cells = {(True, True): [], (True, False): [], (False, True): [], (False, False): []}
declined = set()  # item ids whose reply is a decline; they sit in their cells like any other reply
for item in has_support:
    if item.id not in verdicts:
        continue
    listed = {s.section_id for s in item.supporting_sections if s.version == item.version}
    top = {section.section_id for section in assistant.sections(item)}  # the top k sections, by BM25 rank
    cells[(bool(listed & top), verdicts[item.id] == "PASS")].append(item.id)
    if parse_reply(replies.reply(item.id, 0)).declined:
        declined.add(item.id)

print(f"items in the table: {sum(len(ids) for ids in cells.values())}")
print(f"declined items in the table, counted in their cells: {len(declined)}")
print(f"right section in the top {k}, PASS: {len(cells[(True, True)])}")
print(f"right section in the top {k}, FAIL: {len(cells[(True, False)])}")
print(f"right section not in the top {k}, PASS: {len(cells[(False, True)])}")
print(f"right section not in the top {k}, FAIL: {len(cells[(False, False)])}")
print("passed without the right section retrieved: " + (", ".join(cells[(False, True)]) or "none"))
print("failed with the right section retrieved: " + (", ".join(cells[(True, False)]) or "none"))
```

Output:

```text
top k: 3
dev items: 80
dev items with no supporting section of the asked version, left out: 13
unreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
items in the table: 66
declined items in the table, counted in their cells: 10
right section in the top 3, PASS: 49
right section in the top 3, FAIL: 10
right section not in the top 3, PASS: 0
right section not in the top 3, FAIL: 7
passed without the right section retrieved: none
failed with the right section retrieved: q-0019, q-0023, q-0025, q-0029, q-0033, q-0045, q-0046, q-0048, q-0063, q-0074
```

The top k is 3, read from the baseline's manifest. The table holds 66 items, and 10 of them are declines. A decline sits in its cell like any other reply, since the judge gave it a verdict. Each cell pairs a verdict with whether a supporting section reached the top k.

| Verdict | Supporting section in the top k | No supporting section in the top k |
| --- | --- | --- |
| PASS | 49 | 0 |
| FAIL | 10 | 7 |

Read the PASS row first. The cell for passes with the section in the top k holds 49 passes, and the cell for passes without it holds 0. That is consistent with passes coming from the docs. It does not show that they did, because the table counts how often two things occur together, not which one caused the other. The count of passes is also a check on the reference. A pass with no section retrieved can come from a reply that repeats a common fact, or from a reference answer a reader could guess, and the table cannot say which. Reading those items can.

The FAIL row is where the other checks come in. Of the failures with the section in the top k, 5 are declines, and the rest are answers. [Declining when the docs don't say](declining-when-the-docs-dont-say) covers what a decline means when the docs hold the answer. The example below splits the cell.

Example `fail-with-section` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The dev items the baseline failed although a listed section of the asked version was in its top k, split into declines and answers.

For each answer it prints whether the reply cites a listed section, and the faithfulness judge's verdict on it. A decline has no
faithfulness verdict, because the judge reads only answers. An unreadable verdict is printed and left out, never counted as a fail."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import re

from lib.answers import Assistant
from lib.calibration import readable_verdicts
from lib.contract import citation_outcomes, parse_reply, section_ids_by_version
from lib.prompts import FAITH_SHA256, P1_SHA256, P1_SYSTEM, REF_SHA256
from lib.replay import ReplayModel
from lib.testset import load_items

RECORDING = "assistant-baseline"
k = ReplayModel(RECORDING, P1_SHA256).manifest["k"]  # the baseline's top k, as in retrieval-path
assistant = Assistant(RECORDING, P1_SYSTEM, P1_SHA256, k=k)  # its retriever only
replies = ReplayModel(RECORDING, P1_SHA256)  # the baseline's replies, sample 0, made from prompt P1
faith = ReplayModel("judge-faithfulness-baseline", FAITH_SHA256)
ids = section_ids_by_version()
VERDICT = re.compile(r"^VERDICT:\s*(SUPPORTED|UNSUPPORTED)$")


def listed(item):
    """The item's supporting section IDs in its asked version."""
    return frozenset(s.section_id for s in item.supporting_sections if s.version == item.version)


dev = load_items(split="dev")
verdicts, unreadable = readable_verdicts("judge-answers-baseline", [item.id for item in dev], REF_SHA256)
for item_id, first in unreadable:
    print(f'unreadable verdict, left out: {item_id} ("{first}")')
fails = [item for item in dev if verdicts.get(item.id) == "FAIL" and listed(item) & {s.section_id for s in assistant.sections(item)}]
declined = [item for item in fails if parse_reply(replies.reply(item.id, 0)).declined]
answered = [item for item in fails if item not in declined]
print(f"right section in the top {k}, FAIL, declined: {len(declined)}")
print(f"right section in the top {k}, FAIL, answered: {len(answered)}")
print(f"answered FAIL with the section, citing a listed section: "
      f"{sum('supported' in citation_outcomes(parse_reply(replies.reply(item.id, 0)), item.version, listed(item), ids) for item in answered)}")
judged = {}  # item id -> SUPPORTED, UNSUPPORTED or unreadable, for each answered fail
for item in answered:
    line = next((ln.strip().strip("*`").strip() for ln in faith.reply(item.id, 0).splitlines() if ln.strip()), "")
    match = VERDICT.match(line)
    judged[item.id] = match.group(1) if match else "unreadable"
print(f"answered FAIL with the section, faithfulness SUPPORTED: {sum(v == 'SUPPORTED' for v in judged.values())}")
print(f"answered FAIL with the section, faithfulness UNSUPPORTED: {sum(v == 'UNSUPPORTED' for v in judged.values())}")
print("  " + (", ".join(i for i, v in judged.items() if v == "UNSUPPORTED") or "none"))
print(f"answered FAIL with the section, faithfulness unreadable: {sum(v == 'unreadable' for v in judged.values())}")
```

Output:

```text
unreadable verdict, left out: q-0043 ("VERDICT: PARTIAL")
right section in the top 3, FAIL, declined: 5
right section in the top 3, FAIL, answered: 5
answered FAIL with the section, citing a listed section: 5
answered FAIL with the section, faithfulness SUPPORTED: 4
answered FAIL with the section, faithfulness UNSUPPORTED: 1
  q-0029
answered FAIL with the section, faithfulness unreadable: 0
```

Of the answered failures, 5 cite a section the item lists. The faithfulness judge passes 4 of them and marks 1 unsupported, which is q-0029. The counts here do not show whether the wrong refusals, the version misses or the answer judge's errors account for the rest. Reading the replies can.

One verdict reads PARTIAL, not PASS or FAIL. The judge's verdict on q-0043 is PARTIAL, so the table leaves it out. An unreadable verdict is never counted as a fail.

**Working rule:** Treat retrieval as a gate on passes and as a lead on fails. A pass without the supporting section needs an explanation, and a fail with it points at the reply.

### Where agent harnesses picks up {#where-agent-harnesses-picks-up}

When a task takes several steps, each one may call a tool and change what the next step sees. The outcome can pass while a middle step went wrong, so the count has to be taken over the whole path. This book scores one retrieval and one reply, so its path count is a yes or no per item. The section on a task counted through a path in [Agent Harnesses](https://rollingedition.com/agent-harnesses/s/t18-path) covers how to score runs of that kind, and this chapter does not repeat it.

**Working rule:** In this book the path is one retrieval and one reply, so read the retrieval count beside the verdict for each item.

## Retrieval quality belongs to another book {#retrieval-quality}

How well retrieval orders the sections, and how much of the context it fills, is a question about ranking, and it belongs to *A Primer on Search Relevance*. Its [choosing a ranking metric](https://rollingedition.com/primer-on-search-relevance/s/choose-a-ranking-metric) section covers recall at k, mean reciprocal rank and nDCG, and how to choose among them. [Measuring the context](https://rollingedition.com/primer-on-search-relevance/s/measure-the-context) covers the context a model is given. Read both before you pick a metric for this assistant.

The retrieval-path count asks one question of each item: is a supporting section of the asked version in the top k? It returns yes or no, so it is not a ranking metric. It cannot say how far down the list that section sits, and that is what a ranking metric is for. Its k is read from the baseline's manifest and is not chosen in this chapter, so a change to k moves the table, and the table has to be read again.

Taken together, the four checks say that a pass rate on this assistant hides three different ways of being wrong. The reply can cite the wrong thing, state what no section says, or answer for the wrong version. Each one needs its own count, its own denominator and its own word for what it found. The decisions that follow from the results are these:

- Each reply is checked for citations, faithfulness and version, and each check keeps its own count.
- Every faithfulness rate is printed as uncalibrated until someone checks the judge against people.
- Version-dependent items keep their own count beside the pass rate, and their own version check.
- Retrieval is a gate on passes and a lead on fails. A pass whose section was not retrieved is read before it is counted as grounded.

**Working rule:** Before you call a reply grounded, read its citation, its claims and its version, and read the retrieval that came before it.

# Decline when the docs don't say, and measure both kinds of error

By the end of this chapter you can count both ways an assistant fails to answer from its docs, read the two rates over their own denominators, and say when a rule that declines more often is worth what it costs.

The assistant in this book answers questions about sablefile, a command-line tool that keeps a project's generated files in step with the templates they come from, and it answers from the tool's documentation. When the documentation doesn't cover a question, the assistant is meant to reply with one fixed sentence, "The docs don't say.", and nothing else. That sentence looks like the safe answer. A team that sees a wrong reply in its logs is tempted to make the assistant decline more often, because a refusal looks careful. A refusal can be wrong too. If the docs did answer the question, the user who got the sentence was failed, and no error shows anywhere. If the docs did not answer and the assistant answered anyway, the user has a reply with nothing behind it. Declining more often trades one of these errors for the other. A team that counts only one of them will tune toward a number that looks good and hides the cost of the other.

This chapter counts both errors on the book's own replies. It then tests the obvious fix, a guard that declines when the search score of the top section is low, on the unfiltered variant. On that variant the guard does not help. That is the chapter's result, and most of what follows is about how the result was measured and how far it can be trusted.

## Two errors, both counted {#two-errors}

Every question in the test set has a kind. An answerable question is one that the docs of the asked version answer. An unanswerable question is one they don't. A reply that declines an answerable question makes a [false refusal](#declines-in-code), and a reply that answers an unanswerable question makes a [false answer](#declines-in-code). The first chapter defined both in its [decline check](measure-before-you-change#declines-in-code), and this chapter uses the same definitions on more items.

The two rates have different denominators, and the denominators carry the point. The false refusal rate is the number of answerable items declined, over the number of answerable items. The false answer rate is the number of unanswerable items answered, over the number of unanswerable items. Put both errors over one total, and a set with many answerable items makes the refusal count look small, while the answers that should have been declines get lost in the sum. Each rate needs its own count, and each count needs its own denominator.

Two kinds of item stay out of both rates. A [version-dependent item](#the-twenty-items) asks about something that differs between versions, so its reply is judged against the reference for the asked version, and its own rate, version-correctness, is covered in [chapter 9](grounded-answers#version-correctness). An ambiguous item has two readings and is judged by the pass rate, not by either decline rate. A decline on either kind is not an error here. The code still counts those declines and prints them, so a decline on one of them can't go unnoticed.

The code counts a reply as a decline only when the reply is the decline sentence and nothing more. A reply that gives the sentence and then an answer is not a decline, so it counts as an answer in both rates. The same holds for a refusal in other words: the code matches the sentence exactly, so a refusal worded differently counts as an answer. On an answerable item that is not an error here, and on an unanswerable item it is a false answer, which is why the sentence is fixed in the prompt.

**Working rule:** Report the two decline rates apart, each over its own denominator, and leave version-dependent and ambiguous items out of both. Read the items behind each rate, not only the rate, because a rate can hide the few items that matter.

## What the baseline does {#what-the-baseline-does}

The baseline is the assistant as the first chapter built it. It runs prompt P1, which asks for a citation on every statement and the decline sentence when the sections don't say. It retrieves the top three sections with BM25, from the index of the asked version only. Its replies were recorded once per item, and the example reads that one recorded reply.

Three variants change one thing each. `prompt-b` runs prompt P2, which quotes the sentence the answer relies on before it gives the answer. `k5` gives the model the top five sections instead of three. `unfiltered` retrieves from the indexes of every version together, with each section's version in its header, so the model can see when a section comes from another version. Each variant has its own recording of replies to the same items, so every comparison is between replies to the same questions.

Counting both errors for each recording takes one example, decline-rates. It reads all four recordings on the dev split and on the held-out split, prints each rate with its counts, and names the items behind each error. Its output shows which answerable items each recording declined and which unanswerable items it answered.

Example `decline-rates` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Counts the two decline errors for each recording, on dev and on held-out: answerable items declined (false refusals)
and unanswerable items answered (false answers). Each rate line rests on its own count and denominator.

Version-dependent and ambiguous items are left out of both rates; the last line per split counts the declines among
them, which are not errors here."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.contract import parse_reply
from lib.prompts import P1_SHA256, P2_SHA256
from lib.replay import ReplayModel
from lib.stats import pct
from lib.testset import load_items

VARIANTS = (  # label, recording, the sha256 of the prompt it was made from
    ("baseline", "assistant-baseline", P1_SHA256),  # P1, k=3, filtered to the asked version
    ("unfiltered", "assistant-unfiltered", P1_SHA256),  # P1, k=3, sections from every version
    ("prompt-b", "assistant-prompt-b", P2_SHA256),  # P2, k=3, filtered
    ("k5", "assistant-k5", P1_SHA256),  # P1, k=5, filtered
)

for label, recording, prompt_sha256 in VARIANTS:
    model = ReplayModel(recording, prompt_sha256)  # refuses a recording made from another prompt
    for split in ("dev", "held-out"):
        items = load_items(split=split)
        declined = {item.id: parse_reply(model.reply(item.id, 0)).declined for item in items}  # sample 0 only
        answerable = [item for item in items if item.kind == "answerable"]
        unanswerable = [item for item in items if item.kind == "unanswerable"]
        refused = sorted(item.id for item in answerable if declined[item.id])
        answered = sorted(item.id for item in unanswerable if not declined[item.id])
        other_kinds = ("version-dependent", "ambiguous")
        other = [item.id for item in items if item.kind in other_kinds and declined[item.id]]
        print(f"{label} {split} false refusal rate (answerable declined): {pct(len(refused), len(answerable))}")
        print(f"{label} {split} false refusals: {', '.join(refused) or 'none'}")
        print(f"{label} {split} false answer rate (unanswerable answered): {pct(len(answered), len(unanswerable))}")
        print(f"{label} {split} false answers: {', '.join(answered) or 'none'}")
        for item_id in answered:  # what each false answer cites, and its text, so the chapter's claims can be checked
            reply = parse_reply(model.reply(item_id, 0))
            print(f"{label} {split} reply to {item_id} cites: {', '.join(reply.citations) or 'none'}")
            print(f"{label} {split} reply to {item_id} text: {reply.text}")
        print(f"{label} {split} declined, not counted (version-dependent and ambiguous): {len(other)}")
```

Output:

```text
baseline dev false refusal rate (answerable declined): 10.0% (4 of 40)
baseline dev false refusals: q-0052, q-0060, q-0063, q-0079
baseline dev false answer rate (unanswerable answered): 0.0% (0 of 13)
baseline dev false answers: none
baseline dev declined, not counted (version-dependent and ambiguous): 6
baseline held-out false refusal rate (answerable declined): 30.0% (6 of 20)
baseline held-out false refusals: q-0101, q-0102, q-0103, q-0105, q-0111, q-0118
baseline held-out false answer rate (unanswerable answered): 0.0% (0 of 7)
baseline held-out false answers: none
baseline held-out declined, not counted (version-dependent and ambiguous): 2
unfiltered dev false refusal rate (answerable declined): 12.5% (5 of 40)
unfiltered dev false refusals: q-0005, q-0052, q-0058, q-0060, q-0079
unfiltered dev false answer rate (unanswerable answered): 15.4% (2 of 13)
unfiltered dev false answers: q-0018, q-0039
unfiltered dev reply to q-0018 cites: remote-retries
unfiltered dev reply to q-0018 text: Yes, sablefile 1.1 can fetch a remote template from a private URL that needs a login token, as long as the token is included in the request. The retry behavior for such fetches is up to 5 times before giving up, and this applies to `sync` and `diff` operations [sec:remote-retries]. The `check` command does not fetch anything and thus does not apply to such scenarios.
unfiltered dev reply to q-0039 cites: check-exit-status
unfiltered dev reply to q-0039 text: When check finds a drifted file, it prints `DRIFT` and the path, and exits with status 1. [sec:check-exit-status]
unfiltered dev declined, not counted (version-dependent and ambiguous): 9
unfiltered held-out false refusal rate (answerable declined): 45.0% (9 of 20)
unfiltered held-out false refusals: q-0101, q-0102, q-0103, q-0105, q-0106, q-0111, q-0115, q-0116, q-0117
unfiltered held-out false answer rate (unanswerable answered): 0.0% (0 of 7)
unfiltered held-out false answers: none
unfiltered held-out declined, not counted (version-dependent and ambiguous): 3
prompt-b dev false refusal rate (answerable declined): 7.5% (3 of 40)
prompt-b dev false refusals: q-0052, q-0063, q-0079
prompt-b dev false answer rate (unanswerable answered): 0.0% (0 of 13)
prompt-b dev false answers: none
prompt-b dev declined, not counted (version-dependent and ambiguous): 4
prompt-b held-out false refusal rate (answerable declined): 25.0% (5 of 20)
prompt-b held-out false refusals: q-0101, q-0102, q-0103, q-0105, q-0111
prompt-b held-out false answer rate (unanswerable answered): 0.0% (0 of 7)
prompt-b held-out false answers: none
prompt-b held-out declined, not counted (version-dependent and ambiguous): 0
k5 dev false refusal rate (answerable declined): 7.5% (3 of 40)
k5 dev false refusals: q-0052, q-0062, q-0079
k5 dev false answer rate (unanswerable answered): 0.0% (0 of 13)
k5 dev false answers: none
k5 dev declined, not counted (version-dependent and ambiguous): 5
k5 held-out false refusal rate (answerable declined): 30.0% (6 of 20)
k5 held-out false refusals: q-0102, q-0103, q-0105, q-0106, q-0111, q-0116
k5 held-out false answer rate (unanswerable answered): 0.0% (0 of 7)
k5 held-out false answers: none
k5 held-out declined, not counted (version-dependent and ambiguous): 2
```

The baseline's false refusal rate on dev is 10.0% (4 of 40). Its false refusals are `q-0052`, `q-0060`, `q-0063` and `q-0079`, and each one is an answerable item that the docs cover. Its false answer rate on dev is 0.0% (0 of 13), so it answers none of the unanswerable dev items. Chapter 1 also found no false answer, on only four unanswerable items, which is too few to say much about a rate. The baseline also declined 6 dev items that belong to the two kinds left out of both rates.

[Chapter 2](read-the-outputs) counts 10 over-declines for this baseline. That count covers every item the docs answer, version-dependent ones included, so it holds the false refusals above and the declines on version-dependent and ambiguous items, which this rate leaves out.

The table puts all four recordings side by side, on both splits. Each cell holds a rate with its counts.

| Recording | Dev false refusals | Dev false answers | Held-out false refusals | Held-out false answers |
| --- | --- | --- | --- | --- |
| baseline | 10.0% (4 of 40) | 0.0% (0 of 13) | 30.0% (6 of 20) | 0.0% (0 of 7) |
| unfiltered | 12.5% (5 of 40) | 15.4% (2 of 13) | 45.0% (9 of 20) | 0.0% (0 of 7) |
| prompt-b | 7.5% (3 of 40) | 0.0% (0 of 13) | 25.0% (5 of 20) | 0.0% (0 of 7) |
| k5 | 7.5% (3 of 40) | 0.0% (0 of 13) | 30.0% (6 of 20) | 0.0% (0 of 7) |

Read the unfiltered row first, because it is the only row with false answers on dev. Its false refusal rate is 12.5% (5 of 40). The baseline's is 10.0% (4 of 40). Unfiltered declines two answerable items that the baseline answers, `q-0005` and `q-0058`, and it answers `q-0063`, which the baseline declines. The net is one more declined item of forty. Each refusal difference from the baseline on dev is one item of forty. On these items, one item is no basis for a conclusion about which recording is better, and [intervals and sample size](intervals-and-sample-size) explains what a sample this size can show.

The unfiltered variant also answers two unanswerable dev items, where the baseline answers none. Its false answer rate is 15.4% (2 of 13). That is a count of two items with no interval, so this chapter reports it as a count and not as a measured rise. The two false answers are `q-0018` and `q-0039`.

Both questions are about sablefile 1.1, and the 1.1 docs leave out both answers: whether a remote template can be fetched from a private address with a login token, and what exit status `diff` returns when it finds differences. The baseline declines both. The unfiltered replies answer both, and each cites a neighbouring section. The reply to `q-0018` cites `remote-retries`, the retry rule for remote fetches. The reply to `q-0039` cites `check-exit-status` and gives the exit status of `check`, a different command from the `diff` the question asks about. The example prints each citation and each reply's text, so these claims can be checked without opening the recording. The unfiltered variant changes retrieval, not the prompt, so the citations are the place to look first.

The variants that move the dev refusals the other way are `prompt-b` and `k5`. Each has one fewer dev false refusal than the baseline, and neither answers an unanswerable dev item. The false refusal rate for `prompt-b` is 7.5% (3 of 40). For `k5` it is 7.5% (3 of 40). On these items, each is one fewer than the baseline, and one item is no basis for a conclusion.

The held-out figures are reported for completeness. The false refusal rate for `prompt-b` is 25.0% (5 of 20). For `k5` it is 30.0% (6 of 20). For the baseline it is 30.0% (6 of 20). The gap between `prompt-b` and the baseline is one item of twenty, and on these items it is no basis for a conclusion. No variant is chosen in this chapter, because the dev differences are a single item each.

**Working rule:** Read each false refusal on the dev split by hand before you change a prompt, since one of them may point at a real gap in the docs. Compare variants on the same items, and report a paired interval only where an example prints one. Where none is printed, report the counts on these items and draw no conclusion about which variant is better. Run the held-out split once, to confirm a choice made on dev. The unfiltered variant is the warning here: on dev it answers unanswerable items that the baseline declines. This chapter reports that without an interval, so it is a reason to check, not a measured rise.

## A guard on recorded replies {#a-guard}

The guard rests on a plain idea. BM25 scores each section by the words it shares with the question. When the docs have nothing on a question, the best-matching section should share few words with it, so its score should be low. A guard reads the best-matching section's score and declines when it falls below a threshold. The idea is a hypothesis, and it can fail. A question may share many words with a section that doesn't answer it, and the sweep below is the test.

The guard can be measured without running the model again. Retrieval is deterministic: the same question gives the same top sections and the same scores each time. So the top score for every item can be computed from the index, and the guard becomes a rule applied to replies that have already been recorded. Its grid and rule are constants in `lib/guard.py`. The guard only turns answers into declines. It can never undo a decline, so it can lower false answers and can only raise false refusals.

The guard's score must come from the index that made the replies it judges, and the unfiltered replies were made from sections of every version. So the guard reads the `unfiltered` recording and takes the top BM25 score over the all-version index. A score from the asked version's index would be measured on sections the unfiltered model never saw. The guard is tested on the unfiltered variant because the baseline answers no unanswerable dev item, so a guard has nothing to remove there, while the unfiltered variant answers two. This chapter makes no claim about a guard on the baseline.

The grid runs from 0.00 to 10.00 in steps of a quarter, above every dev score. For each threshold the rule adds the two rates, each over its own denominator, and picks the threshold with the smallest sum. That equal weighting is a choice, and [the too-many-and-too-few section](#too-many-and-too-few) returns to it. A tie goes to the lower threshold, because it changes fewer replies. The rule is stated in `lib/guard.py` and picks on dev. The held-out split is not untouched, since the first table in [what the baseline does](#what-the-baseline-does) already shows its unguarded rates. The guard's held-out run came once, at the frozen threshold.

The sweep runs in tune-guard-dev, which prints both rates at every step of the grid and then the rule's pick. It opens with the check that matters most: the recorded sections of each item must match what the index returns, or the scores would describe a retrieval that never made the replies.

Example `tune-guard-dev` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Sweeps the decline guard's threshold over dev replies of the unfiltered recording, printing both rates at each
threshold, and the threshold the dev rule picks. The line for 0.00 is the recorded replies with no guard, since no top
score is below zero; the rule's pick rests on the rate lines below it. Only dev is read here."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.guard import GRID, GUARD_RECORDING, RULE, contexts_match, denominators, errors, pick_threshold, rows
from lib.stats import pct

dev = rows("dev")
answerable, unanswerable = denominators(dev)
print(f"recording: {GUARD_RECORDING} (all versions, dev items)")
print(f"answerable dev items: {answerable}")
print(f"unanswerable dev items: {unanswerable}")

mismatches = contexts_match("dev")
print(f"dev items whose recorded sections differ from the retrieval: {mismatches}")
assert mismatches == 0, "the scores come from a retrieval that did not make the recorded replies"
highest = max(row.score for row in dev)
print(f"highest top score on dev: {highest:.3f}")
assert highest < GRID[-1], "the grid must reach above every dev score"

print(f"rule: {RULE}")
for threshold in GRID:
    refusals, answers = errors(dev, threshold)
    print(f"threshold {threshold:.2f}, false refusal rate: {pct(len(refusals), answerable)}")
    print(f"threshold {threshold:.2f}, false answer rate: {pct(len(answers), unanswerable)}")

refusals, answers = errors(dev, 0.0)
print(f"recorded false refusals (no guard): {', '.join(refusals) or 'none'}")
print(f"recorded false answers (no guard): {', '.join(answers) or 'none'}")
scores = {row.item_id: row.score for row in dev}
for item_id in answers:  # the top score of each recorded false answer, so a reader can see where it falls on the grid
    print(f"top score of recorded false answer {item_id}: {scores[item_id]:.3f}")

chosen = pick_threshold(dev)
assert chosen == 0.0, f"the dev rule now picks {chosen:.2f}, not 0.00"
refusals, answers = errors(dev, chosen)
print(f"threshold chosen by the dev rule: {chosen:.2f}")
print(f"at the chosen threshold, false refusal rate: {pct(len(refusals), answerable)}")
print(f"at the chosen threshold, false answer rate: {pct(len(answers), unanswerable)}")
```

Output:

```text
recording: assistant-unfiltered (all versions, dev items)
answerable dev items: 40
unanswerable dev items: 13
dev items whose recorded sections differ from the retrieval: 0
highest top score on dev: 9.949
rule: pick the threshold in GRID whose false refusal rate plus false answer rate is smallest, each rate over its own denominator; when two thresholds tie, take the lower one
threshold 0.00, false refusal rate: 12.5% (5 of 40)
threshold 0.00, false answer rate: 15.4% (2 of 13)
threshold 0.25, false refusal rate: 12.5% (5 of 40)
threshold 0.25, false answer rate: 15.4% (2 of 13)
threshold 0.50, false refusal rate: 12.5% (5 of 40)
threshold 0.50, false answer rate: 15.4% (2 of 13)
threshold 0.75, false refusal rate: 12.5% (5 of 40)
threshold 0.75, false answer rate: 15.4% (2 of 13)
threshold 1.00, false refusal rate: 12.5% (5 of 40)
threshold 1.00, false answer rate: 15.4% (2 of 13)
threshold 1.25, false refusal rate: 12.5% (5 of 40)
threshold 1.25, false answer rate: 15.4% (2 of 13)
threshold 1.50, false refusal rate: 12.5% (5 of 40)
threshold 1.50, false answer rate: 15.4% (2 of 13)
threshold 1.75, false refusal rate: 12.5% (5 of 40)
threshold 1.75, false answer rate: 15.4% (2 of 13)
threshold 2.00, false refusal rate: 12.5% (5 of 40)
threshold 2.00, false answer rate: 15.4% (2 of 13)
threshold 2.25, false refusal rate: 15.0% (6 of 40)
threshold 2.25, false answer rate: 15.4% (2 of 13)
threshold 2.50, false refusal rate: 17.5% (7 of 40)
threshold 2.50, false answer rate: 15.4% (2 of 13)
threshold 2.75, false refusal rate: 17.5% (7 of 40)
threshold 2.75, false answer rate: 15.4% (2 of 13)
threshold 3.00, false refusal rate: 20.0% (8 of 40)
threshold 3.00, false answer rate: 15.4% (2 of 13)
threshold 3.25, false refusal rate: 30.0% (12 of 40)
threshold 3.25, false answer rate: 15.4% (2 of 13)
threshold 3.50, false refusal rate: 32.5% (13 of 40)
threshold 3.50, false answer rate: 15.4% (2 of 13)
threshold 3.75, false refusal rate: 37.5% (15 of 40)
threshold 3.75, false answer rate: 15.4% (2 of 13)
threshold 4.00, false refusal rate: 40.0% (16 of 40)
threshold 4.00, false answer rate: 7.7% (1 of 13)
threshold 4.25, false refusal rate: 45.0% (18 of 40)
threshold 4.25, false answer rate: 7.7% (1 of 13)
threshold 4.50, false refusal rate: 55.0% (22 of 40)
threshold 4.50, false answer rate: 7.7% (1 of 13)
threshold 4.75, false refusal rate: 60.0% (24 of 40)
threshold 4.75, false answer rate: 0.0% (0 of 13)
threshold 5.00, false refusal rate: 62.5% (25 of 40)
threshold 5.00, false answer rate: 0.0% (0 of 13)
threshold 5.25, false refusal rate: 70.0% (28 of 40)
threshold 5.25, false answer rate: 0.0% (0 of 13)
threshold 5.50, false refusal rate: 77.5% (31 of 40)
threshold 5.50, false answer rate: 0.0% (0 of 13)
threshold 5.75, false refusal rate: 77.5% (31 of 40)
threshold 5.75, false answer rate: 0.0% (0 of 13)
threshold 6.00, false refusal rate: 80.0% (32 of 40)
threshold 6.00, false answer rate: 0.0% (0 of 13)
threshold 6.25, false refusal rate: 82.5% (33 of 40)
threshold 6.25, false answer rate: 0.0% (0 of 13)
threshold 6.50, false refusal rate: 87.5% (35 of 40)
threshold 6.50, false answer rate: 0.0% (0 of 13)
threshold 6.75, false refusal rate: 87.5% (35 of 40)
threshold 6.75, false answer rate: 0.0% (0 of 13)
threshold 7.00, false refusal rate: 92.5% (37 of 40)
threshold 7.00, false answer rate: 0.0% (0 of 13)
threshold 7.25, false refusal rate: 92.5% (37 of 40)
threshold 7.25, false answer rate: 0.0% (0 of 13)
threshold 7.50, false refusal rate: 92.5% (37 of 40)
threshold 7.50, false answer rate: 0.0% (0 of 13)
threshold 7.75, false refusal rate: 92.5% (37 of 40)
threshold 7.75, false answer rate: 0.0% (0 of 13)
threshold 8.00, false refusal rate: 92.5% (37 of 40)
threshold 8.00, false answer rate: 0.0% (0 of 13)
threshold 8.25, false refusal rate: 95.0% (38 of 40)
threshold 8.25, false answer rate: 0.0% (0 of 13)
threshold 8.50, false refusal rate: 95.0% (38 of 40)
threshold 8.50, false answer rate: 0.0% (0 of 13)
threshold 8.75, false refusal rate: 95.0% (38 of 40)
threshold 8.75, false answer rate: 0.0% (0 of 13)
threshold 9.00, false refusal rate: 97.5% (39 of 40)
threshold 9.00, false answer rate: 0.0% (0 of 13)
threshold 9.25, false refusal rate: 97.5% (39 of 40)
threshold 9.25, false answer rate: 0.0% (0 of 13)
threshold 9.50, false refusal rate: 97.5% (39 of 40)
threshold 9.50, false answer rate: 0.0% (0 of 13)
threshold 9.75, false refusal rate: 97.5% (39 of 40)
threshold 9.75, false answer rate: 0.0% (0 of 13)
threshold 10.00, false refusal rate: 100.0% (40 of 40)
threshold 10.00, false answer rate: 0.0% (0 of 13)
recorded false refusals (no guard): q-0005, q-0052, q-0058, q-0060, q-0079
recorded false answers (no guard): q-0018, q-0039
top score of recorded false answer q-0018: 3.762
top score of recorded false answer q-0039: 4.592
threshold chosen by the dev rule: 0.00
at the chosen threshold, false refusal rate: 12.5% (5 of 40)
at the chosen threshold, false answer rate: 15.4% (2 of 13)
```

The check passes: zero dev items differ, and the example asserts that count is zero. The highest top score on dev is 9.949, which sits below the top of the grid.

Start at the bottom of the sweep. At 0.00 no score falls below the threshold, so the line shows the recorded replies with no guard. The false refusal rate there is 12.5% (5 of 40). The false answer rate is 15.4% (2 of 13). Every line up to 2.00 reads the same. At 2.25 the false refusals begin to rise, to 15.0% (6 of 40).

The false answers don't fall until 4.00, where the false answer rate is 7.7% (1 of 13). At 4.75 the false answer rate is 0.0% (0 of 13), so no unanswerable dev item is answered. The false refusal rate at 4.75 is 60.0% (24 of 40).

No line of the sweep lowers both rates below the no-guard line. Each threshold that lowers the false answers raises the false refusals, and each threshold that leaves the refusals alone leaves the answers alone. On dev, the guard makes a trade and no improvement.

The sweep also shows the limit of the score. The top score of `q-0018` is 3.762, and the top score of `q-0039` is 4.592. A threshold catches `q-0018` only from 4.00 up, where the false refusal rate is already 40.0% (16 of 40). It catches `q-0039` only from 4.75 up. No cut catches both answered items with fewer than half of the answerable items declined. The likely cause is the score itself: it counts shared words, and a section about the same command can share many words with a question that section doesn't answer. That is a reading of the sweep, not a measured fact. Testing it would mean scoring the items by hand, which this chapter does not do.

The rule's pick is 0.00. At that threshold the false refusal rate is 12.5% (5 of 40). The false answer rate is 15.4% (2 of 13). Both are the same as the recorded replies with no guard. The pick is a tie-break, not a finding. The lines from 0.00 to 2.00 have the same sum, and the rule takes the lowest of them, so the guard the rule chooses is no guard.

**Working rule:** Build a guard's scores from the same index that made the replies, and check that match before you read any rate. Then read the whole sweep with both rates on every line. A pick that ties with doing nothing says the rule found no reason to change a reply, and that is an answer worth keeping.

## Too many refusals and too few {#too-many-and-too-few}

The sweep shows the trade-off a team would face if it added the guard, and it shows what each choice of measure would report. Suppose a team counted only false answers. The lowest threshold that stops every false answer on dev is 4.75, so that team would pick 4.75 and report that the assistant no longer answers questions it can't support. The 4.75 line of [the sweep](#a-guard) shows the price that report hides. At 4.75 the false refusal rate is 60.0% (24 of 40). Nothing in the report would say so, because the team never counted the other error.

In production this is the failure to watch for. A user who asks a question the docs answer gets the decline sentence, and the reply gives no sign that the sentence was wrong. The only place the cost appears is the false refusal count. A team that doesn't count it reads the zero on false answers and ships.

The rule weights both errors equally, and that is a choice that belongs to the team. A team might judge a false answer as far costlier than a false refusal, for example when a confident wrong answer sends a user to the wrong fix. That team would get a different pick on dev once its weight is large enough to move the sum, and the sweep shows the direction: a higher weight on false answers pushes the pick toward thresholds that decline more. Such a team should write its weights down before anyone reads the sweep, and then give them to the rule. This chapter does not re-run the rule with other weights, and it does not reopen the held-out split to do it. Choosing a weight after seeing the held-out numbers would be a second round of tuning on that split.

The threshold also belongs to the assistant that produced the replies. The scores depend on the index, which depends on how sections are split and which versions are in it. Change any of those, and the scores move, so the threshold has to be picked again on dev, and the first check in the example has to pass again before anyone reads a rate.

**Working rule:** Before you tune a threshold, write the cost of each error in the team's own terms, and give the rule those weights before anyone reads the sweep.

## What a reader should accept {#what-to-accept}

The held-out split was run once, with the threshold frozen at 0.00. The run is guard-held-out, and it first asserts that the dev rule (described in [the guard](#a-guard)) still picks that value. A change to the dev data or to the rule therefore stops the run before any held-out number prints.

Example `guard-held-out` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Applies the decline guard, with its threshold frozen from dev, to the held-out items of the unfiltered recording, and
prints both rates with 95% bootstrap intervals over items. The threshold is a constant chosen by tune-guard-dev's rule
on dev; the assert checks that the rule still picks it, so a change to the dev data or the rule fails here."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.guard import denominators, errors, guarded, pick_threshold, rows
from lib.stats import bootstrap_mean, interval_pct, pct

# Picked on dev by tune-guard-dev (the rule in lib/guard.py), then frozen. Held-out is not used to change it.
FROZEN_THRESHOLD = 0.0

dev_pick = pick_threshold(rows("dev"))  # reads dev only
assert dev_pick == FROZEN_THRESHOLD, f"the dev rule now picks {dev_pick:.2f}, not the frozen {FROZEN_THRESHOLD:.2f}"

held = rows("held-out")
answerable, unanswerable = denominators(held)
refusals, answers = errors(held, FROZEN_THRESHOLD)
print(f"threshold (frozen from dev): {FROZEN_THRESHOLD:.2f}")
print(f"held-out answerable items: {answerable}")
print(f"held-out unanswerable items: {unanswerable}")
changed = [row.item_id for row in held if guarded(row, FROZEN_THRESHOLD) != row.declined]
print(f"held-out replies the guard changed to declines: {len(changed)} of {len(held)}")

# One 0/1 flag per item: the bootstrap over these flags is the bootstrap of the rate, over items.
refused_flags = [float(guarded(row, FROZEN_THRESHOLD)) for row in held if row.kind == "answerable"]
answered_flags = [float(not guarded(row, FROZEN_THRESHOLD)) for row in held if row.kind == "unanswerable"]
assert len(refused_flags) == answerable and len(answered_flags) == unanswerable

refusal_lo, refusal_hi = bootstrap_mean(refused_flags, level=0.95)  # 2,000 resamples, seeded (lib/stats.py)
answer_lo, answer_hi = bootstrap_mean(answered_flags, level=0.95)
print(f"false refusal rate: {pct(len(refusals), answerable)} interval {interval_pct(refusal_lo, refusal_hi)}")
print(f"false refusals: {', '.join(refusals) or 'none'}")
print(f"false answer rate: {pct(len(answers), unanswerable)} interval {interval_pct(answer_lo, answer_hi)}")
print(f"false answers: {', '.join(answers) or 'none'}")
```

Output:

```text
threshold (frozen from dev): 0.00
held-out answerable items: 20
held-out unanswerable items: 7
held-out replies the guard changed to declines: 0 of 27
false refusal rate: 45.0% (9 of 20) interval [25.0%, 70.0%]
false refusals: q-0101, q-0102, q-0103, q-0105, q-0106, q-0111, q-0115, q-0116, q-0117
false answer rate: 0.0% (0 of 7) interval [0.0%, 0.0%]
false answers: none
```

The frozen guard changes 0 of 27 held-out replies, so the held-out output is the recorded output with no guard. On the held-out split the false refusal rate is 45.0% (9 of 20) interval [25.0%, 70.0%]. The false answer rate is 0.0% (0 of 7) interval [0.0%, 0.0%].

The false refusal rate is the same as the unfiltered variant's in the first table, and the held-out run agrees with dev on the main point: the guard does nothing at the threshold the rule chose. A team that looked only at the false answer rate on held-out would see zero and conclude that the assistant doesn't answer what it can't support. The refusals are where the error sits, and no threshold can remove them, because the guard only adds declines.

The held-out false answer rate is 0.0% (0 of 7) interval [0.0%, 0.0%].

Its printed interval has no width. The interval resamples the seven held-out unanswerable items. None of them is answered wrongly, so every resample has no errors, and the bootstrap cannot vary. [Faithfulness](grounded-answers#faithfulness) gives the reasoning for this zero-events case, and this example prints no exact bound, so the interval is not read as a bound. It cannot tell a false answer rate of zero from a small rate that these seven items happened to miss. Seven items show only that none of them was answered wrongly.

The dev split is no better. On dev, the unfiltered replies give 15.4% (2 of 13) with no guard. At 4.00 they give 7.7% (1 of 13), one item fewer.

The false refusal interval is wide for a different reason: twenty items leave a band of rates that the data can't tell apart. The held-out figure is 45.0% (9 of 20) interval [25.0%, 70.0%]. [Intervals and sample size](intervals-and-sample-size) explains the bootstrap over items that builds these intervals, and how many items a paired difference needs. Kohavi and colleagues make the same point for controlled experiments on the web. In their guide, [Controlled experiments on the web: survey and practical guide](https://www.exp-platform.com/Documents/GuideControlledExperiments.pdf), an underpowered test is named as a common mistake. A held-out split of seven items is underpowered for a rate this small, and the chapter says so rather than let the zero stand for certainty.

Taken together, the results say something specific. On the unfiltered variant, at the threshold the dev rule picks, the guard changes no reply, so this chapter does not add it. Both errors remain in the unfiltered replies on dev. On held-out, the false refusals are the cost, and nothing in this chapter reduces them. The result is a decision, and it is the one the measurement supports.

The chapter's decisions are these:

- Both decline rates go into every comparison, each over its own denominator.
- Version-dependent and ambiguous items are left out of both rates, and their declines are counted apart.
- A decline rule is picked on dev, with its weights written down first, the whole sweep read, and the pick frozen before held-out is run.
- Held-out runs once per rule. An interval over zero events is reported with that caveat, not as a finding of zero.
- The retrieval-score guard is not added to the unfiltered variant. Its dev pick changes no reply, and its held-out run changes none.

**Working rule:** A guard earns its place only when its held-out run changes replies in the direction its dev sweep promised. When it doesn't, keep the result, since it shows which errors the assistant has and that a threshold on the retrieval score does not fix them.

## Outside this book {#outside-this-book}

This chapter does not cover attacks. A user who tries to make the assistant say what it shouldn't is a security question with its own threats and defences, and a separate book is planned for it, *Securing LLM Applications and Agents*. The decline rates here measure honest questions about the docs, and nothing in them tells you how an assistant behaves under pressure from a user who is trying to break it.

Other researchers have measured the same trade on other models, and the claims are worth knowing without borrowing their numbers. XSTest is a test suite of safe prompts that look unsafe. Its authors built it because models refuse safe prompts that use wording similar to unsafe ones, or that touch a sensitive topic ([Röttger and colleagues, 2024](https://arxiv.org/abs/2308.01263)). OR-Bench, published at ICML 2025, measured many models and reports that the extra safety of safety-tuned models often came with more over-refusal ([Cui and colleagues, 2024](https://arxiv.org/abs/2405.20947)). A survey of abstention in language models gives the general name for a model's refusal to answer, and it organises abstention by the query, the model and human values ([Wen and colleagues, 2024](https://arxiv.org/abs/2407.18418)). A 2026 benchmark in healthcare reports over-refusal of benign prompts in that domain, on its own set of prompts ([Zhang and colleagues, 2026](https://arxiv.org/abs/2601.17642)).

These papers test other models on other sets, so their rates don't carry over to this assistant. What carries over is their shape, which is the claim this chapter tests on its own case. On this set, the guard can lower false answers only by declining more answerable items, which is a trade of false answers for false refusals. Prompt and retrieval changes can move both errors, and the unfiltered variant did. The only results this chapter reports are the book's own runs.

**Working rule:** Before adopting a decline rule because a paper reports that models over-refuse, measure the rule on your own assistant and your own items, with both kinds of error counted. Keep the paper's numbers out of your results, and cite the paper for the idea that prompted the measurement.

# Put the evaluation in CI, and compare variants on traffic

By the end of this chapter you can put the evaluation into a gate that blocks a change only when the change is measurably worse. You can also say what a live comparison of two variants can show, and what it cannot.

The sablefile assistant answers from the tool's docs, and a team changes its prompt, its retrieval or its model from week to week. The team wants to know before a change merges whether the assistant got worse. A gate that fails on any drop in the pass rate looks like the safe choice, and it is not. With 80 dev items, one item moves a pass rate by about 1.3 points, so a gate on the raw rates would block on a difference of one or two items. A team that is blocked on noise often learns to override the gate, and then the gate protects nothing.

[Chapter 1](measure-before-you-change) ran the whole evaluation for the first time: the test set, the checks, the judge and the replayed answers. This chapter keeps that run in place. It takes three parts in turn: a gate that respects the interval, the items whose verdicts change between samples, and the tests of the evaluation's own code. Then it maps the same parts onto an evaluation framework's words, and it ends with what a comparison of variants on live traffic can and cannot show.

## A gate that respects the interval {#a-gate-that-respects-the-interval}

A gate is a rule that stops a change from merging. For this book, the rule rests on one number: the paired difference in pass rate between a candidate and the baseline, over the same dev items. The comparison is paired, as [Chapter 8](intervals-and-sample-size#paired-comparisons) defines it. For each item, the candidate's verdict minus the baseline's verdict, averaged over the items, with the interval bootstrapped over items. Because both runs answer the same questions, the items on which both runs agree cancel out of the difference.

The gate has one rule. It fails only when the upper bound of the interval is below minus the threshold. The upper bound is the least negative reading the data still allows, so the gate fails when even the most optimistic reading of the data shows a drop bigger than the threshold. A naive gate would fail whenever the point estimate of the difference is below minus the threshold, and it would fail on noise. The price of the book's rule is that it passes some real drops the data cannot yet prove, which is why a pass is read with its width, as the next paragraphs explain.

The threshold is five points. The example's comment says it is a choice for this book, not a standard. With the dev set at this size, one item is about 1.3 points, so five points is four items net.

The first run is a large drop, as this judge measures it. The second model, Qwen3-0.6B, is smaller than the 4B baseline, and it answered the same dev items. The judge is Qwen3-4B-Instruct-2507, the baseline's own model, so the gap can't separate a weaker assistant from self-preference. [Self-preference](when-the-judge-stops-being-reliable#self-preference) explains why this book cannot test that. Read the decision last, since it follows from the interval.

Example `gate-real-drop` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Runs the gate on second-model against the baseline over the eighty dev items: both pass rates with their counts, the paired
difference, its interval, the threshold and the decision. The gate fails only when the upper bound of the interval is below
minus the threshold, so a drop that the interval cannot rule out below the threshold does not fail it."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.calibration import readable_verdicts
from lib.gate import FAIL, gate
from lib.judging import passed
from lib.prompts import REF_SHA256
from lib.stats import interval_points, pct
from lib.testset import load_items

BASELINE = "judge-answers-baseline"
CANDIDATE = "judge-answers-second-model"
# Five points is this gate's threshold, chosen for the book, not a standard. With eighty dev items one item moves a rate
# by 1.25 points, so five points is four items net: a smaller threshold would ask the gate to see drops the interval
# cannot resolve at this size.
THRESHOLD_POINTS = 5.0

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
baseline_found, baseline_unreadable = readable_verdicts(BASELINE, items, REF_SHA256)
candidate_found, candidate_unreadable = readable_verdicts(CANDIDATE, items, REF_SHA256)
for label, unreadable in (("baseline", baseline_unreadable), ("second-model", candidate_unreadable)):
    details = "".join(f' ({item_id}, "{first}")' for item_id, first in unreadable)
    print(f"unreadable verdicts, {label}: {len(unreadable)}{details}")
# An item the judge did not give a readable verdict on, under either run, is left out of both sides: a paired
# difference needs the same items under both runs.
excluded = {item_id for item_id, _ in baseline_unreadable + candidate_unreadable}
kept = [item_id for item_id in items if item_id not in excluded]
first = passed({item_id: baseline_found[item_id] for item_id in kept})  # 1.0 or 0.0 per item, in id order
second = passed({item_id: candidate_found[item_id] for item_id in kept})

result = gate(first, second, threshold_points=THRESHOLD_POINTS)
print(f"dev items compared (n): {result.n}")
print(f"baseline pass rate: {pct(int(first.sum()), result.n)}")
print(f"second-model pass rate: {pct(int(second.sum()), result.n)}")
print(f"difference, second-model minus baseline: {100 * result.difference:+.1f} points")
print(f"{100 * result.level:.0f}% interval of the difference: {interval_points(result.lo, result.hi)}")
print(f"upper bound of the interval: {100 * result.hi:+.1f} points")
print(f"threshold: {THRESHOLD_POINTS:.1f} points")
if result.decision == FAIL:
    print(f"decision: fail, since the upper bound {100 * result.hi:+.1f} is below {-THRESHOLD_POINTS:+.1f}")
else:
    print(f"decision: pass, since the upper bound {100 * result.hi:+.1f} is not below {-THRESHOLD_POINTS:+.1f}")
```

Output:

```text
unreadable verdicts, baseline: 1 (q-0043, "VERDICT: PARTIAL")
unreadable verdicts, second-model: 0
dev items compared (n): 79
baseline pass rate: 78.5% (62 of 79)
second-model pass rate: 40.5% (32 of 79)
difference, second-model minus baseline: -38.0 points
95% interval of the difference: [-48.1, -26.6] points
upper bound of the interval: -26.6 points
threshold: 5.0 points
decision: fail, since the upper bound -26.6 is below -5.0
```

The gate fails, and the reason is in the interval, not in the point estimates. The baseline passes 78.5% (62 of 79). The second model passes 40.5% (32 of 79).

The comparison covers 79 items, not all of the dev items, because an item whose baseline verdict was unreadable is left out of both runs. A paired difference needs the same items under both. The difference, second model minus baseline, is -38.0 points, and its 95% interval is [-48.1, -26.6] points.

The upper bound is -26.6 points, and the threshold is 5.0 points, so even the upper bound sits well below the threshold. This drop is not one the interval can explain away, so the gate is built to fail on a drop of this kind, as this judge measures it. The decision reads: fail, since the upper bound -26.6 is below -5.0.

A smaller change gives a different result. Suppose the change drops the pass rate by less, or the set is smaller and the interval widens, so that the upper bound reaches above minus five points. The gate then passes. It has found no drop larger than the threshold, and it has not shown that the candidate is as good as the baseline. A pass with a wide interval is a weak pass, so the width of the interval is the reason to read it, not only the decision. How wide an interval gets at a given size, and how many items a small paired difference needs before it resolves, is what [Chapter 8](intervals-and-sample-size#how-many-questions) works through.

The gate's limit is its size. On 79 items, the upper bound has to fall well below minus the threshold before the gate fails, so small drops pass. [Chapter 8](intervals-and-sample-size#how-many-questions) estimates, at the disagreement it assumes, that 79 items are enough to show a ten-point difference with 80% power, and a five-point difference needs 316. The gate is a stricter test than that power calculation, so a real ten-point drop can still pass. The threshold and the set size have to be chosen together.

Some gates check more than one metric. Each extra metric is another chance for noise to look like a drop, so the gate takes a count of comparisons. With m metrics, each interval is taken at the level 1 minus 0.05 divided by m: 97.5% for two metrics and about 98.3% for three. Together the checks keep the chance of a false fail at or below five percent. The adjusted level is the one [Chapter 8](intervals-and-sample-size#many-comparisons) gives, and the gate uses the same one. The resample count stays at 2,000, and the seed stays fixed, so a rerun gives the same interval.

The gate runs on the dev items for every change, because a gate has to run each time and the team may have tuned on those items. Held-out is run once per release decision, against rules fixed before the run, as [Held-out, once](intervals-and-sample-size#held-out-once) explains. A fresh held-out set is needed after that.

To put the gate in CI, a job computes the gate on the candidate's recording and fails the build when the decision is fail. The gate reads only the per-item values, so the job needs no other state. A job that fails to run counts as a failed gate, because a skipped gate would look like a pass. The gate's own tests check the rule above. They run with the rest of the evaluation's tests in [the evaluation's own tests](#the-evaluations-own-tests).

**Working rule:** Report every paired difference with its interval and its n, and let the decision follow from the upper bound. Don't call a change better or worse from a point estimate, and don't read a pass with a wide interval as proof that the candidate matches the baseline.

## Flag flaky items, never edit them {#flaky-cases}

An item [flips](#variance-within-one-item) when its sampled replies get different verdicts. Each dev item has five verdicts from the variance set, and an item is flaky when they are not all the same. The variance set and the flip rate, the share of items that flip, are introduced in [Variance within one item](intervals-and-sample-size#variance-within-one-item). A team watches the flip rate to see how far the verdicts move when the assistant's replies do.

Each flagged item comes with its PASS count out of five, in id order, and the rule it is held to is printed beside the list.

Example `flaky-items` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""The dev items flagged as flaky: the judge's verdict differs across their five variance samples. Each flagged item prints its
PASS count out of five, sorted by id, and the count matches the flip rate in flip-rate.py, since both read the same per-item
counts. A flagged item is kept as it is, so the rule printed last is the whole policy: flagged, never edited or deleted."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from lib.testset import load_items
from lib.variance import VARIANCE_SAMPLES, item_pass_counts

VARIANCE = "judge-answers-variance"
SAMPLES = len(VARIANCE_SAMPLES)

items = [item.id for item in load_items(split="dev")]  # eighty dev items, sorted by id
counts = item_pass_counts(VARIANCE, items)
for row in counts:
    if not row.complete:  # an unreadable sample is not a FAIL: the item is named here and not flagged
        samples = ", ".join(str(sample) for sample in row.unreadable)
        print(f"left out, unreadable verdict, {row.item_id}: samples {samples}")
flagged = [row for row in counts if row.flaky]
for row in flagged:
    print(f"{row.item_id}: {row.passes} of {SAMPLES} PASS")
print(f"flagged items: {len(flagged)}")
print(f"dev items compared (n): {sum(1 for row in counts if row.complete)}")
print("rule: flagged, never edited or deleted")
```

Output:

```text
left out, unreadable verdict, q-0043: samples 1, 4
q-0047: 4 of 5 PASS
q-0058: 1 of 5 PASS
flagged items: 2
dev items compared (n): 79
rule: flagged, never edited or deleted
```

Few items flip on this set. The flagged items come to 2 of the 79 dev items compared. Both flagged items are worth reading.

On q-0047, the verdicts across its five replies are 4 of 5 PASS, so most of its replies passed. On q-0058, the verdicts are 1 of 5 PASS, so most of its replies failed.

One item is set aside rather than used. The item q-0043 is left out, since its verdict was unreadable in samples 1, 4. An unreadable sample is not a fail. It is a sample the example cannot read, so the item is neither flagged nor counted on that basis.

The rule is that a flagged item stays flagged, never edited or deleted. It stays in the set with the verdicts the judge gave. Editing it until the judge agrees more often would hide the very signal the flag exists to show. Deleting it would shrink the set and make the pass rate look steadier than it is. The flag does its work when someone reads it. A flagged item with one PASS in five is a question about the item, the rubric, the judge or the assistant's replies, and it should be looked at before the next change is judged on it.

The count is small, and it describes this assistant, this judge and these dev items. It shows that few items flip in this set. It does not show that other assistants, sampling settings or judges flip as rarely. A team that changes any of them should run the variance set again before it trusts a low flip rate.

A flagged item can also move a paired difference from one run to the next. When a gate fails on a difference that depends on a flagged item, read the flag before the change.

**Working rule:** Flag an item whose verdicts differ across its samples, print its count beside the flag, and leave the item as it is. Report the flip rate with its denominator, and say that a low rate holds for this assistant, this judge and this set only.

## The evaluation's own tests {#the-evaluations-own-tests}

The evaluation is code, and its code can be wrong in ways that look like a result. A bug in the bootstrap, the verdict reader or the paired difference moves a number in the same way a change in the assistant would. So the evaluation has tests of its own, in a tests folder inside the book's library, and each module that the book adds brings its tests with it. The tests check the scoring code. They do not check the assistant. A passing test says that the code counts what it says it counts. It says nothing about whether the assistant answers well.

Those tests run in one process, and the outcomes are counted module by module. The example then prints pytest's exit code, from 9.2.0. The [exit-code reference](https://raw.githubusercontent.com/pytest-dev/pytest/9.2.0/doc/en/reference/exit-codes.rst) lists 0 as every test collected and passed, and 5 as no tests collected. So a run that tests nothing shows up as 5, not as a silent pass.

Example `own-tests-run` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pydantic 2.14.3 · pytest 9.2.0 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""Runs the book's own tests, lib/tests, in this process and prints how many passed and failed in each module, then pytest's
exit code. Exit code 0 means every test was collected and passed. Exit code 5 means nothing was collected, so the example
prints the code it got rather than a silent pass. No paths and no timings are printed: they vary from run to run."""
import lib  # first: fixes the thread counts and seeds before anything else loads

from collections import Counter
from pathlib import Path

import pytest

TESTS = Path(lib.__file__).resolve().parent / "tests"  # read here, never printed


class Outcomes:
    """Counts each test's outcome by module. A failed setup or teardown counts as a failure of its test."""

    def __init__(self) -> None:
        self.passed = Counter()
        self.failed = Counter()

    def pytest_runtest_logreport(self, report) -> None:
        # The node id starts with the file's path; keep only the module's name, so no path is printed.
        module = Path(report.nodeid.split("::")[0]).stem
        if report.when == "call" and report.passed:
            self.passed[module] += 1
        elif report.failed:
            self.failed[module] += 1


collector = Outcomes()
# -p no:terminal turns off pytest's own printing, which carries timings. That also removes the -q option, which belongs to
# the terminal reporter, so it is not passed. -p no:cacheprovider keeps .pytest_cache out of the working copy.
exit_code = pytest.main(
    [str(TESTS), "-p", "no:terminal", "-p", "no:cacheprovider"],
    plugins=[collector],
)

modules = sorted(path.stem for path in TESTS.glob("test_*.py"))  # every module, even one with no failures
print("tests by module (passed, failed):")
for module in modules:
    print(f"  {module}: {collector.passed[module]} passed, {collector.failed[module]} failed")
print(f"total: {sum(collector.passed.values())} passed, {sum(collector.failed.values())} failed")
print(f"pytest exit code: {int(exit_code)}")
```

Output:

```text
tests by module (passed, failed):
  test_bench: 11 passed, 0 failed
  test_bias: 14 passed, 0 failed
  test_both_orders: 14 passed, 0 failed
  test_calibration: 9 passed, 0 failed
  test_calibration_bootstrap: 8 passed, 0 failed
  test_contract: 5 passed, 0 failed
  test_gate: 11 passed, 0 failed
  test_guard: 4 passed, 0 failed
  test_judging: 3 passed, 0 failed
  test_stats: 10 passed, 0 failed
  test_support_verdict: 5 passed, 0 failed
  test_testset: 5 passed, 0 failed
  test_variance: 15 passed, 0 failed
total: 114 passed, 0 failed
pytest exit code: 0
```

The table gives the outcome for each module, with what the module tests beside it.

| Module | What it tests | Result |
| --- | --- | --- |
| test_bench | The benchmark loaders and scorers | 11 passed, 0 failed |
| test_bias | The length bands, unreadable verdicts, the flip direction and the band gap | 14 passed, 0 failed |
| test_both_orders | The pairwise reader and both orders: the mapping back to the replies, and the four kinds | 14 passed, 0 failed |
| test_calibration | Judge-against-human counts, agreement, kappa, TPR, TNR and the corrected pass rate | 9 passed, 0 failed |
| test_calibration_bootstrap | The crossover point and the bootstrap interval of the corrected pass rate | 8 passed, 0 failed |
| test_contract | The reply and verdict formats that the judge and the assistant must keep | 5 passed, 0 failed |
| test_gate | The gate's rule: a fail only when the upper bound is below minus the threshold | 11 passed, 0 failed |
| test_guard | The decline guard's rule and its threshold, on hand-made rows | 4 passed, 0 failed |
| test_judging | Reading judge verdicts, and the paired difference type | 3 passed, 0 failed |
| test_stats | The printing helpers and the seeded bootstrap | 10 passed, 0 failed |
| test_support_verdict | The faithfulness verdict reader: SUPPORTED or UNSUPPORTED on the first line | 5 passed, 0 failed |
| test_testset | Loading the test set, and the rules that look across items | 5 passed, 0 failed |
| test_variance | Per-item verdict counts over the variance set | 15 passed, 0 failed |

The totals come last: 114 passed, 0 failed, and the exit code is 0, the code for a run in which every collected test passed.

Read the table as a check on the instruments, not on the assistant. Run these tests first in the gate job, so that a broken scorer stops the job before it reads a pass rate. Nothing that varies between runs is printed: no timings and no paths. A reader can rerun the code and compare the output line for line.

**Working rule:** Run the evaluation's own tests before any run that reads a number, and treat exit code 5 as a failure, not a pass. A failing test in the scoring code means the numbers are not yet trustworthy, whatever they say.

## Running it as a framework task {#running-it-as-a-framework-task}

Teams that already use an evaluation framework can map the parts of this book onto its words, and it helps to see the mapping once. In Inspect AI 0.3.291, dated 17 November 2026 in [its changelog](https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/0.3.291/CHANGELOG.md), a task is the recipe for an evaluation, made minimally of a dataset, a solver and a scorer, as [the framework's documentation at that version](https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/0.3.291/docs/tasks.qmd) describes. On that reading, the test set is the dataset, with one sample per item. The assistant's path of retrieving and then answering is the solver, and the checks and the judge make up the scorer. The book does not run that mapping, and it does not claim to. Instead, the book keeps one example environment: one set of files that every chapter's examples import, pinned to the same versions, so the replay model, the checks and the gate are the same code in every chapter. This chapter describes where each part would go, so that a reader on a framework can move the parts over while the book's examples stay as they are.

**Working rule:** When you map an evaluation onto a framework, name the framework's version and the date of its documentation. Keep the test set, the checks and the replay model as the files the book's examples run, and treat the map as a reading of the documentation until someone runs it.

## Variants on traffic {#variants-on-traffic}

Offline, the test set and the replayed answers can say which variant is better on the items the team wrote. Once the assistant is live, a team may want to compare two variants on real questions, such as two prompts or two models, each sent to a share of users. That is a comparison, and it is only as good as the way users were assigned to the variants. If the people who see one variant are the ones who ask harder questions, the difference reflects them and not the variant. Random assignment, and an interval read the way the rest of this chapter reads one, are what turn a live comparison into a test. Kohavi and colleagues' guide, which [chapter 10](declining-when-the-docs-dont-say#what-to-accept) cites, recommends deciding the power and the effect to detect, and estimating the variability, before the test starts. Without that, a live comparison is a list of who saw which variant.

The feedback table in this section is not traffic. Its rows are generated. Each dev item's vote is drawn from a stated model with a fixed seed. An answer is right with probability 0.85. A user who sees a wrong answer gives a thumbs-down with probability 0.5 and a thumbs-up with 0.05. A user who sees a right answer gives a thumbs-down with 0.03 and a thumbs-up with 0.15. Most items get no vote at all. Whether each answer was right is drawn and then dropped, as it would be in a real log, so the table holds only the votes. It is a teaching device, and its figures are what the stated model produces. They say nothing about this assistant.

Example `feedback-table` (Python 3.12.15 · numpy 2.5.5 · pydantic 2.14.3 · typing-extensions 4.16.0), passing in Edition 2026.11.3:

```python
"""A generated table of thumbs-up and thumbs-down rows over the eighty dev items, with what such a table can show and what it
can't. The rows are generated from a stated model with a fixed seed, not taken from users, so the table is a teaching
device only.

The model: an answer is right with probability 0.85. A user who sees a wrong answer gives thumbs-down with probability 0.5 and
thumbs-up with 0.05; one who sees a right answer gives thumbs-down with 0.03 and thumbs-up with 0.15. Most items get no vote.
The table keeps the votes only: whether each answer was right is drawn and then dropped, as it would be in a real log."""
import lib  # first: fixes the thread counts and seeds before anything else loads

import numpy as np

from lib import SEED
from lib.stats import pct
from lib.testset import load_items

P_RIGHT = 0.85
P_DOWN_WRONG, P_UP_WRONG = 0.50, 0.05
P_DOWN_RIGHT, P_UP_RIGHT = 0.03, 0.15

rng = np.random.default_rng(SEED)  # one generator for the whole table, so the same seed gives the same rows
items = [item.id for item in load_items(split="dev")]
rows = []  # (item id, vote) for each item that got a vote
for item_id in items:
    right = rng.random() < P_RIGHT  # hidden: the table has no column for it
    draw = rng.random()
    down, up = (P_DOWN_RIGHT, P_UP_RIGHT) if right else (P_DOWN_WRONG, P_UP_WRONG)
    if draw < down:
        rows.append((item_id, "thumbs-down"))
    elif draw < down + up:
        rows.append((item_id, "thumbs-up"))

print("generated rows from a stated model, not user feedback (seed 0)")
print("item    vote")
for item_id, vote in rows:
    print(f"{item_id}  {vote}")

ups = sum(vote == "thumbs-up" for _, vote in rows)
downs = sum(vote == "thumbs-down" for _, vote in rows)
voted = len({item_id for item_id, _ in rows})
print(f"dev items: {len(items)}")
print(f"items with a vote: {voted}")
print(f"items nobody voted on: {len(items) - voted}")
print(f"thumbs-down among those who voted: {pct(downs, ups + downs)}")
print(f"thumbs-up among those who voted: {pct(ups, ups + downs)}")
print("what the table can show: the share of each kind of vote, among the votes cast, with its n.")
print("what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.")
```

Output:

```text
generated rows from a stated model, not user feedback (seed 0)
item    vote
q-0002  thumbs-down
q-0006  thumbs-down
q-0007  thumbs-down
q-0008  thumbs-up
q-0009  thumbs-up
q-0011  thumbs-up
q-0020  thumbs-down
q-0027  thumbs-down
q-0028  thumbs-up
q-0030  thumbs-up
q-0035  thumbs-up
q-0040  thumbs-down
q-0056  thumbs-down
q-0057  thumbs-down
q-0059  thumbs-down
q-0060  thumbs-up
q-0064  thumbs-down
q-0070  thumbs-down
q-0072  thumbs-up
q-0079  thumbs-down
q-0080  thumbs-down
dev items: 80
items with a vote: 21
items nobody voted on: 59
thumbs-down among those who voted: 61.9% (13 of 21)
thumbs-up among those who voted: 38.1% (8 of 21)
what the table can show: the share of each kind of vote, among the votes cast, with its n.
what it can't show: anything about the items nobody voted on; whether a thumbs-up answer was right.
```

Under the stated model, only 21 of the 80 dev items get a vote, and 59 get none. So the table says little about the other items.

A vote is not a correctness label, and this table shows why. Under the stated model, answers are right 85% of the time, yet most votes are thumbs-down, because a wrong answer draws a vote far more often. Among the votes cast, thumbs-down is 61.9% (13 of 21). Thumbs-up is 38.1% (8 of 21).

A table of votes can show the share of each kind of vote, among the votes cast, with the count beside it, and the example's last lines say exactly that. It cannot show whether a thumbs-up answer was right, since the table has no column for correctness. It says nothing about the items nobody voted on, and a vote comes from someone who chose to vote on what they chose to read.

Joachims and colleagues studied clicks on search results in 2005 and found them informative but biased, so a click is not an absolute judgment of relevance. They report that relative preferences derived from clicks are reasonably accurate on average, [in their paper on interpreting clickthrough data](https://www.sigir.org/wp-content/uploads/2017/06/p003.pdf). This book uses that finding as an analogy only, and it is the book's own reading. A thumbs-up is a vote, not a label. A right answer can still draw a thumbs-down, and the table has no way to tell which votes those are.

Two groups of users are compared differently from the paired items above. Each user sees one variant, so the two groups are separate, and the interval for their difference is a different calculation from the paired one this chapter uses. [Chapter 8](intervals-and-sample-size#paired-comparisons) covers the paired comparison this chapter uses. It does not cover an interval for two separate groups.

A vote table also cannot compare variants unless each vote is tied to the variant the user saw. This table has no variant column, so it cannot compare anything. A live comparison would need that column, the random assignment above, and an interval. If few items get a vote, as in this generated table, the interval would be wide, and most items would have no vote to compare.

Before a real vote table could be trusted, it would need three things. Each vote would need to be tied to the variant the user saw. The share of items that get a vote would need to be reported with its count, since a table of votes speaks only to the items that were voted on. And a sample of its votes would need checking against correctness, so that a thumbs-up or a thumbs-down can be read against what it is meant to measure.

The book's parts add up to an evaluation for a feature of your own. A labelled test set comes first ([chapter 3](build-the-test-set)), then checks in code ([chapter 4](checks-in-code)). A judge with a rubric and a measured error follows ([chapter 5](a-judge-with-a-rubric), [chapter 6](calibrate-the-judge)). A comparison reports its interval and n ([chapter 8](intervals-and-sample-size)), and the gate in CI reads all of it. [Chapter 7](when-the-judge-stops-being-reliable) says where the judge stops being reliable. [Chapter 9](grounded-answers) and [chapter 10](declining-when-the-docs-dont-say) apply the same counts to grounding and declining.

The book leaves three subjects to others. [*A Primer on Search Relevance*](https://rollingedition.com/primer-on-search-relevance/) covers ranking metrics. [*Agent Harnesses*](https://rollingedition.com/agent-harnesses/) covers tasks that take several steps. *Observability for LLM Applications* covers tracing one request through retrieval and the model, and counting what each request costs. It is planned, so it is named here and not linked.

**Working rule:** Don't compare variants from traffic without random assignment and the variant recorded beside each vote or reply. Read a vote table as a share of the votes cast, with its count, and never as a measure of correctness.
