---
title: "Small Open Models"
book: small-open-models
edition: 2026.11.9
verified: 2026-11-20
edition_code: ND0S-78AT
canonical: https://rollingedition.com/small-open-models
license: CC BY-NC 4.0
more_permissions: https://rollingedition.com/policies/reuse
attribution: "Small Open Models, Edition 2026.11.9, examples passed Nov 20, 2026. By Rolling Edition, https://rollingedition.com/small-open-models. Text CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), with added permissions (https://rollingedition.com/policies/reuse). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.11.9. Examples passed Nov 20, 2026. 36 of 36 examples pass. Written with AI. Verified by people.

To cite a section, name Edition 2026.11.9 and link to the section's permanent link: https://rollingedition.com/small-open-models/s/{section_id}

# Build a router for the bank's inbox that hands off when unsure

By the end of this chapter, you can build a router that returns a team, a confidence and a hand-off flag for one message, score it on a fixed sample by three measures, and see why an untrained small model, prompted as it stands, can't be the router's first version.

Customer messages to a bank land in one support queue, which this book calls the inbox. A router sends each message in the inbox to one of 77 teams. A router that picks a team for every message will sometimes pick the wrong one. A wrong team costs the customer time, and it costs support staff a step to pass the message on. This chapter builds a router that says when it is unsure.

A team is a label the router can return, such as `card_arrival`, and it is never a group of people. The people who read messages are the bank's support staff, and a message the router cannot send on goes to one of them.

The messages and their teams come from BANKING77, a public dataset of customer messages to a bank, published under [CC BY 4.0](https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/57ec275d8078af65b7731c2a98be812d844a6d6b/LICENSE). BANKING77 calls each label an intent. This book calls it a team from here on, and the labels are written as they appear in the dataset.

The router is the whole system: a model, a prompt, a threshold and the code that sends a message on. This chapter builds a rough router. A router is rough when its threshold is a placeholder and its model is a stand-in, and this one is a working router because it returns an answer for every message and can be scored on every output.
You need Python and one call to a hosted model's API, which you may already have written. This chapter does not ask you to know how a transformer works inside. Where an internal matters, it gets one sentence.

Two shared pieces come first. The first is a recording of a stand-in model's answers to the test messages, which [the section on asking a model](#ask-a-model-for-one-team) introduces. Most examples read those answers offline, and one runs a small model live, in [the section that shows its failure](#show-what-a-small-model-does-first). The second is `router_lib`, a shared module in the book's `examples` folder. It holds the team labels, the router prompt, the hand-off rule and the measures. Every example imports it, so a function means the same thing in every chapter.

## What the router returns {#what-the-router-returns}

For one message, the router returns three things. The team is one of the labels, written as the label itself. The confidence is a number from zero to one that says how sure the router is of that team. The hand-off flag is true when the confidence is too low to send the message on, and a person takes the message instead.

In this chapter, the confidence has a narrow meaning. It is the probability llama-cpp-python returns for the constrained answer, and the recording keeps it as a log-probability. In plain terms, it is how much probability the model gives the team it wrote. In formula form, confidence = exp(sum of log p(token)), the product of the probabilities of the answer's tokens. A token is a piece of text the model reads and writes. This is not a [softmax](#a-head-that-scores-the-teams) over the teams, which chapter 5 uses. It is a score, not a calibrated probability, and nothing in this chapter shows that a confidence of a given size matches how often the team is right. Chapter 5 checks calibration on the dev split, the training messages held back for choosing settings ([chapter 2](measure-the-routing-task#dev-test-and-fixed-samples)): it groups the messages into confidence bands and prints each band's count and accuracy.

| Output | What it is | What uses it |
| --- | --- | --- |
| Team | One label from the banking set | The destination the message is sent to, when the flag is false |
| Confidence | The probability llama-cpp-python returns for the constrained answer, as a score | The hand-off rule |
| Hand-off flag | True when the confidence is below the threshold | The person who takes the message |

The flag matters most, and the reason is operational. A router that always returns a team sends every message somewhere, including the messages it has no basis to route. A hand-off gives the message to a person who can route it, and a router that says when it is unsure makes that person's work visible. We recommend that every router return the flag in every answer, true or false, so that each measure in the sections below can be computed from the output alone.

Letting a router say when it is unsure has a name in the research literature: selective classification, or the reject option. [Geifman and El-Yaniv's 2017 preprint on deep networks](https://arxiv.org/abs/1705.08500) describes the trade. A classifier that declines some inputs covers fewer of them, and the techniques can potentially and significantly improve a deep network's predictions on the inputs it keeps, by giving up coverage. Their method also lets a user set a target level of risk. The hand-off rule in this book does not do that. It picks a threshold on held-out data, and it gives no guarantee about the risk that remains.

The measurements in this chapter return teams only to be scored. The one-message example, in [the hand-off section](#hand-off-the-unsure-messages), returns a team only when the flag is false.

**Working rule:** Return the flag with every team, and send a message to a person whenever the flag is true. Return the team to the caller only when the flag is false. Keep the flag even when removing it would make a router look more complete, because the hand-off measure counts it.

## Ask a model for one team {#ask-a-model-for-one-team}

The router's team comes from a model, and the model is steered by a prompt. The prompt is the part of the router that the bank changes most often, so we fix it in one place. `router_prompt` builds one prompt for one message. The prompt asks for one team and gives the model the name of every team, the message, and the word `Team:` at the end, for the model to complete. It holds the team names and the one message being routed, and nothing else, so no other message from the sample reaches the model with it.

Every live and recorded run uses the same prompt text, from `router_lib.router_prompt`, so the words are the same in every run. What the model reads is not always the same. The recording wraps those words in the stand-in's chat template, and the live run in [the section on a small model](#show-what-a-small-model-does-first) gives the small model the raw text, without a template. A changed prompt is a different recording, so its scores can't be compared with the old ones. A prompt is part of the router's version, and a reader who changes it should record the change, the same way the book records a change of model.

The bank is the only name this book uses for the organisation. The bank's own router is a hosted model that it calls for each message. We have no access to one, and this book never calls a hosted model. In its place we use the stand-in, Qwen3-4B-Instruct-2507, a larger open model, and every router in the book is measured against the stand-in's answers. A reader's own hosted model will answer some messages differently, and score differently for that reason.

We chose the stand-in because anyone can reproduce its answers. Its licence is Apache-2.0, as [the model card of the GGUF repository](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/raw/a06e946bb6b655725eafa393f4a9745d460374c9/README.md) declares for the file at the revision this book pins. The file is `Qwen3-4B-Instruct-2507-Q4_K_M.gguf`, a quantized build that the llama-cpp-python library runs on a CPU. Its recording manifest names the file's SHA-256, so a reader who downloads the file can check it before anything runs.

The stand-in answered every message of `test_1001`, a fixed sample of 1,001 test messages, once, and `test_40` is its first 40 messages. The recording used greedy decoding, which takes the most likely token at each step, and a fixed seed. It also used a grammar that allowed only the team names, so every recorded answer is one of the 77 labels. That grammar is why the recorded answers are always team names. Chapter 4 explains it, since it is the same idea that fixes the failure in [the section on a small model](#show-what-a-small-model-does-first).

A reader who has a model of their own can record its answers the same way. You write one function, `call_model`. It takes one message and returns the team and the log-probability of the answer, which is the sum of the answer's token log-probabilities, as the stand-in's recording stores it. The function `record_answers` calls it for every message in the sample, writes one line per message in the stand-in's format, and raises an error if the function returns a name that is not one of the teams. Everything else is the same for every model, so two recordings on the same sample can be compared line by line.

Example `record-your-own-answers` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Write a model's team answers in the stand-in's JSONL shape, then check record_answers against the stand-in's recording.
The line to read is the last one printed, which counts the lines that match the recording. The stand-in here replays
that recording, so no model runs."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402
import tempfile  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()


def record_answers(call_model, messages, path):
    """Write one JSONL line per (index, text); call_model(text) returns (team, logprob)."""
    with open(path, "w") as out:
        for index, text in messages:
            team, logprob = call_model(text)
            if team not in router_lib.LABELS:
                raise ValueError("the model returned a name that is not a team")
            label = router_lib.LABELS.index(team)  # stored as the label, the same encoding as the gold labels
            row = {"index": index, "label": label, "team": team, "logprob": logprob}
            out.write(json.dumps(row) + "\n")


RECORDING = router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl"
rows = router_lib.read_rows("test")  # each row is (text, gold label)
sample = router_lib.read_sample("test_40")
messages = [(i, rows[i][0]) for i in sample]  # element 0 of a row is its text, which the model reads
with open(RECORDING) as f:
    recording_lines = f.read().splitlines()
recorded = {json.loads(line)["index"]: json.loads(line) for line in recording_lines}  # keyed by message index
by_text = {rows[i][0]: (recorded[i]["team"], recorded[i]["logprob"]) for i in sample}  # keyed by text, not index


def standin(text):
    """Stand in for a model: return the recorded team and logprob for this message."""
    return by_text[text]


with tempfile.TemporaryDirectory() as folder:
    path = os.path.join(folder, "answers.jsonl")
    record_answers(standin, messages, path)
    with open(path) as f:
        written = f.read().splitlines()

matching = sum(line in recording_lines for line in written)  # a line matches when it equals a recorded line exactly
print(f"messages written: {len(written)}")
print(f"lines matching the recording: {matching}/{len(written)}")
```

Output:

```text
messages written: 40
lines matching the recording: 40/40
```

Before you trust your own recording, check `record_answers` on the stand-in's answers. Here the stand-in function replays the recording, so the check tests `record_answers` and not a model. Every line it writes matches the recording, all 40 of them. When you swap that function for a call to your own model, the same check shows whether your recording has the same shape as the stand-in's.

**Working rule:** Score every model on the same fixed sample, and record its answers with `record_answers` before you compare it with the stand-in. Say which sample each score is on, and set two scores side by side only when their samples match.

## Hand off the unsure messages {#hand-off-the-unsure-messages}

[The stand-in](#ask-a-model-for-one-team)'s answers give each message a confidence, and this section turns that confidence into the hand-off flag.

The hand-off rule is one comparison. `handoff(confidence, threshold)` returns true when the confidence is below the threshold, and false when it is equal to the threshold or above it. A message that is handed off goes to a person. A message that is not handed off goes to the team the router named. Every route in this book that returns a team to a caller uses this function, so no caller can receive a team without the decision having been made.

The rule is simple to read, and the work is in choosing the number. This chapter uses a placeholder threshold, 0.5, and every output that uses it says so. A placeholder threshold has no rule behind it: it is a number that lets the rule run.

Chapter 5 chooses the threshold on the dev split, by a rule stated there. It records the chosen value in the [adapter](#adapters-train-a-small-part-of-the-model)'s manifest, so a reader can see which threshold a result used and what it was chosen on. The test sample is never used to choose it. Until then, the placeholder is the only threshold this chapter has, and its results are for teaching the measures, not for judging the router.

The hand-off rate is the share of messages that go to a person, and it moves with the threshold. Raise the threshold and more messages go to people; lower it and more go straight to a team. The trade-off is the reason the choice is a measured one. Chapter 5 shows the trade-off on the dev split, at more than one setting, before it picks one.

Example `route-one-message` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Route the first message of test_40 from the stand-in's recorded answer, at the placeholder threshold.
The prose reads the printed team and the hand-off line. No model runs here, and the message's text is never printed."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402
import math  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()

THRESHOLD = 0.5  # Placeholder. Chapter 5 chooses the threshold on the dev split.
RECORDING = router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl"


def route(row):
    """Return (team, confidence, handed off) for one recorded answer. The team is None when handed off."""
    confidence = math.exp(row["logprob"])  # the recording stores a log-probability; exp gives a probability
    flag = router_lib.handoff(confidence, THRESHOLD)  # True below THRESHOLD, and for a NaN; equal to THRESHOLD is kept
    return (None if flag else row["team"]), confidence, flag


with open(RECORDING) as f:
    recorded = {r["index"]: r for r in map(json.loads, f)}  # keyed by message index, as the sample lists them

first = router_lib.read_sample("test_40")[0]  # the first of the sample's 40 message indexes
team, confidence, flag = route(recorded[first])
assert not flag  # the prose states this message is kept; a change that hands it off fails the run
print(f"team: {team or 'none (handed off)'}")
print(f"confidence: {confidence:.2f}")
print(f"handed off: {'yes' if flag else 'no'}")
```

Output:

```text
team: top_up_reverted
confidence: 1.00
handed off: no
```

Follow one message through the whole route, from the recorded answer to a printed team. The example takes the first message of `test_40`, reads the stand-in's recorded answer for it, and routes it through `handoff` at the placeholder threshold. It prints the team, the confidence rounded to two places and whether the message was handed off, and it never prints the message's text. This message's confidence is 1.00, which clears the placeholder, so the router keeps the message and returns its team, top_up_reverted. The script asserts that the message is not handed off. One message shows the shape of the output. It can't show how often the router is right.

**Working rule:** Choose the threshold on the dev split, never on the test sample, and report the hand-off rate beside accuracy every time. While the threshold is a placeholder, say so in each output, and keep its accuracy out of any claim about the router's result.

## Score the router on a fixed sample {#score-the-router-on-a-fixed-sample}

A router is scored on messages whose team is already known. This chapter uses `test_40`, a fixed sample of 40 test messages, so that every reader scores the same messages in the same order. [Chapter 2](measure-the-routing-task#dev-test-and-fixed-samples) explains how the samples were drawn, which split is used to choose settings, and why the test sample is kept out of that choice. The numbers below describe this sample only.

The router is measured by three numbers, and each one has its count beside it.

- **Accuracy** is the messages whose team equals the gold label, divided by all the messages in the sample. A message that is handed off counts in the denominator, and it is wrong if its team is wrong.
- **Hand-off rate** is the messages whose confidence is below the threshold, divided by all the messages.
- **Accuracy on messages not handed off** is the correct messages among those the router keeps, divided by the messages it keeps. It is reported as not defined when the router keeps none.

Each measure answers a different question. Accuracy asks how often the router is right overall. The hand-off rate asks how much of the work goes to people. The accuracy on kept messages asks whether the messages the router keeps can go straight to a team. Accuracy alone does not separate the two kinds of error. A wrong team that was handed off counts against accuracy, and so does a wrong team that reached a team unchecked. Only the third measure isolates the second kind, and that is the kind support staff feel directly.

Example `route-recorded-answers` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Score the stand-in's recorded answers on test_40, with a placeholder hand-off threshold.
The last printed line, accuracy on messages not handed off, is the one the prose reads against the others: it is the
accuracy of the messages a team receives. No model runs here, and the answers come from the recording."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402
import math  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()

THRESHOLD = 0.5  # Placeholder. Chapter 5 chooses the threshold on the dev split.
RECORDING = router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl"

answers = {}
with open(RECORDING) as f:
    for line in f:
        row = json.loads(line)
        answers[row["index"]] = row

rows = router_lib.read_rows("test")
sample = router_lib.read_sample("test_40")

golds = [rows[i][1] for i in sample]  # element 1 of a test row is its gold label; element 0 is its text
preds = [answers[i]["label"] for i in sample]  # the recorded label is the team's position in LABELS, as the gold is
confs = [math.exp(answers[i]["logprob"]) for i in sample]  # exp turns each recorded log-probability into a confidence

# metrics applies the hand-off rule at THRESHOLD; accuracy's denominator still includes handed-off messages
result = router_lib.metrics(preds, golds, confs, THRESHOLD)
n = result["n"]

print(f"messages: {n}")
print(f"placeholder threshold: {THRESHOLD}")
print(f"accuracy: {result['correct']}/{n} ({result['accuracy']:.1%})")
print(f"handed off: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")
if result["accuracy_not_handed_off"] is None:  # None when the router keeps no message, so the measure is undefined
    print("accuracy on messages not handed off: not defined (0 messages not handed off)")
else:
    m = result["not_handed_off"]
    print(
        f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} "
        f"({result['accuracy_not_handed_off']:.1%})"
    )
```

Output:

```text
messages: 40
placeholder threshold: 0.5
accuracy: 20/40 (50.0%)
handed off: 2/40 (5.0%)
accuracy on messages not handed off: 20/38 (52.6%)
```

Here are the three measures on the sample, at [the placeholder threshold](#hand-off-the-unsure-messages). The example reads the recorded answers, applies the threshold to their confidences, and prints each measure with its count. The line to watch is the accuracy on messages not handed off, because it is the accuracy of the messages a team actually receives.

Read together, the three measures say that the placeholder does not protect the teams. The router is right on 20/40 (50.0%) of the sample, which no team should accept on its own. The threshold hands off only 2/40 (5.0%) of the messages, so most of the sample reaches a team. Those kept messages are routed right at 20/38 (52.6%), only a little above the accuracy on the whole sample. A router that returns a team for every message is not yet one the bank can use.

A sample this small can show that a router fails, and it can show how the measures move. It cannot tell two routers apart with confidence. Chapter 2 sets out the larger samples, and chapter 5 measures its chosen threshold on the same kind of sample.

**Working rule:** Report accuracy, the hand-off rate and the accuracy on messages not handed off together, each with its count, on the same fixed sample, and compare two routers only when both were measured on that sample.

## Show what a small model does first {#show-what-a-small-model-does-first}

The obvious shortcut is to drop [the stand-in](#ask-a-model-for-one-team) and ask a small model directly. SmolLM2-135M-Instruct is the smallest model in the family that this book keeps, and it is the core model for the rest of the book. It runs on a CPU and loads offline from its pinned revision. If it answered with a team name, the router would need no recording at all, so the shortcut is worth testing before anything else is built on it.

The test asks the model the same question the router would ask. The model gets the router prompt for the first 20 messages of the sample, with no constraint on its answer and a cap of a few tokens. Make a guess first: how many of those answers will be team names, with nothing holding the model to the list? Then count them.

Example `unconstrained-first-try` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Ask the 135M model for a team with the raw router prompt and no constraint, on the first 20 messages of test_40.
Look at the last printed line, which counts the answers that are exactly a team name. The answers stay in this file
and are never returned to a caller."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"

tokenizer, model = router_lib.load_model(SMALL)
rows = router_lib.read_rows("test")
sample = router_lib.read_sample("test_40", n=20)  # the first 20, to keep the run short
teams = set(router_lib.LABELS)

answers = []
with torch.no_grad():
    for i in sample:
        # the raw prompt: no chat template and no grammar, so the model may write anything
        inputs = tokenizer(router_lib.router_prompt(rows[i][0]), return_tensors="pt")
        # at most 10 new tokens, and greedy: the most likely token at each step
        out = model.generate(
            **inputs, max_new_tokens=10, do_sample=False, pad_token_id=tokenizer.eos_token_id
        )
        # the output starts with the prompt, so skip its tokens and keep only the answer
        new_tokens = out[0, inputs["input_ids"].shape[1] :]
        answers.append(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

print(f"messages: {len(answers)}")
# an answer counts only when it is exactly a team name, with no extra words around it
print(f"answers that are a team name: {sum(a in teams for a in answers)}/{len(answers)}")
```

Output:

```text
messages: 20
answers that are a team name: 0/20
```

None of the model's answers is a team name, so the router has nothing to send to a team. The measurement keeps its answers and never returns them to a caller.

This failure does not show that a small model can never route. It shows that this model, asked this way, does not return a team. There are two different fixes, and we take them in order. Chapter 4 constrains the answer, so the model can only write a team name. Chapter 5 trains a head on the model, so each team gets a score from a single forward pass. This chapter does neither, which is why its router runs on the stand-in's recorded answers.

**Working rule:** Before a model routes anything, count its answers that are team names on a fixed sample, and compare the count with the sample's size. If the model does not return a team for every message, do not route with it as it stands. Constrain its answer to the teams, or train it for the task. A hand-off rule placed on an answer that is not a team protects a team that does not exist.

## Where a rough router breaks {#where-a-rough-router-breaks}

A rough router breaks in four ways in production, and each one leaves a sign that support staff can see.

The first is a team name that does not exist. A free-text model can write a team that is not on the list, or a sentence, and a router that sends that text on has sent the message to nobody. [The stand-in](#ask-a-model-for-one-team)'s recording cannot do this, because its grammar allows only the labels, but a reader's model can. The sign is an answer that is not a team name, and [the unconstrained test](#show-what-a-small-model-does-first) counts the answers that are team names. A router should treat such an answer as an error, count it, and send the message to a person. `record_answers` already raises an error for that case, and the router must do the same.

The second is drift. The stand-in's answers hold only for the file, the prompt and the sample they were recorded with. The recording's manifest names the file's SHA-256, the prompt's hash, the decoding and the sample, so a change to any one of them is a different recording. A reader who sees a score move with no change to the code should compare the manifest first, and then the prompt.

The third is a threshold that stops fitting. A threshold chosen on one dev split can stop fitting when the mix of messages changes. The sign is a shift in the hand-off rate, or in accuracy on the messages kept, when the router is measured on new dev messages. Chapter 5 records the threshold with the dev split it was chosen on, so the bank can see what the number was fitted to, and [chapter 9](version-watch-and-fall-back#watch-the-hand-off-rate) watches for the shift.

The fourth is cost and the load on people. A hosted model bills each call. A router that the bank runs bills in compute on its own machine, and [chapter 3](choose-a-base-model#measure-what-one-message-costs) counts what one message costs in tokens and forward passes, while [chapter 10](when-a-small-model-is-the-wrong-choice#count-the-bill-and-the-machine) turns a recorded speed into a bill. Each hand-off is work for a person, and the hand-off rate is the share of that work. We recommend putting the hand-off rate on every report about the router, beside accuracy, so the cost of the flag is never left out.

A router that cannot say when it is unsure hides its mistakes, and the flag makes them countable. The chapter made these decisions. Later chapters keep them, or change them and say why:

- The router returns a team, a confidence and a hand-off flag, and every route that returns a team goes through `handoff`.
- The threshold is a placeholder until chapter 5 chooses it on the dev split.
- Every router is scored by accuracy, the hand-off rate and accuracy on messages not handed off, on the same fixed sample, with counts beside each.
- The stand-in is the baseline, and its answers are recorded. A reader's own model records its answers the same way, with `record_answers`.
- A model that does not return a team for every message is not yet a router.

# Measure the routing task before you choose a model

By the end of this chapter, you can say what each of the book's samples is for, compute the three measures the router is scored on with their denominators, and read a baseline that was recorded once and is reused for every comparison.

Today a hosted model routes the messages that reach the bank's support inbox. The book's router is a small open model that the bank would run itself, and the question is whether it can match that hosted model on the messages it keeps, and say so when it is not sure of a message. Before any model is compared, the comparison needs fixed data, fixed samples and fixed measures. Otherwise two numbers that look alike may come from different messages, or be divided by different counts.

This book doesn't call a hosted model, so it uses a stand-in, a larger open model: Qwen3-4B-Instruct-2507. Its answers come from the 4-bit Q4_K_M GGUF conversion that a third party published, at its pinned revision, with its answers limited to the 77 team names. It answers every test message once, in a recording, and the examples read those answers offline, so anyone can reproduce them. A reader with a model of their own can record its answers the same way, as [chapter 1](route-a-message#ask-a-model-for-one-team) shows.

## The data and its licence {#the-data-and-its-licence}

The router is trained and measured on BANKING77, a public dataset of customer service queries in the banking domain (Casanueva et al., 2020). Each message carries one of 77 labels. BANKING77 calls its labels intents; they are the inbox's teams ([chapter 1](route-a-message)).

Casanueva et al. (2020) introduce the dataset in *[Efficient Intent Detection with Dual Sentence Encoders](https://arxiv.org/abs/2003.04807)*, in the Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45 (Anthology ID 2020.nlp4convai-1.5; arXiv 2003.04807). Their abstract describes it as a hard dataset from one domain, with 13,083 labelled examples over 77 intents, and their section 3 gives the same count for the banking-domain set, which they name BANKING 77. The data card gives the commit the files come from. The two files together hold the same 13,083 messages, which the next section counts.

The data card also states the facts the book relies on. It says that the files are a copy of the source's own train and test split, which is not drawn again, and that every team appears in both files. A router trained on the training file and scored on the test file is scored on the source's own held-out messages, not on a split that the book chose.

The licence is CC BY 4.0. The LICENSE file that sits beside the data begins "Attribution 4.0 International". Under that licence the data may be shared and adapted, provided the source is credited. The book keeps the licence file beside the data and carries the citation in the data card, so a reader who reuses the files has what the licence asks for.

The messages stay as published. No example prints message text, and no chapter quotes one, so the content of the messages does not reach a page. Everything the examples print from the data is a count or a rate.

**Working rule:** Keep the licence file beside the data, cite the paper in the data card, and print counts, never message text. If you reuse a dataset of real customer messages, read its card before you print anything from it, and cite the source where the data is documented.

## Dev, test and fixed samples {#dev-test-and-fixed-samples}

The [two files of the source split](#the-data-and-its-licence) are not enough by themselves. A router needs messages to tune on, messages to report on, and a small set that examples can run in seconds. Each one has to be fixed in advance, or a later result can be explained by a different sample rather than by a different model.

The dev split is carved from the training file, never from the test file. For each team, in the order of the categories file, the book draws about a tenth of that team's training messages with a fixed seed, and it commits the row numbers to `examples/data/banking77/samples.json`. The training messages that are not in dev are the ones the models train on. Because the draw is made team by team, each team's share of dev follows its share of the training file, and a team with few training messages keeps a small dev share instead of none.

The test split is the source's own held-out file. The book draws one stratified sample from it, 13 messages for each team, which gives 1001 messages in all. The sample is shuffled once, with the same generator as the dev draw, continuing after it, and the shuffled order is committed. Three prefixes of that order are named. The first 40 messages of the shuffled test sample are `test_40`, the first 200 are `test_200`, and the whole sample is `test_1001`. A prefix of a shuffled sample is a random draw, so `test_40` does not read the teams in order.

The counts come from the committed samples, and the book quotes only those. The first of them, count-the-splits, also checks that every dev row number is unique and falls inside the training file, so every dev message is a training row, and that `test_40` and `test_200` are the first 40 and first 200 of `test_1001`.

The first thing to see is how big each split is and how unevenly the teams are spread across them. Counting needs no model, so the reader can run it first. It reads the training file, the test file and the dev sample, and counts each split:

Example `count-the-splits` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Count each BANKING77 split and its messages per team, and check the committed samples; no model runs.

The prose's point rests on the four "per label" lines for the test and training files.
"""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402
from collections import Counter  # noqa: E402

router_lib.setup_runtime()

train = router_lib.read_rows("train")
test = router_lib.read_rows("test")
dev = router_lib.read_sample("dev")  # dev row numbers index train.csv, never test.csv

assert len(set(dev)) == len(dev), "dev indices repeat"
assert max(dev) < len(train), "dev indices fall outside train.csv"

train_per_label = Counter(label for _, label in train)
test_per_label = Counter(label for _, label in test)

test_1001 = router_lib.read_sample("test_1001")  # committed order: prefixes are the first n of it
test_40 = router_lib.read_sample("test_40")
test_200 = router_lib.read_sample("test_200")
# test_40 and test_200 are prefixes of the shuffled order, not new draws
assert test_1001[:40] == test_40, "test_40 is not the first 40 of test_1001"
assert test_1001[:200] == test_200, "test_200 is not the first 200 of test_1001"
sample_per_label = Counter(test[i][1] for i in test_1001)  # rows are (text, label index); [1] is the label

train_counts = [train_per_label[i] for i in range(len(router_lib.LABELS))]  # one entry per team, so a team with no rows counts 0
test_counts = [test_per_label[i] for i in range(len(router_lib.LABELS))]
sample_counts = [sample_per_label[i] for i in range(len(router_lib.LABELS))]

print(f"train rows: {len(train)}")
print(f"dev rows: {len(dev)}")
print(f"test rows: {len(test)}")
print(f"labels: {len(router_lib.LABELS)}")
print(f"test per label min: {min(test_counts)}")
print(f"test per label max: {max(test_counts)}")
print(f"train per label min: {min(train_counts)}")
print(f"train per label max: {max(train_counts)}")
print(f"test_1001 per label min: {min(sample_counts)}")
print(f"test_1001 per label max: {max(sample_counts)}")
```

Output:

```text
train rows: 10003
dev rows: 1003
test rows: 3080
labels: 77
test per label min: 40
test per label max: 40
train per label min: 35
train per label max: 187
test_1001 per label min: 13
test_1001 per label max: 13
```

The test file is even, with the same count for every team, and the training file is not: its teams run from 35 messages to 187. That spread is why the dev draw is made team by team rather than as one random tenth of the file. The test file's own fewest and most, 40 and 40, are equal.

The stratified sample is even as well, at 13 to 13 messages a team.

The even test file is a property of the source, and the book relies on it. Equal counts per team mean that the test split does not reward a router for being good at the largest team. A router that does well on a sample with uneven counts can look better than it is, because most of its score comes from the teams with the most messages.

[Chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) chooses the hand-off threshold on dev, and it records the choice in the adapter's manifest. The rule behind that choice is to tune on dev, never on test. The prompt, the model and the recipe are fixed before any test result is read, and none is changed because of one, and the test results are read once, to report them, on the fixed samples above.

In production the same rule holds for the inbox. Keep a dev set drawn from messages the inbox already has, and keep a test set that nobody reads while the router is being tuned. A test set that is read during tuning still looks like a test set, but it stops measuring anything new, and a router that is later measured on it will look better than it will be on the traffic that follows.

**Working rule:** Draw dev from training rows only, team by team, with a fixed seed, and commit its row numbers. Draw each test sample once, commit its order, and read it only to report. Choose every threshold, prompt and setting on dev. When dev is too small to decide, draw more dev messages from the training file, never from the test file.

## Three measures, each with its denominator {#three-measures-each-with-its-denominator}

The [samples](#dev-test-and-fixed-samples) say which messages to score. The measures say what to count. A router that returns a team, a confidence and a hand-off flag can be scored three ways, and each way answers a different question for the bank that runs it.

Accuracy asks how often the router's team is the right one. The hand-off rate asks how often a person has to read a message. Accuracy on the messages not handed off asks how often the router is right on the messages it keeps. Each question has its own denominator, and the book keeps the three apart in every output line. Confidence is the probability the router gives its team, and the threshold is the value below which a message is handed off, as [chapter 1](route-a-message#hand-off-the-unsure-messages) defines them.

| Measure | Counts | Divided by |
| --- | --- | --- |
| Accuracy | Messages whose team equals the gold team | All messages in the sample |
| Hand-off rate | Messages whose confidence is below the threshold | All messages in the sample |
| Accuracy on messages not handed off | Correct messages among those not handed off | Messages not handed off |

As formulas, with N messages in the sample: accuracy = correct / N; hand-off rate = handed off / N; accuracy on messages not handed off = correct among kept / kept, where kept = N minus handed off.

Two rules make these measures comparable across candidates. The first is that a handed-off message counts in the accuracy denominator, as wrong when its team is wrong. Accuracy alone does not reward a hand-off, because a handed-off message scores as correct only when its team is right. Accuracy on the messages not handed off, taken alone, hides the cost of the hand-off, because it leaves out exactly the messages that went to people. The hand-off rate shows that cost. The three numbers have to be read together, and a router that is better on one of them alone has not yet shown it is better.

The second rule is about printing. Every rate is rounded to one decimal place as a percentage, and it is printed beside its count, so a reader can check a rate by dividing by hand. A rate with no count beside it cannot be checked, and a rate that is printed without its denominator can be quoted in a way that hides its sample.

When the denominator of the third measure is zero, the measure is not defined. That happens when every message is handed off, so no message is kept. The book prints "not defined" in that case, as the [baseline section](#the-baseline-from-recorded-answers) shows, and does not print zero, because zero would say that every kept message was wrong when there are no kept messages at all.

These measures score labels. The scripts in this chapter, and the measurements in [chapter 1](route-a-message#ask-a-model-for-one-team) and [chapter 4](prompt-and-constrain), return teams only to be scored, and they return nothing to a caller. A router that returns a team to a caller does so through the hand-off rule that [chapter 1](route-a-message) introduces, so that a message below the threshold goes to a person rather than to a team.

In production, the hand-off rate is the measure that changes first. The mix of messages changes with the seasons, with a product launch, or with a fault that floods the inbox with one kind of question. A router whose accuracy holds can still hand off far more messages than the dev split predicted, and the only way to see it is to print the rate with its count on the traffic the router actually receives.

**Working rule:** Print each of the three measures with its count and its denominator, and round every rate to one decimal place. Report accuracy on the messages not handed off beside the hand-off rate, never alone. When no message is kept, print "not defined" rather than zero.

## The baseline from recorded answers {#the-baseline-from-recorded-answers}

A measure means little until something has been scored against it. The baseline is the stand-in's score on `test_1001`, the sample that the final comparison in [chapter 10](when-a-small-model-is-the-wrong-choice) uses. Later chapters score candidates on its prefixes, `test_40` and `test_200`, which are the first 40 and first 200 messages of the same fixed sample. Each prefix is scored from the same recording, so the baseline on it is the same answers for the same messages.

The stand-in answered each of the messages in `test_1001` once. The answers are stored in a recording, `examples/data/recorded/standin-baseline/answers.jsonl`, and each answer names a team and gives its log-probability. The recording has a manifest beside it. The manifest names the script that made the recording, the model and its pinned revision, the hash of the prompt, the decoding settings and the seed, and the hash of the answers file. The code checks the answers file against that hash before it reads a line, so a changed file fails the run rather than changing a score.

The recording is read, not run. Every example that uses the baseline reads the same answers, so the baseline cannot change from one run to the next, even if the runtime that made the recording is later updated. The reason for recording, rather than calling a model in each example, is the same reason the pins exist. A recording fixes the answers to one model at one revision, so a reader who runs the same example later sees the same numbers. The book never calls a hosted API, because a hosted model's answers are not fixed in the same way. The stand-in is an open model that anyone can run, and its recording is the only route to its answers that the book takes.

The baseline is also not a ceiling. It is the score a candidate has to beat on the same messages, and it says nothing about the hand-off trade-off by itself. A candidate that beats it on accuracy may still hand off more messages than the support staff can read, so the baseline is always read beside the three measures and never as a single number.

What does the placeholder threshold cost? Scored with the three measures on `test_1001`, the stand-in shows how many messages a threshold of 0.5 hands off, and how often it is right on the rest. Its confidence is the probability that llama-cpp-python returns for its constrained answer, which is exp of the recorded log-probability. That threshold is chapter 1's placeholder, so its hand-off rate is not the book's final one, and [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) chooses the real threshold on dev. The example also applies a threshold above every confidence, which hands off every message.

Example `baseline-measures` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Score the stand-in's recorded answers on test_1001 with the three measures, at chapter 1's placeholder threshold and at one above every confidence; no model runs. The point rests on the accuracy and hand-off lines, and the answers' hash is checked first.
"""

import hashlib, json, math, os

os.environ.update(HF_HUB_OFFLINE="1", TRANSFORMERS_OFFLINE="1")

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()

PLACEHOLDER = 0.5  # Chapter 5 chooses the real threshold on the dev split.
ABOVE_ALL = 1.01  # Confidence is at most 1.0, so this hands off every message.
RECORDED = router_lib.HERE / "data" / "recorded" / "standin-baseline"
manifest = json.loads((RECORDED / "manifest.json").read_text())
raw = (RECORDED / "answers.jsonl").read_bytes()
assert hashlib.sha256(raw).hexdigest() == manifest["output_sha256"]["answers.jsonl"]

answers = {row["index"]: row for row in map(json.loads, raw.decode().splitlines())}  # keyed by test.csv row, not by sample position
rows = router_lib.read_rows("test")
sample = router_lib.read_sample("test_1001")
golds = [rows[i][1] for i in sample]  # in the sample's committed order, so position k of each list is one message
preds = [answers[i]["label"] for i in sample]
# Confidence is the probability llama-cpp-python returns for the constrained answer: exp of its log-probability.
confs = [math.exp(answers[i]["logprob"]) for i in sample]

n = len(sample)
first = router_lib.metrics(preds, golds, confs, PLACEHOLDER)  # one call returns the counts and rates of all three measures
high = router_lib.metrics(preds, golds, confs, ABOVE_ALL)
print(f"messages: {n}")
print(f"placeholder threshold: {PLACEHOLDER}")
print(f"accuracy: {first['correct']}/{n} ({first['accuracy']:.1%})")
print(f"handed off: {first['handed_off']}/{n} ({first['handoff_rate']:.1%})")
m = first["not_handed_off"]
print(f"accuracy on messages not handed off: {first['correct_not_handed_off']}/{m} ({first['accuracy_not_handed_off']:.1%})")
print(f"threshold above every confidence: {ABOVE_ALL}")
print(f"handed off at that threshold: {high['handed_off']}/{n} ({high['handoff_rate']:.1%})")
if high["accuracy_not_handed_off"] is None:  # None means no message is kept, so the measure has no denominator
    print("accuracy on messages not handed off at that threshold: not defined (0 messages not handed off)")
else:
    print(f"accuracy on messages not handed off at that threshold: {high['correct_not_handed_off']}/{high['not_handed_off']}")
```

Output:

```text
messages: 1001
placeholder threshold: 0.5
accuracy: 560/1001 (55.9%)
handed off: 30/1001 (3.0%)
accuracy on messages not handed off: 554/971 (57.1%)
threshold above every confidence: 1.01
handed off at that threshold: 1001/1001 (100.0%)
accuracy on messages not handed off at that threshold: not defined (0 messages not handed off)
```

At the placeholder threshold, the stand-in gets 560/1001 (55.9%) right and hands off 30/1001 (3.0%). On the messages it keeps, its accuracy is 554/971 (57.1%).

At a threshold above every confidence, the hand-off rate is 1001/1001 (100.0%), so the accuracy on messages not handed off is not defined (0 messages not handed off).

A reader's own hosted model is a different baseline. It will score differently on the same messages, and the book does not claim that the stand-in's numbers are the inbox's numbers. The reader's baseline is their model's recorded answers on `test_1001`, scored with the three measures in this chapter. The comparison that matters is between the reader's candidate and that baseline, on the same messages, with the same counts.

The [next section](#where-the-labels-collide) counts the stand-in's wrong answers and shows which teams they fall between. The wrong-answer count is the baseline's first number.

**Working rule:** Record the baseline once, on `test_1001`, the fixed sample that every candidate's prefixes come from. Store the model's pinned revision, the prompt's hash and the decoding settings in a manifest, and check the answers file against its hash before you read a line. Compare each candidate only with the baseline scored on the same messages, whether that is `test_40`, `test_200` or `test_1001`. When your own hosted model is the baseline, record its answers the same way, and say in the write-up that its scores differ from the stand-in's.

## Where the labels collide {#where-the-labels-collide}

The [baseline's counts](#the-baseline-from-recorded-answers) say how often the stand-in is wrong. They do not say which teams it mixes up, and the mix-ups are what the support staff can act on. A pair of teams that the router confuses might need a clearer team description, a rule for that pair, or more messages for one of the two teams in training. None of those is known from the total count, so the book reads the pairs from the recorded answers rather than assuming which teams are close.

Counting the wrong answers needs the recording itself, so the next example reads it, checks it against the manifest, checks that it holds exactly the messages of `test_1001`, and then counts the wrong answers and the three gold-to-answer pairs that occur most often. Ties break on each team's position in the categories file, so the same pairs print in the same order on every run.

Example `confusions-in-the-baseline` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Count the stand-in's wrong answers, its three most frequent gold-to-answer pairs and the teams it never answers, on test_1001; no model runs. The point rests on the "pair" and "never answered" lines.
"""

import hashlib, json, os

os.environ.update(HF_HUB_OFFLINE="1", TRANSFORMERS_OFFLINE="1")

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402
from collections import Counter  # noqa: E402

router_lib.setup_runtime()

RECORDED = router_lib.HERE / "data" / "recorded" / "standin-baseline"
manifest = json.loads((RECORDED / "manifest.json").read_text())
raw = (RECORDED / "answers.jsonl").read_bytes()
assert hashlib.sha256(raw).hexdigest() == manifest["output_sha256"]["answers.jsonl"]

records = [json.loads(line) for line in raw.decode().splitlines()]
sample = router_lib.read_sample("test_1001")
assert sorted(r["index"] for r in records) == sorted(sample), "recorded rows differ from test_1001"
assert all(router_lib.LABELS[r["label"]] == r["team"] for r in records), "label and team disagree"
test, LABELS = router_lib.read_rows("test"), router_lib.LABELS

gold_totals = Counter(test[i][1] for i in sample)
answered = Counter(r["label"] for r in records)
pairs = Counter((test[r["index"]][1], r["label"]) for r in records if r["label"] != test[r["index"]][1])  # (gold, answer); [1] is the gold label
wrong = sum(pairs.values())
# Ties break on the lower gold label index, then the lower answer index.
top = sorted(pairs.items(), key=lambda kv: (-kv[1], kv[0][0], kv[0][1]))[:3]
never = sorted(team for k, team in enumerate(LABELS) if answered[k] == 0)  # sorted by name, so uppercase names print first

print(f"messages: {len(sample)}")
print(f"wrong answers: {wrong}")
print(f"wrong rate: {100 * wrong / len(sample):.1f}% ({wrong} of {len(sample)})")
for rank, ((gold, answer), count) in enumerate(top, start=1):
    print(f"pair {rank} count: {count}")
    print(f"pair {rank} gold: {LABELS[gold]}")
    print(f"pair {rank} answer: {LABELS[answer]}")
    print(f"pair {rank} of gold team: {count}/{gold_totals[gold]}")
print(f"teams never answered: {len(never)}")
for rank, team in enumerate(never, start=1):
    gold = LABELS.index(team)
    count, answer = max(((c, a) for (g, a), c in pairs.items() if g == gold), key=lambda ca: (ca[0], -ca[1]))  # most frequent wrong answer; ties go to the lower answer index
    print(f"never answered {rank} team: {team}")
    print(f"never answered {rank} messages: {gold_totals[gold]}")
    print(f"never answered {rank} most often as: {LABELS[answer]}")
    print(f"never answered {rank} most often count: {count}")
```

Output:

```text
messages: 1001
wrong answers: 441
wrong rate: 44.1% (441 of 1001)
pair 1 count: 13
pair 1 gold: Refund_not_showing_up
pair 1 answer: request_refund
pair 1 of gold team: 13/13
pair 2 count: 13
pair 2 gold: failed_transfer
pair 2 answer: transfer_not_received_by_recipient
pair 2 of gold team: 13/13
pair 3 count: 12
pair 3 gold: card_arrival
pair 3 answer: card_delivery_estimate
pair 3 of gold team: 12/13
teams never answered: 4
never answered 1 team: Refund_not_showing_up
never answered 1 messages: 13
never answered 1 most often as: request_refund
never answered 1 most often count: 13
never answered 2 team: card_arrival
never answered 2 messages: 13
never answered 2 most often as: card_delivery_estimate
never answered 2 most often count: 12
never answered 3 team: failed_transfer
never answered 3 messages: 13
never answered 3 most often as: transfer_not_received_by_recipient
never answered 3 most often count: 13
never answered 4 team: order_physical_card
never answered 4 messages: 13
never answered 4 most often as: get_physical_card
never answered 4 most often count: 10
```

The stand-in is wrong on 44.1% (441 of 1001). With no threshold applied, the error is one minus accuracy, so that wrong rate and the accuracy in [the baseline section](#the-baseline-from-recorded-answers) add up to 100%.

Each of the three most frequent pairs takes all or nearly all of its gold team's sample messages. The first pair takes 13/13, the second 13/13, and the third 12/13, so the first two take every message of their gold team and the third all but one. The team names suggest a reason for each pair: the first concerns a refund, the second a transfer, and the third a card's arrival and its delivery estimate. That is a reading of the names. The chapter does not read the messages, and no example prints them, so the pairs are a reason to look, not a finding about what the messages say.

The stand-in never names 4 of the 77 teams, so every message whose gold team is one of those is wrong. For card_arrival, one of those teams, the most common wrong answer is card_delivery_estimate, which takes 12 of its messages.

For order_physical_card, another of them, the most common wrong answer is get_physical_card, which takes 10 of its messages. The recording does not show why these teams are never given, so the list is a reason to look.

The pairs describe the stand-in on this sample, not the teams or the inbox. A reader's own model will have its own pairs, and a reader should count them from their own recorded answers. When the question is how to fix a pair, the fix has to be chosen and checked on dev, as [the dev and test section](#dev-test-and-fixed-samples) requires. A change that is chosen by reading the pairs on `test_1001` has been tuned on the test split, and its result on that split cannot be reported as a measure of the router.

When hand-offs or complaints rise for one team in production, repeat this count first. Count the pairs again on the inbox's own recorded answers, print each pair with its count, and compare the list with the one from the baseline. A change in the list says where the router's behaviour has moved, even when the overall rate has not.

The chapter's decisions, in short:

- Dev is drawn team by team from the training file, with a fixed seed, and its row numbers are committed.
- The test samples are fixed in order, and they are read only to report.
- Each measure is printed with its count and denominator, and "not defined" replaces a zero denominator.
- The stand-in's recorded answers on `test_1001` are the baseline, read from a manifest-checked file.

The baseline's three numbers, read together, are the score a candidate has to beat on the same messages. [Chapter 3](choose-a-base-model#measure-what-one-message-costs) counts the router prompt's tokens on `test_40` before it chooses a base model. [Chapter 5](fine-tune-with-lora) chooses the hand-off threshold on dev, with the rule stated there. [Chapter 10](when-a-small-model-is-the-wrong-choice) compares the router with the baseline on the same `test_1001` sample.

**Working rule:** Count the wrong answers and the most frequent pairs for every candidate, on the same sample, and print each pair with its count. Read the pairs as a list of where to look. Print the three most frequent pairs rather than only the first, because one pair invites over-reading, and a list shows whether the errors cluster or scatter. Check any fix on the dev split before the test split is read again.

# Choose a base model by counting what one message costs

By the end of this chapter, you can name the two base models this book keeps, load each one at its pinned revision, and count what one routing message costs in tokens and parameters before you commit to a model.

Every message in the bank's support inbox costs something to route: a model reads it and picks its team, and the model the router starts from sets what that costs. The router is the whole system that [chapter 1](route-a-message) builds, and this chapter chooses the small open model under it. [Chapter 2](measure-the-routing-task#dev-test-and-fixed-samples) fixed the samples that the router is measured on. This chapter chooses by cost first, because the bank can count what a message costs before it builds anything around the model.

## Which candidates we keep {#the-candidates-and-what-we-keep}

A base model is a model already pretrained on a large amount of text, which you adapt to one job. Every later chapter builds on this choice, through the prompt, the training, the distillation and the serving, so it is worth making on evidence. There are many open models, so the book narrows the field with two questions: can the model run on the hardware the bank has, and can the bank use its licence?

The licence comes first in practice. It decides what the bank may do with the model in its own system, and a model the bank may not run is not a candidate, whatever its size. The book records each model's licence beside its pin, so a reader can see the same record the book used.

The first candidate is SmolLM2, a family of models published on Hugging Face. The [model card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/tree/12fd25f77366fa6b3b4b768ec3050bf629380bac) for the 135M model, at the commit this book pins, gives the family's three sizes in one sentence: 135M, 360M and 1.7B parameters. The same card lists the licence as `license: apache-2.0`, which the book's pins record for both models it keeps. The card also gives the 135M model's training data as 2 trillion tokens. Those figures come from the card for the 135M model only.

The book keeps two of the three sizes. The list below names them, and the table after it sets all three side by side, with the licence each pin records.

- SmolLM2-135M-Instruct is the core model. The rest of the book trains and serves it.
- SmolLM2-360M-Instruct is a comparison. The book loads it, counts its parameters and layers, and uses it beside the core model in this chapter and in [chapter 4](prompt-and-constrain). The book does not train it.

| Model | Role in this book | Pinned in `models.toml` | Licence |
| --- | --- | --- | --- |
| SmolLM2-135M-Instruct | Core model, trained and served by the rest of the book | Yes | Apache-2.0 |
| SmolLM2-360M-Instruct | Comparison beside the core model | Yes | Apache-2.0 |
| SmolLM2 1.7B | Not pinned and not run | No | None recorded |

The 1.7B model is not pinned in the book's `models.toml` file, and the book does not run it. The book's examples set two CPU threads, so each candidate has to load and run within that budget. For a bank with that budget, we recommend the 135M model as the starting point, for the reasons in the rest of this chapter.

The book's reference point is a different model. In the case, a hosted model routes the bank's messages today, and this book never calls one. Its stand-in is Qwen3-4B-Instruct-2507, a larger open model whose answers to the test messages are recorded once and then read offline. The stand-in is the baseline the candidates are measured against, not a candidate for the router. [Chapter 1](route-a-message#ask-a-model-for-one-team) introduces it.

**Working rule:** Start with the smallest model in the family that passes the measures you set on the dev split, the training messages held back for choosing ([chapter 2](measure-the-routing-task#dev-test-and-fixed-samples)). Move up one size only when the smaller model misses a measure, and only after you have measured the larger model on the same samples. This is a rule of thumb. It holds until your own measurements say otherwise.

## Load each model at its pinned revision {#load-each-model-at-its-pinned-revision}

The [previous section](#the-candidates-and-what-we-keep) named two models. A name alone does not say which version of the model's files you load, because a model is stored in a repository of commits, and a branch such as `main` can move to a new commit. A revision pinned by a full commit hash names one snapshot of the files, and the pins file, `examples/models.toml`, records the SHA-256 of every file in that snapshot, which the runtime checks. The shared loader passes the commit to the transformers library, which the book pins at 5.23.0, and the loader looks up the commit for each model in that file.

For each model, the file holds the repository, the commit, the licence and the SHA-256 of every file the model needs. Its header says the runtime fetches each model once, checks it file by file, and then reads it offline. The file is generated by `re-check models --pin`, and its header says not to edit it by hand. So a reader who changes a pin runs that command rather than typing a new commit into the file. A model without a pin has no commit to look up, so the shared loader does not load it.

The examples set the library's offline flags before they import it, so a load reads only what the runtime already holds. The loader also sets the weights to float32 and runs attention in the library's plain eager form. The book's determinism rules fix both, so every run of an example gives the same numbers. If an offline load fails on a reader's machine, compare the files on disk with the list in the pins file before you change anything else.

The example also watches the pins from the output side. It prints each model's parameter count, so a change in a model's size shows up as a changed line in the book's own output, and every number the chapter quotes from that line changes with it. A reader who sees a different count knows which pin to compare first. A change of commit that keeps the size the same does not show there, so the pin itself stays the record.

One example loads both kept models from the files the runtime fetched and checked against the pins, with no network, and prints the book's first size counts.

Example `load-both-models-offline` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Load the 135M and 360M SmolLM2 models at their pins, offline, and print each one's parameter count and layer count. The chapter's size claims rest on both lines; the parameter line is rounded to whole millions."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

# Seeds, deterministic algorithms and float32, set before either model loads.
router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"
MEDIUM = "HuggingFaceTB/SmolLM2-360M-Instruct"


def size(model):
    """Return (parameter count in millions, rounded to a whole number) and the layer count."""
    # parameters() yields a shared tensor once, so the tied embedding and output head count once.
    total = sum(p.numel() for p in model.parameters())
    return round(total / 1e6), model.config.num_hidden_layers


# load_model returns (tokenizer, model). This example needs only the models.
_, small = router_lib.load_model(SMALL)
_, medium = router_lib.load_model(MEDIUM)

small_millions, small_layers = size(small)
medium_millions, medium_layers = size(medium)

print(f"parameters, millions, rounded: {small_millions} and {medium_millions}")
print(f"layers: {small_layers} and {medium_layers}")
```

Output:

```text
parameters, millions, rounded: 135 and 362
layers: 30 and 32
```

A parameter is one of the numbers the model learns during training, and a layer is one of the stacked blocks that a token passes through. Parameters and layers are the only two sizes this chapter uses. The 360M is the larger of the two. The first output line gives each model's parameter count, rounded to whole millions: 135 and 362. The second line gives the layer counts, 30 and 32, in the same order. Speed is not among them, because a time depends on the machine. The `throughput` recording measures the classifier and its int8 copy against the stand-in on a named machine ([chapter 7](quantize-and-measure#measure-speed-in-a-recording) says how, and [chapter 10](when-a-small-model-is-the-wrong-choice#count-the-bill-and-the-machine) reports the ratios). The recording does not measure the 360M.

The counts come from loading the models here, not from their cards.

In production, the pin answers the first question you ask when a result changes, which is which files produced it. A change of model shows up as a diff in `models.toml`, where the commit and the file hashes sit. When a reader's copy of a model gives different output from the book's, compare the commit first.

**Working rule:** Pin every model by its commit in `examples/models.toml`, and load it with that commit and the offline flags. Record the parameter and layer counts next to the pin, so that a change in a model's size shows up as a changed output line and not as a surprise in a later chapter.

## Measure what one message costs {#measure-what-one-message-costs}

The [previous section](#load-each-model-at-its-pinned-revision) showed what the two models are. This section shows what one message costs them. A message's cost comes from what the model reads and what it writes. The router's prompt holds the 77 team names, the message and a request for the team, and the team list is the same in every call, so every message pays for it.

The prompt opens with the sentence "Route this customer message from an online bank to one support team." It asks for the team name only, lists the teams, then gives the message and ends with the word "Team:". Everything in that text is the same from one message to the next except the message itself, so the cost of a message is the fixed text plus the message's own length.

The tokenizer turns the prompt's text into tokens, the units a model reads. The count is in tokens rather than characters because a model reads tokens. Counting needs the tokenizer's rules but no weights, so the count is the first cost you can take before you choose a design. A classifier reads the message once and returns a score for each team. A generative model reads the prompt once and then writes its answer one token at a time, and each token it writes is one more pass through the network. The later chapters use both designs.

One example counts the forward passes, each one call of the network, for a single test message under each design. It uses the 135M model with no adapter. The classifier's head is untrained here, so only the pass count matters.

Example `count-forward-passes` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Count the forward passes one test_40 message costs a classifier and a generative model, with the 135M model and no adapter. The point rests on the generative line: the model makes one pass for each token it writes, plus one for the end token."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402
from transformers import AutoModelForSequenceClassification  # noqa: E402

router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"
texts = [text for text, _ in router_lib.read_rows("test")]
# read_sample gives row indices into the test split; index 0 is the sample's first message.
message = texts[router_lib.read_sample("test_40")[0]]


def count_passes(model, run):
    """Return how many forward calls the model makes while run() executes."""
    calls = []
    # A forward hook on the whole model fires once per call to its forward, so each generation step counts.
    handle = model.register_forward_hook(lambda module, args, output: calls.append(1))
    run()
    handle.remove()
    return len(calls)


tokenizer, causal = router_lib.load_model(SMALL)
# SmolLM2's tokenizer sets a pad token; this line keeps the example working with one that doesn't.
tokenizer.pad_token = tokenizer.pad_token or tokenizer.eos_token
# The trie holds each team name's tokens, so constrained_team can write only a team name, one token per pass, then the end token.
trie = router_lib.build_trie(tokenizer)
# The pinned commit from models.toml, and the float32 weights and eager attention that load_model also sets.
# The head is new, with one score per team (77), and untrained here; only the pass count matters.
classifier = AutoModelForSequenceClassification.from_pretrained(
    SMALL, revision=router_lib._pins()[SMALL], num_labels=len(router_lib.LABELS),
    dtype=torch.float32, attn_implementation="eager",
).eval()
# The classification head finds each message's last token through this id, so the config needs it too.
classifier.config.pad_token_id = tokenizer.pad_token_id

classifier_passes = count_passes(classifier, lambda: router_lib.classify(classifier, tokenizer, [message]))
answers = []
generative_passes = count_passes(causal, lambda: answers.append(router_lib.constrained_team(tokenizer, causal, trie, message)))
# answers[0] holds (label index, label text). The text is encoded with a leading space, as the model writes it after "Team:".
answer_tokens = len(tokenizer.encode(" " + answers[0][1], add_special_tokens=False))

print(f"classifier forward passes: {classifier_passes}")
print(f"generative forward passes: {generative_passes}")
print(f"answer tokens, end token not counted: {answer_tokens}")
```

Output:

```text
classifier forward passes: 1
generative forward passes: 8
answer tokens, end token not counted: 7
```

A generative model's cost grows with its answer, and a classifier's does not. For this message the classifier makes 1 pass. The generative model makes 8 passes, one for each of the 7 tokens in its answer and one for the end token.

A second example counts the router prompt's tokens for each of the 40 messages in the fixed test sample, using the 135M model's tokenizer. It runs no model.

Example `count-prompt-tokens` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Count the router prompt's tokens for each message in test_40, with the 135M model's tokenizer. No model runs. The point rests on the mean line: a day's prompt cost is the mean, not the median, times the number of messages."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402
from transformers import AutoTokenizer  # noqa: E402

router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"

# Only the tokenizer is read from the pinned commit; no weights load.
tokenizer = AutoTokenizer.from_pretrained(SMALL, revision=router_lib._pins()[SMALL])

texts = [text for text, _ in router_lib.read_rows("test")]
# read_sample gives row indices into the test split, in the order the sample lists them.
messages = [texts[i] for i in router_lib.read_sample("test_40")]

# router_prompt builds the same prompt the router sends. encode keeps its default, so any special tokens are counted too.
counts = sorted(len(tokenizer.encode(router_lib.router_prompt(message))) for message in messages)

# The median of an even count is the lower of the two middle values, so it is a whole number of tokens.
median = counts[(len(counts) - 1) // 2]

total = sum(counts)
mean = round(total / len(counts))

print(f"messages: {len(counts)}")
print(f"prompt tokens min: {counts[0]}")
print(f"prompt tokens median: {median}")
print(f"prompt tokens max: {counts[-1]}")
print(f"prompt tokens total: {total}")
print(f"prompt tokens mean, rounded: {mean}")
```

Output:

```text
messages: 40
prompt tokens min: 580
prompt tokens median: 585
prompt tokens max: 614
prompt tokens total: 23493
prompt tokens mean, rounded: 587
```

The prompts differ only in their messages, because the team list is the same text in every prompt. So the spread runs from 580 tokens for the shortest prompt to 614 for the longest, with a median of 585 tokens. The median is the lower of the two middle values, so it is a whole number of tokens.

In production, the count moves when the team list moves. Adding a team adds its name to every prompt, so every message costs more from that day on. Rerun the count after each change to the list, and keep the list in the same order, so that the rest of the prompt does not change with it.

To turn the count into a figure for a day, use the mean, not the median, because a day's prompt tokens are the sum of its prompts. The 40 sample messages total 23493 tokens, a mean of 587 tokens each. The mean is above the median here, so the median times the count would undercount the prompt tokens. Multiply the mean by the number of messages the router handles that day to get the prompt tokens it reads, before any output is counted. The output side depends on the design, and the designs come in later chapters.

This section gives counts only. The time a message takes is the recording's job, as [the load step](#load-each-model-at-its-pinned-revision) noted.

**Working rule:** Count the prompt's tokens with the tokenizer of the model you will run, before you choose a design, and count again whenever the prompt changes. The count belongs to that tokenizer, so a count from another model's tokenizer answers a different question.

## Why 135M is the core {#why-135m-is-the-core}

The [earlier sections](#the-candidates-and-what-we-keep) gave the 135M model its place: it is smaller, and it is the model the rest of the book trains and serves. This section gives the reasons for that choice, and says where the choice is not yet proven.

There are two broad ways to get a small model to route a message. One is to ask a model, in the prompt, which team to pick. The other is to train a model on labelled messages. Bucher and Martini compared fine-tuned BERT-style models with large hosted models, such as ChatGPT and Claude Opus, prompted zero-shot, in a 2024 preprint. Their abstract reports that the smaller fine-tuned models beat the larger prompted ones, consistently and significantly ([arXiv:2406.08660v2](https://arxiv.org/abs/2406.08660v2)). The comparison pits fine-tuned small models against prompted large ones, not small against small, and it comes from one preprint. The book takes it as the reason to measure a fine-tuned small model first. It does not take it as proof that the fine-tuned model wins on the inbox.

The two designs also fail in different ways, and the cost counts do not show that. A generative model writes its answer as text, so the text can be a string that is not one of the 77 teams. A classifier scores each of the 77 teams, so its answer is always one of them, and the score is what the hand-off rule reads to send unsure messages to a person. [Chapter 4](prompt-and-constrain) constrains the generative answer to the 77 names, and [chapter 5](fine-tune-with-lora) trains the classifier.

The cost reasons rest on the counts in the [load step](#load-each-model-at-its-pinned-revision). By both counts it printed, the 135M model is the smaller: 135 and 362 million parameters and 30 and 32 layers, the 135M's figure first in each pair. The [model card for the 135M model](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/tree/12fd25f77366fa6b3b4b768ec3050bf629380bac) describes it as light enough to run on-device, and capable across a wide range of tasks. The book's examples are the evidence for the counts, and the card is the evidence for the description.

The card also limits what the model should do: "These models should be used as assistive tools rather than definitive sources of information." That caveat agrees with the router's design. A model this size is an assistant to support staff, and a message it is unsure about goes to a person. [Chapter 1](route-a-message#hand-off-the-unsure-messages) introduces that hand-off.

This chapter does not show that the 135M model passes the measures. [Chapter 4](prompt-and-constrain) runs both models on the same messages, which is the comparison this section lacks. [Chapter 5](fine-tune-with-lora) trains the 135M model on the labelled messages and measures it on the dev split against the measures set in [chapter 2](measure-the-routing-task#three-measures-each-with-its-denominator). Until then, the choice rests on the cost counts and on the study above.

Three things would change this choice, and each one is settled by a measurement rather than by argument. If the 135M model misses a dev measure after fine-tuning, the next step is to train the 360M from the same training rows, measure it the same way, and compare the two on the dev split. If the 135M model passes the dev measures but hands off more messages than support staff can take, the lever is the hand-off threshold. [Chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) sets that threshold on the dev split together with the hand-off rate it costs. If the bank routes a different set of messages, the choice starts again, because the measures in this chapter describe the banking messages and no others.

The choice has a cost the reader should weigh, and this chapter does not measure it. It gives neither model's load time nor its run time, and the working rule in the [first section](#the-candidates-and-what-we-keep) asks for that measurement before the choice is final. This chapter makes the choice provisionally, and the later chapters either confirm it or replace it on the same measures.

**Working rule:** Fine-tune the smaller model first, and measure it on the dev split against the measures you set before you looked at the test split. Keep the larger model as a comparison until the smaller one has missed a measure on the dev split, and do not switch models on the test split.

## What a bigger model would need {#what-a-bigger-model-would-need}

The [previous section](#why-135m-is-the-core) chose the smaller model and left the question of the larger one, SmolLM2-360M-Instruct, open. This section sets out what that question needs, and what the book does and does not know about it.

A larger model is not automatically a better one for a narrow task. A study first posted in 2024, and accepted at EMNLP 2025, compares fine-tuning, instruction-tuning, prompting and in-context learning across eight language models and eight text classification tasks. Its abstract concludes that bigger models do not consistently give better performance or lower variance ([arXiv:2402.12819v4](https://arxiv.org/abs/2402.12819v4)). It is one study, and the book uses it to justify measuring the larger model, not to rule the larger model out.

The open question is whether the 360M, trained with the same recipe as the 135M, would pass the dev measures by a margin that pays for its extra cost. The book does not train the 360M, so it does not know. It has not measured the training time or the memory that a 360M run would take on the two CPU threads the book's examples use. [Chapter 4](prompt-and-constrain) uses the 360M as a comparison on the same prompts.

A reader who wants that answer can follow the same path the book follows. The 360M is already pinned and loads offline, as [the load step](#load-each-model-at-its-pinned-revision) shows. Count the router prompt's tokens with its tokenizer, then train it on the same training rows as the 135M. Each step is a separate change, and the comparison stays on the same data, so the two results can be set side by side on the dev split.

The family's largest model, the 1.7B, would need more still. The book does not pin it, and it makes no claim about it.

A larger model's cost per message arrives with every message it serves, and the book does not measure that cost for the 360M. The [load step](#load-each-model-at-its-pinned-revision) already shows how much bigger the 360M is before anyone trains it, with 135 and 362 million parameters and 30 and 32 layers, the 135M's figure first in each pair. The `throughput` recording in [chapter 7](quantize-and-measure#measure-speed-in-a-recording) measures the classifier and its int8 copy against the stand-in on a named machine, and the 360M is not in it.

The decisions this chapter made, for a reader to check against their own bank's needs:

- The core model is SmolLM2-135M-Instruct, and the comparison is SmolLM2-360M-Instruct.
- Both models load at the commits in `examples/models.toml`, offline, through the shared loader.
- The router prompt's cost is counted in tokens with the 135M model's tokenizer, and no model runs for the count.
- No time or memory figure appears in this chapter. Those come later, from a measurement on a named machine.
- The 360M is not trained in this book.

The lesson to carry forward is that a published study can say a bigger model may not help, and only your own measurement on your own samples says whether it does.

**Working rule:** Before you choose a larger model, train it with the same recipe you used for the smaller one, and measure it on the same dev and test samples, on your own machine. Judge it by its measures and by what it costs to run on that machine, and not by a published score.

# Prompt a small model and constrain its answers to the teams

By the end of this chapter, you can write a router prompt for a small open model, constrain its answer so that it can only be a team name, and measure how many of those answers are right. You will also see why a valid answer is not a correct one, which is why the next chapter trains a model instead of asking it better questions.

A router needs every answer to be a team name, and when [chapter 1](route-a-message) let a small model answer in free text, the model did not return teams. This chapter first makes the answer a team by construction. It then runs the two open models from [chapter 3](choose-a-base-model), SmolLM2-135M-Instruct and SmolLM2-360M-Instruct, with one prompt and one constraint, and measures what they return. The 135M model is the one [chapter 5](fine-tune-with-lora) trains, so the counts here are the ones that training has to beat.

Two kinds of number appear, and the chapter keeps them apart. The live examples run on the first six messages of `test_40`, the first 40 messages of the [fixed sample](measure-the-routing-task#dev-test-and-fixed-samples), and the same six for both models. They show the mechanism. Six messages cannot separate two models, so the comparison uses a recording: each model and prompt answered all 200 messages of `test_200`, the first 200 of that sample, which includes those six, once, and the answers were saved. Reading the recording loads no model, and its counts come from the saved answers and nothing else.

## The prompt, settled before any recording {#the-prompt-settled-before-any-recording}

Every example in this chapter uses one router prompt. It tells the model that the message comes from an online bank, asks for one support team and the team name only, lists the 77 teams, and ends with `Team:`, so the model's next words are the answer. A raw model that has not been trained for the task gets nothing else, so the wording matters. This chapter does not tune it. The prompt was fixed before any recording, and the recording's manifest holds the prompt's hash. A changed prompt is a different recording, and its counts cannot be set beside the ones here.

The list of teams in the prompt is the same list the constraint uses. The trie in the next section is built from the same label names in the same order, so the prompt and the constraint cannot disagree about which teams exist. The same dependency works the other way. A reader who adds or renames a team changes the prompt, and so changes the recording's hash. The counts in this chapter then no longer describe that prompt, and the reader must record again before comparing.

The raw form passes the router prompt's text to the model as it stands. The chat form wraps the router prompt's text in the model's chat template, which adds a default system prompt. To see what the template changes, run the chat form on the same six messages:

Example `constrained-decode-135m-chat` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Route the first 6 messages of test_40 with the 135M model through its chat template, decoding only team names.
The template adds the model's default system prompt. The prose quotes this run's accuracy line next to the raw form's, and the messages are never printed."""

import os

# no network: the pinned model must already be in the local cache
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import collections  # noqa: E402

import torch  # noqa: E402

# two threads, so the sums add in the same order on every run
torch.set_num_threads(2)

import router_lib  # noqa: E402

# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats
router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"

tokenizer, model = router_lib.load_model(SMALL)
trie = router_lib.build_trie(tokenizer)
# (text, label index) for every test row, in file order; the index is the gold team
rows = router_lib.read_rows("test")
# row numbers of the first six messages of the fixed 40-message sample
sample = router_lib.read_sample("test_40", 6)
teams = set(router_lib.LABELS)

results = []
for i in sample:
    # the chat template wraps the message in the model's chat format, with its default system prompt
    text = router_lib.chat_prompt(tokenizer, rows[i][0])
    # the template has already written the special tokens, so the tokenizer adds none
    results.append(router_lib.constrained_team_text(tokenizer, model, trie, text, add_special_tokens=False))
valid = [text in teams for _, text in results]
# the walk's index, never the decoded text, is what counts
right = [index == rows[i][1] for (index, _), i in zip(results, sample)]
given = collections.Counter(text for _, text in results)
# most given team; a tie in count goes to the team listed first
top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))

print(f"messages: {len(sample)}")
print(f"answers that are a team: {sum(valid)}/{len(sample)}")
print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")
print(f"distinct teams given: {len(given)}")
print(f"most given team: {top}, {given[top]}/{len(sample)}")
```

Output:

```text
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 3
most given team: cancel_transfer, 3/6
```

The book uses the raw form as its default. The choice was made before any recording was read: both models in the comparison are measured on it, so the comparison is between models, not between prompts. The chat form was recorded afterwards, as an observation. On the 200 messages it got 11/200 (5.5%) right, and the raw form got 9/200 (4.5%). A gap of two messages in 200 is not a difference this sample can show in either direction, so the chat form is not shown to help, and the raw form is not shown to hurt, and the default is not changed because of this count. The trie's names begin with a space, which follows `Team:` in the raw form but not the new line that opens the answer in the chat template. The chat count is therefore for this trie as well as for the template. The result is not evidence that raw prompts beat chat templates for small models in general, and one sample of 200 messages does not settle that.

**Working rule:** Use one prompt for every model you compare, in the form the comparison needs. Before you adopt a chat template, measure it on the same sample as the raw form, and report a small gap as a small gap.

## Why free text fails {#why-free-text-fails}

Chapter 1 asked the raw model for one team with no constraint, and the model decoded freely: greedy, taking the highest-scoring token at each step, as in this chapter, but with no mask on the tokens it could pick. Its example counted the answers. 0/20 were team names, so the router had nothing to send on.

Nothing in the model's own decoding stops an answer from being a sentence, a team that is not on the list, or a near-miss spelling of one. At each step the decoder may pick any token in the vocabulary, so the answer can go anywhere. Chapter 1's stand-in did not show this failure, because its recorded answers came from a grammar that allowed only the team names. A model that runs in the router needs the same guarantee from the decoder itself.

The tempting repair is to take whatever the model wrote and match it to the nearest team name. That repair is a guess. A near-miss spelling may match the right team, or a wrong team that is close in letters, and the router would then send the message to a team the model never named. Chapter 1 said what to do instead: an answer that is not a team counts as an error, is counted on the sample, and the message goes to a person. A constraint removes most of those answers before they are written. The count is still kept, so that a constraint which fails to hold shows up as a number and not as a quiet change in routing.

**Working rule:** Treat an answer that is not a team as a failed route. Count such answers on a fixed sample before a model routes anything, and do not match a near-miss to the closest team. Constrain the answer at decode time, and keep the count, so that a broken constraint shows up as a number.

## A token trie allows only team names {#a-token-trie-allows-only-team-names}

A model writes tokens, not words, and a team name is a short run of them. `build_trie` in `router_lib`, the book's shared module, stores the 77 names as a trie: a tree whose paths spell out each team's tokens, with the team's index at the node where its path ends. The function asserts that no two teams end at the same node, so each path names one team.

The decoder walks this tree as it writes. At the start, the allowed tokens are the first token of each team name. After each token, the allowed set narrows to the children of the node the decoder has reached. Where a path has ended on a team, the end-of-answer token joins the allowed set, so the answer can stop there. No one of the 77 names is the start of another, so every path ends at a leaf. The code would still handle a name that was: at that node the decoder could stop, or continue along the longer name. A walk of this kind cannot leave the list, because every token it may choose leads toward a team.

Greedy decoding then takes the highest-scoring of the allowed tokens at each step. Transformers applies the rule through a callback, `prefix_allowed_tokens_fn`, which returns the allowed tokens for each position. The processor starts from a mask of minus infinity for every token, sets the allowed tokens' entries to zero, and adds the mask to the scores with `scores_processed = scores + mask`. Every token that is not allowed keeps minus infinity, so greedy decoding cannot pick it ([transformers 5.23.0, `src/transformers/generation/logits_process.py`](https://raw.githubusercontent.com/huggingface/transformers/v5.22.2/src/transformers/generation/logits_process.py)).

Chapter 1's stand-in got the same guarantee from its library. `recordings/common.py` builds a grammar, `root ::= "card_arrival" | ...`, that lists the 77 names as literal strings, and the recordings applied it through the chat template. llama-cpp-python adds a grammar to its sampler before the greedy step. In `llama_cpp/llama.py` the line `sampler.add_grammar(self._model, grammar)` runs under `if grammar is not None:`, and `sampler.add_greedy()` runs after it, under `elif temp == 0.0:` ([llama-cpp-python v0.3.36, `llama_cpp/llama.py`](https://raw.githubusercontent.com/abetlen/llama-cpp-python/v0.3.36/llama_cpp/llama.py)). So the stand-in's answers were limited to the names by the library, where this chapter's callback does the same job in the decoder.

Two details of that code shape the callback. The processor calls it as `self._prefix_allowed_tokens_fn(batch_id, sent)`, so the batch index comes first. And an empty list is an error: the processor raises `ValueError` when the callback returns no tokens. The book's callback therefore never returns an empty list. When no path can continue, it returns the end-of-answer token, and the answer stops there. The end-of-answer token is the model's `<|im_end|>`, which is also its pad token (id 2). The decoder stops after 32 new tokens (`max_new_tokens=32`). Every one of the 600 recorded answers ended on a team. A node with no children always holds a team, so the empty-list case is a guard and not a case the 77 names produce.

First, the 135M model with the trie, on those six messages.

Example `constrained-decode-135m` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Route the first 6 messages of test_40 with the 135M model, decoding only team names, and count how many right answers that gives.
The point rests on the count of answers that are team names, which the trie guarantees, and on the accuracy line, which the measurement section reads. The messages are read from the test split and are never printed."""

import os

# no network: the pinned model must already be in the local cache
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import collections  # noqa: E402

import torch  # noqa: E402

# two threads, so the sums add in the same order on every run
torch.set_num_threads(2)

import router_lib  # noqa: E402

# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats
router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"

tokenizer, model = router_lib.load_model(SMALL)
trie = router_lib.build_trie(tokenizer)
# (text, label index) for every test row, in file order; the index is the gold team
rows = router_lib.read_rows("test")
# row numbers of the first six messages of the fixed 40-message sample
sample = router_lib.read_sample("test_40", 6)
teams = set(router_lib.LABELS)

# (label index, team text); the index comes from the trie walk, not from parsing the text
results = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]
valid = [text in teams for _, text in results]
# the walk's index, never the decoded text, is what counts
right = [index == rows[i][1] for (index, _), i in zip(results, sample)]
given = collections.Counter(text for _, text in results)
# most given team; a tie in count goes to the team listed first
top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))

print(f"messages: {len(sample)}")
print(f"answers that are a team: {sum(valid)}/{len(sample)}")
print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")
print(f"distinct teams given: {len(given)}")
print(f"most given team: {top}, {given[top]}/{len(sample)}")
```

Output:

```text
messages: 6
answers that are a team: 6/6
accuracy: 1/6 (16.7%)
distinct teams given: 3
most given team: card_arrival, 4/6
```

Run on 6 messages, the model's answers are team names in every case, 6/6, which is the guarantee the trie gives. The accuracy on the same messages is 1/6 (16.7%). The [measurement section](#measure-the-constrained-answers) takes that second number apart.

The router does not read the team out of the text. `constrained_team` returns the index stored at the node where the walk ended, and the text is decoded only so that a reader can see it. The index is fixed by the trie and not by a parser that runs after the model. That is the reason the constraint is worth the code: a parser that matches text to teams can fail in ways the trie cannot. These examples return the team only so that it can be counted. Nothing in this chapter routes a message. A route returns a team only through `handoff`, which sends an uncertain answer to a person, and the examples here do not call it.

**Working rule:** Constrain the answer to the labels with a token trie built from the label names, and count the answers that are labels on a fixed sample. The constraint makes every answer a team. It does not make any answer right, so count the right answers as a separate measure.

## Decode greedily so every count repeats {#ties-and-determinism}

Every example here uses greedy decoding: the highest score wins, and nothing is sampled. That matters for the counts. A sampled answer would differ from one run to the next, and then no printed count could be checked against a recording. The examples fix their seeds and float32 through `router_lib.setup_runtime` and set two threads, so on one machine a run repeats its answers. Each example calls `torch.set_num_threads(2)` itself, at the top, so that it does not depend on how its runner was started. Each example's printed output is compared with its recorded output on every run, and a changed line fails the check.

Repeatable on one machine is not the same as repeatable everywhere. The [PyTorch 2.15.0+cpu notes on randomness](https://docs.pytorch.org/docs/2.15/notes/randomness.html) say that completely reproducible results are not guaranteed across releases, individual commits or platforms. So the book claims repeatability for the versions pinned in `examples/requirements.txt`, on the kind of machine the book ran on. A reader on another platform should expect to check the recorded lines again. If a recorded line fails there, read the changed line before touching the example. A different count of right answers can come from a different machine, a different library version or a changed prompt, and each of those needs a different fix.

Ties need a fixed rule as well. Label scoring in [the last section](#why-the-prior-wins) breaks a tie in score by the lower team index, through its sort key `(score, -index)`, and the most-given-team count breaks a tie in count by team name. Constrained decoding has no team scores. Its greedy step takes the first maximum among the allowed tokens, so a tie there goes to the lower token id.

**Working rule:** Use greedy decoding, fix the seeds and the thread count, and compare each printed line with a recorded copy on every run. Do not describe a result as reproducible beyond the versions and the kind of machine it was run on.

## Measure the constrained answers {#measure-the-constrained-answers}

The [live examples](#a-token-trie-allows-only-team-names) cover six messages. The 135M model with the raw prompt got 1/6 (16.7%) right, and the 360M model got 0/6 (0.0%). Six messages cannot rank these models: one right answer is the whole difference between those two counts. The 135M model's most common answer was card_arrival, 4/6, and the chat form on the same six messages got 0/6 (0.0%).

The 360M run is the same program with the larger model.

Example `constrained-decode-360m` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Decode team names for the first 6 messages of test_40 with the 360M model, and count how many of them are right.
The prose sets its accuracy line against the 135M model's, on the same six messages. The messages are read from the test split and are never printed."""

import os

# no network: the pinned model must already be in the local cache
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import collections  # noqa: E402

import torch  # noqa: E402

# two threads, so the sums add in the same order on every run
torch.set_num_threads(2)

import router_lib  # noqa: E402

# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats
router_lib.setup_runtime()

MEDIUM = "HuggingFaceTB/SmolLM2-360M-Instruct"

tokenizer, model = router_lib.load_model(MEDIUM)
trie = router_lib.build_trie(tokenizer)
# (text, label index) for every test row, in file order; the index is the gold team
rows = router_lib.read_rows("test")
# row numbers of the first six messages of the fixed 40-message sample
sample = router_lib.read_sample("test_40", 6)
teams = set(router_lib.LABELS)

# (label index, team text); the index comes from the trie walk, not from parsing the text
results = [router_lib.constrained_team(tokenizer, model, trie, rows[i][0]) for i in sample]
valid = [text in teams for _, text in results]
# the walk's index, never the decoded text, is what counts
right = [index == rows[i][1] for (index, _), i in zip(results, sample)]
given = collections.Counter(text for _, text in results)
# most given team; a tie in count goes to the team listed first
top = min(given, key=lambda team: (-given[team], router_lib.LABELS.index(team)))

print(f"messages: {len(sample)}")
print(f"answers that are a team: {sum(valid)}/{len(sample)}")
print(f"accuracy: {sum(right)}/{len(sample)} ({100 * sum(right) / len(sample):.1f}%)")
print(f"distinct teams given: {len(given)}")
print(f"most given team: {top}, {given[top]}/{len(sample)}")
```

Output:

```text
messages: 6
answers that are a team: 6/6
accuracy: 0/6 (0.0%)
distinct teams given: 4
most given team: card_payment_wrong_exchange_rate, 3/6
```

For the comparison, each variant answered 200 messages. The recording's counts are in the table, with the same messages in every row:

| Variant on test_200 | Answers outside the team list | Correct | Distinct teams given |
| --- | --- | --- | --- |
| 135M, raw prompt | 0/200 | 9/200 (4.5%) | 18 |
| 135M, chat template | 0/200 | 11/200 (5.5%) | 17 |
| 360M, raw prompt | 0/200 | 12/200 (6.0%) | 8 |

This example reads the table's counts from the recording:

Example `prompting-on-test-200` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Compare the three prompting variants on test_200, from the recording only. No model is loaded.
The chapter's table takes its three columns from the first three lines printed for each variant, and the point rests on the correct count.

recordings/prompting_eval.py wrote one answer per variant and message to examples/data/recorded/prompting-eval/answers.jsonl.
"""

import collections
import json
from pathlib import Path

import torch

# two threads, as in the other examples
torch.set_num_threads(2)

import router_lib  # noqa: E402

# seeds 0 for random and torch, deterministic algorithms and float32, as in the other examples
router_lib.setup_runtime()

ANSWERS = Path(__file__).resolve().parent / "data" / "recorded" / "prompting-eval" / "answers.jsonl"
teams = set(router_lib.LABELS)
# (text, label index) for every test row, in file order; the index is the gold team
rows = router_lib.read_rows("test")
# the 200 messages of the fixed sample; every variant answers all of them
total = len(router_lib.read_sample("test_200"))

by_variant = collections.defaultdict(list)
with open(ANSWERS) as f:
    for line in f:
        answer = json.loads(line)
        by_variant[answer["variant"]].append(answer)

for variant in sorted(by_variant):
    answers = by_variant[variant]
    outside = sum(answer["team"] not in teams for answer in answers)
    # answer["label"] is the team index the variant chose; rows[...][1] is the message's gold index
    right = sum(answer["label"] == rows[answer["index"]][1] for answer in answers)
    given = collections.Counter(answer["team"] for answer in answers)
    # most given team; a tie in count goes to the team name that sorts first, not to the index
    top = min(given, key=lambda team: (-given[team], team))
    # the gold index becomes a team name, to compare with the most given team
    gold_top = sum(router_lib.LABELS[rows[answer["index"]][1]] == top for answer in answers)
    print(f"{variant} answers outside the team list: {outside}/{total}")
    print(f"{variant} correct: {right}/{total} ({100 * right / total:.1f}%)")
    print(f"{variant} distinct teams given: {len(given)}")
    print(f"{variant} most given team: {top}, {given[top]}/{total}")
    print(f"{variant} gold for most given team: {gold_top}/{total}")
```

Output:

```text
135m-chat answers outside the team list: 0/200
135m-chat correct: 11/200 (5.5%)
135m-chat distinct teams given: 17
135m-chat most given team: topping_up_by_card, 55/200
135m-chat gold for most given team: 1/200
135m-raw answers outside the team list: 0/200
135m-raw correct: 9/200 (4.5%)
135m-raw distinct teams given: 18
135m-raw most given team: card_arrival, 165/200
135m-raw gold for most given team: 0/200
360m-raw answers outside the team list: 0/200
360m-raw correct: 12/200 (6.0%)
360m-raw distinct teams given: 8
360m-raw most given team: card_payment_wrong_exchange_rate, 94/200
360m-raw gold for most given team: 4/200
```

No row is close to a router, though the constraint holds in every one: no answer in any row is outside the team list. The 360M raw row has the most right answers, at 12/200 (6.0%), and no row beats another by more than a few messages.

Distinct teams alone mislead. The 135M raw row gives 18 distinct teams, against 8 for the 360M raw row, but most of the 135M row's answers go to one team, card_arrival, 165/200, while the 360M raw row's most given team gets only card_payment_wrong_exchange_rate, 94/200.

To read a row, start with the first column. If it is not zero, the constraint has not held, and the other two columns mean nothing until that is fixed. The second column is the one that matters for routing: it counts the messages whose team is the gold label, out of the whole sample, with the wrong answers counted as wrong. The third column says how many teams the variant uses at all. A variant that gives only a few teams is either guessing a few popular teams or has learned something narrow, and the count does not say which. A large count does not rule out concentration either, as the 135M raw row shows. A reader who runs the same three counts on a sample of their own messages gets a comparison on their own data, with the same meaning in each column.

The 360M model is the larger of the two, and [chapter 3](choose-a-base-model) counted how much larger it is. The table shows what that cost buys on this prompt and this constraint: 12/200 (6.0%) right, against 9/200 (4.5%) for the 135M model on the same raw form. That is a few more messages for a larger model, and the chapter does not count it as a reason to pay for the extra size.

The book computes no interval for these counts, and a gap of a few messages out of 200 could come from the sample, so the table supports one statement: the three variants are close, and none of them is a router. A reader who wants to choose between two prompts should run both on a sample of their own messages and count the same three things.

The trie checks the shape of the answer and nothing else. It does not read the message beyond the prompt, and it does not know which team is right. Any team that is on the list is allowed at every step where its name could still be written, so the decoder's choice among them is the model's choice, and the mask only decides which teams can be written at all.

Here is one valid wrong answer, described by its teams only, since messages are never quoted in this book. One test message whose gold team is `card_not_working` got `card_arrival` from the 135M raw row, a team on the list, so the constraint let it through and the count of answers outside the list stays at zero.

The lesson the outputs support is that constrained decoding makes every answer a team, but a valid answer is not a right one. The constraint only removes answers that are not teams. It does not choose a better team, and the prompt change did not either. Chapter 5 is the next step: it trains a head on the labelled messages and measures whether the trained model routes better on the same sample.

In production the failure does not show up as an error. The count of answers that are not teams stays at zero, so a check that watches only that count stays green. Only a count of right answers on a fixed, labelled sample shows the problem.

The cost falls on the people who receive the message. A valid wrong team sends the message to the wrong destination, where support staff see a request that is not theirs, and they must find the right team and move it, which is the same work a hand-off creates, with no sign on the message that it was misrouted. A router that cannot say when it is unsure gives those people nothing to act on. Constrained decoding also returns no score for the other 76 teams, so a hand-off rule built on it would have no per-team scores to compare.

**Working rule:** Report the count of right answers beside the count of valid answers, on one fixed sample, for every variant you compare. A router whose answers are all valid can still be wrong on most messages, and only the second count shows it.

## Why a prior is not a router {#why-the-prior-wins}

Constrained decoding returns one answer and no scores. Label scoring gives every team a score. For each team, label scoring sums the log-probability of the team's tokens after the prompt, where a log-probability is the logarithm of the probability the model gives a token. The team with the highest sum is the model's top team for the message.

Run it on three messages and read the top team of each:

Example `label-scoring-prior` (Python 3.12.15 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Score every team for the first 3 messages of test_40 with the 135M model, by the summed log-probability of its tokens.
This shows the model's prior, the team it favours before any fine-tuning. The point rests on the most frequent top team and how many messages it tops. The messages are never printed."""

import os

# no network: the pinned model must already be in the local cache
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import collections  # noqa: E402

import torch  # noqa: E402

# two threads, so the sums add in the same order on every run
torch.set_num_threads(2)

import router_lib  # noqa: E402

# seeds 0 for random and torch, deterministic algorithms and float32, so greedy decoding repeats
router_lib.setup_runtime()

SMALL = "HuggingFaceTB/SmolLM2-135M-Instruct"

tokenizer, model = router_lib.load_model(SMALL)
rows = router_lib.read_rows("test")
# row numbers of the first three messages of the fixed 40-message sample
sample = router_lib.read_sample("test_40", 3)
# each team's token ids, padded to the longest name, built once for all messages
table = router_lib.label_token_table(tokenizer)

tops = []
for i in sample:
    scores = router_lib.label_scores(tokenizer, model, table, rows[i][0])
    # index of the highest-scoring team; a tie goes to the lower index
    tops.append(max(range(len(scores)), key=lambda k: (scores[k], -k)))

counts = collections.Counter(tops)
# most frequent top team; a tie in count goes to the lower index
best = min(counts, key=lambda k: (-counts[k], k))

print(f"messages: {len(sample)}")
print(f"distinct top teams: {len(counts)}")
print(f"most frequent top team name: {router_lib.LABELS[best]}")
print(f"most frequent top team: {counts[best]} of {len(sample)}")
```

Output:

```text
messages: 3
distinct top teams: 1
most frequent top team name: card_arrival
most frequent top team: 3 of 3
```

Before any training, the model puts one team first for every message: card_arrival tops 3 of 3 messages. That team is the model's prior, and three messages are too few to say more about it.

If the top team is the same for every message, the score is the model's prior and says little about any one message. The 200-message recording shows the same prior in constrained decoding: card_arrival, 165/200, the 135M raw row's favourite, is the gold team of 0/200 messages in the sample.

The sum favours short names. Each token of a team's name adds a log-probability, which is never above zero, so a team with more tokens has more terms to pull its total down. The example adds the terms without dividing by length, and the chapter does not claim to correct for it. A reader who uses label scoring should check the top teams against a sample with gold labels before trusting them.

Label scoring does give a score for each team, which is what a hand-off rule needs. Each message is scored against all 77 team names, which the code scores in one batch after the prompt runs once. Chapter 5 gets a score for each team from a single forward pass through a head that is trained on labelled messages. Label scoring is the bridge from this chapter to that one: it shows that the model already has per-team scores, and that before training they say little about the message.

**Working rule:** Use label scoring to see which team a model favours before training, and do not route with it. Do not route messages with a prompted model of this size on this task; the 200-message recording is the evidence. Train a head on the labelled messages instead, as [chapter 5](fine-tune-with-lora) does.

## What this chapter decided {#what-this-chapter-decided}

The rules in this chapter are scoped to these two models, this prompt and this constraint, on the 200 messages of the recording. A larger model, a different prompt or a longer sample could move the counts, and the book has not tested any of them on that sample. What the chapter does establish is narrower: with the raw prompt and the trie, a model of this size returns valid teams and few right ones, and the count of right ones is the number to watch.

Each decision is tested in its own section: the prompt in the first, the trie in [its section](#a-token-trie-allows-only-team-names) and greedy decoding in [its section](#ties-and-determinism), and the two counts in the measurement section. A reader who changes any of them must record again before comparing, as the first section says. A constraint makes every answer valid, and only training can make it right.

This chapter made these decisions:

- The router prompt is the raw form, and both models in the comparison use it.
- Constrained decoding uses a token trie over the 77 team names, with greedy decoding and fixed seeds.
- Every variant is reported with its count of valid answers and its count of right answers, on one fixed sample.
- The live examples show how each piece works, and only the 200-message recording compares models.

# Fine-tune a classification head with LoRA

By the end of this chapter, you can train a classification head on the labelled messages with LoRA, count the parameters that training changes, choose a hand-off threshold on the dev split by a stated rule, and record that threshold beside the results it produced.

The router so far runs the [stand-in](route-a-message#ask-a-model-for-one-team), which writes its answer as text in place of a hosted model, on a placeholder threshold, 0.5, that no one chose and every output labels as one. The answer has to be a team, and the threshold has to be chosen, so this chapter replaces that answer with a head that returns a team and a confidence in one forward pass, and it chooses the threshold on dev before any test result is reported. Training needs labelled messages for the router's teams, and these come from [BANKING77](measure-the-routing-task#the-data-and-its-licence), a public dataset. [Chapter 3](choose-a-base-model) chose SmolLM2-135M-Instruct as the base model, and this chapter fine-tunes that model and no other: the 360M stays a comparison, as chapter 3 said. The router is measured on the [samples](measure-the-routing-task#dev-test-and-fixed-samples) that chapter 2 fixed.

This chapter does not compare the adapter with the 360M, and it does not distill or quantize anything. Chapters 6, 7 and 9 distill, quantize and version the adapter.

## A head that scores the teams {#a-head-that-scores-the-teams}

The stand-in writes its answer as text, and a grammar limits that text to the 77 team names while it decodes ([chapter 4](prompt-and-constrain#a-token-trie-allows-only-team-names)). A classification head works differently from the stand-in: it keeps the language model's layers and adds one output layer with 77 outputs, one for each team. For each message, the model produces 77 scores. Softmax turns the scores into 77 probabilities that sum to one. The router returns the team with the highest probability, and the confidence is that probability.

The answer is an index into the 77 teams, so it is always one of them. [Chapter 4](prompt-and-constrain) reaches the same guarantee for a prompted model by constraining its text. This head never writes text, so it has nothing to constrain.

A message costs one forward pass: one run of the network from input to scores, with no loop that writes tokens one at a time. The team and the confidence come from that same pass. This matters for the hand-off. The flag that sends a message to a person is computed from the same probabilities as the team it protects, so the two cannot drift apart between two calls. A separate scoring call would need its own check that the two agree.

The confidence here has a narrower meaning than the stand-in's. In chapter 1, a confidence was the probability the stand-in's recorded answer had, which is not a softmax over teams. In this chapter, the confidence is the softmax probability of the chosen team. The two numbers are both called confidence, and both are compared with a threshold, but they are not the same quantity, so a threshold chosen for one is not a threshold for the other.

When two teams tie, `router_lib`'s `classify` function takes the lower label index. That follows from torch's `argmax`, whose documentation says: "If there are multiple maximal values then the indices of the first maximal value are returned." ([torch 2.15.0+cpu documentation](https://docs.pytorch.org/docs/2.15/generated/torch.argmax.html)). A fixed rule for ties keeps the output reproducible.

**Working rule:** Read the team and the confidence from one forward pass, and take the team as the output with the highest probability. Keep both in one function, so that a hand-off decision and the team it protects come from the same answer. Do not set a head's confidence beside a stand-in's as though they measured the same thing.

## Adapters train a small part of the model {#adapters-train-a-small-part-of-the-model}

Full fine-tuning changes every weight in the base model. LoRA takes a different route. The paper that introduced LoRA, [Hu et al. (2021)](https://arxiv.org/abs/2106.09685), says it freezes the pre-trained weights and injects trainable low-rank matrices into the layers of the Transformer, and the trained LoRA matrices are the adapter. The base model stays as it was, and the adapter is the part this book stores, loads and versions.

The rank, `r`, is the inner size of each added matrix, and it is kept small. Peft scales the added update by `lora_alpha` divided by `r` ([peft 0.21.3, `src/peft/tuners/lora/layer.py`](https://raw.githubusercontent.com/huggingface/peft/v0.21.2/src/peft/tuners/lora/layer.py), line 281: `self.scaling[adapter_name] = lora_alpha / r`), so `lora_alpha` sets how strongly the update counts. This book uses a rank of 8, with `lora_alpha` set to 16. The added matrices go in the projection layers named `q_proj` and `v_proj` in each attention block. The abstract does not name those layers, so this choice is the book's, not something the paper prescribes. We recommend it as a starting point, and the recorded adapter below is the test of that recommendation on this task. Other targets are a choice to measure, not a fact about the method.

The head is trained as well. Peft does that for this book's setup: a model built as a sequence classifier lists its `classifier` and `score` layers in `modules_to_save` ([peft 0.21.3, `src/peft/peft_model.py`](https://raw.githubusercontent.com/huggingface/peft/v0.21.2/src/peft/peft_model.py), line 1830: `classifier_module_names = ["classifier", "score"]`), so training moves the 77-output layer and the adapter together. Count only the adapter's matrices and you get a smaller number than the one the code trains, and the difference is the head. The next section's run counts the split from the loaded model: `460800` in the adapter and `44352` in the head, which together make `505152`.

The adapter's manifest records the base model's revision, so a reader can check that revision against `models.toml` before scoring.

**Working rule:** Set the rank, the scaling and the target layers in one configuration, and count the trainable parameters from the loaded model before training starts. Print the total with the head in it. A count that leaves out the head reports part of what training changes, and it will not match what the optimizer updates.

## Thirty steps are not enough {#thirty-steps-are-not-enough}

Before the book uses a recording, it shows what a short run does with the same recipe. The run below takes 30 steps, at batch size 8 and seed 0, on the training rows outside dev, and then scores `test_200`, the first 200 entries of the [fixed sample](measure-the-routing-task#dev-test-and-fixed-samples) that chapter 2 drew. It is a failure, and the book shows it as one. The short run checks that the pipeline works from start to finish, and that is its only job here.

Example `lora-thirty-steps` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Train for 30 steps on the recorded recipe, then score test_200 at the placeholder threshold of 0.5.
Read the three step losses against the uniform-guess loss, then the hand-off line at that threshold. The recording runs 1,125 steps."""

import math

import torch

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
REPO = "HuggingFaceTB/SmolLM2-135M-Instruct"
STEPS, BATCH, SEED = 30, 8, 0

train, order = router_lib.training_rows(SEED)  # the seed sets the shuffled order of the training rows, the one the recording used
tok, model = router_lib.new_classifier(REPO)  # SmolLM2 with a 77-way head, LoRA rank 8, lora_alpha 16, on q_proj and v_proj
lora, head = router_lib.parameter_split(model)  # LoRA tensors have "lora_" in their names; every other trainable tensor is the head
print(f"trainable parameters: {lora + head}")
print(f"LoRA parameters: {lora}")
print(f"head parameters: {head}")

losses = router_lib.train_steps(model, tok, train, order, STEPS, BATCH)  # AdamW at learning rate 1e-3, router_lib's default
print(f"training steps: {STEPS}")
print(f"batch size: {BATCH}")
print(f"seed: {SEED}")
for step in (10, 20, 30):
    print(f"loss at step {step}: {losses[step - 1]:.2f}")  # train_steps returns one loss per step, from step 1 at index 0, so step 10 is index 9
print(f"loss of a uniform guess over 77 teams: {math.log(77):.2f}")

result = router_lib.score_test_sample(model, tok, "test_200", 0.5)  # 0.5 is chapter 1's placeholder; this run chooses no threshold
n = result["n"]
print(f"messages on test_200: {n}")
print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")
print(f"handed off at placeholder 0.50: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")
if result["accuracy_not_handed_off"] is None:  # None when no message is kept, so there is no accuracy to print
    print("accuracy on messages not handed off: not defined (0 messages not handed off)")
else:
    m = result["not_handed_off"]
    print(f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} ({result['accuracy_not_handed_off']:.1%})")
```

Output:

```text
trainable parameters: 505152
LoRA parameters: 460800
head parameters: 44352
training steps: 30
batch size: 8
seed: 0
loss at step 10: 4.13
loss at step 20: 6.12
loss at step 30: 4.38
loss of a uniform guess over 77 teams: 4.34
messages on test_200: 200
accuracy on test_200: 5/200 (2.5%)
handed off at placeholder 0.50: 200/200 (100.0%)
accuracy on messages not handed off: not defined (0 messages not handed off)
```

Thirty steps are not enough to route with. The head this run trains gets 5/200 (2.5%) right on `test_200`, and at the placeholder threshold it hands every message to a person: 200/200 (100.0%). A router that hands off every message gives no message to a team, so the labels it predicts never reach a caller. Because no message is kept, the accuracy on messages not handed off is not defined, as the output says.

The loss does not fall steadily across the three printed steps: it is 4.13 at step 10, 6.12 at step 20, and 4.38 at step 30. Step 20 is higher than step 10, and step 30 is lower than step 20 but higher than step 10. Three points from 30 steps are not a learning curve. The loss at step 30 sits close to the uniform guess of 4.34, so at that point the head is still close to chance. A uniform guess gives each team the same probability, one in 77, so its loss is the natural log of 77. The recording used in the section on the recorded adapter runs the same recipe for far more steps, and its accuracy is the one this book reports.

Scoring `test_200` here only reports the failure, and nothing is chosen from it: the recipe is fixed before the run, and the threshold is chosen later on dev. To see whether training works, read the loss and the dev measures, not the test sample, which is the rule chapter 2 set for every sample. Tuning a recipe against the test sample leaves no clean number to report, and the sample stops being a test.

**Working rule:** Run a short training to check that the pipeline works from start to finish, and do not route with its head. Judge a fine-tuned head on the dev split, with the measures set in chapter 2, after a run long enough to reach them. Do not read a loss curve of a few points as evidence that the head learns.

## Train on the training rows, never on the test rows {#train-on-the-training-rows-never-on-the-test-rows}

The training file holds labelled messages, and chapter 2 drew the dev split from it, team by team, with a fixed seed. Training uses the training rows that are not held back for dev. The test file is never read during training. The training order in [thirty steps are not enough](#thirty-steps-are-not-enough) is built from the rows left after dev is removed, shuffled with seed 0, so it is the same on every run.

The dev split carries the decisions. It holds 1003 messages from `train.csv`, the training file. No training step reads them, and they are used only to choose the threshold and to check the calibration in the last section.

The recipe is written into the adapter's manifest, so a reader can see what the adapter was trained on: the base model's pinned revision, the data commit, the batch size, the seed and the number of steps. Repeating the training takes those values, and checking that a result came from a clean split takes the data commit.

The rule matters most when a result looks good. A training run that reads the test sample has seen the messages it is then scored on, so its test accuracy measures nothing about new messages. Keeping dev and test apart is what makes the test accuracy in the next section a fair report.

**Working rule:** Train only on training rows that are outside dev. Record the seed, the batch size, the step count and the data commit in the manifest. When dev is too small to decide a threshold, draw more messages from the training file, never from the test file, as chapter 2 says.

## Use the recorded adapter {#use-the-recorded-adapter}

The working router uses `banking77-lora-v1`: the adapter from the full recorded run, version 1, written by `recordings/lora_full.py`. That script trained the adapter once, on the recipe in [the section on training rows](#train-on-the-training-rows-never-on-the-test-rows), and wrote its files and a manifest. The manifest holds the base model's revision, the data commit, the threshold, the recipe's hash and the hashes of the output files. Offline, the adapter loads and scores `test_200` at the threshold from its manifest, and nothing is trained.

Take the adapter as it was recorded. A different recipe becomes a new run, named `v2`; the book does not edit the adapter by hand. Keeping the recording fixed is what lets a reader check the numbers in this chapter against the files they downloaded.

Example `lora-recorded-adapter` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Load the recorded adapter banking77-lora-v1 and score test_200 at the threshold in its manifest; no training runs here.
Read the accuracy line with the two hand-off lines printed under it."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
manifest = json.loads((RECORDED / "manifest.json").read_text())
threshold = manifest["handoff_threshold"]  # the threshold the recording chose on dev; this script chooses none

tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")  # the base model comes from the adapter's config, at its revision in models.toml
test = router_lib.read_rows("test")
sample = router_lib.read_sample("test_200")
golds = [test[i][1] for i in sample]
preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])  # a tie goes to the lower label index

result = router_lib.metrics(preds, golds, confs, threshold)  # a message is handed off when its confidence is below the threshold
n = result["n"]
print(f"adapter: {manifest['adapter']}")
print(f"handoff threshold from manifest: {threshold:.2f}")
print(f"messages on test_200: {n}")
print(f"accuracy on test_200: {result['correct']}/{n} ({result['accuracy']:.1%})")
print(f"handed off: {result['handed_off']}/{n} ({result['handoff_rate']:.1%})")
if result["accuracy_not_handed_off"] is None:  # None when no message is kept, so there is no accuracy to print
    print("accuracy on messages not handed off: not defined (0 messages not handed off)")
else:
    m = result["not_handed_off"]
    print(
        f"accuracy on messages not handed off: {result['correct_not_handed_off']}/{m} "
        f"({result['accuracy_not_handed_off']:.1%})"
    )
```

Output:

```text
adapter: banking77-lora-v1
handoff threshold from manifest: 0.63
messages on test_200: 200
accuracy on test_200: 172/200 (86.0%)
handed off: 33/200 (16.5%)
accuracy on messages not handed off: 156/167 (93.4%)
```

The recorded adapter is right on 172/200 (86.0%) of the messages, against 9/200 (4.5%) for the prompted 135M model that [chapter 4](prompt-and-constrain) measured on the same 200. The 30-step run, at 5/200 (2.5%), falls below the prompted model. A handed-off message counts as wrong when its team is wrong. The output beside this paragraph gives the hand-off count and the accuracy on messages not handed off, each with its count and denominator, as chapter 2 requires.

The threshold is not chosen on this sample. It came from the manifest, which [the section on choosing the threshold](#choose-the-threshold-on-dev) explains. It is the threshold the recorded adapter's hand-off counts on `test_200` use: 0.63. Reading it from the manifest keeps the choice on dev, where the rule made it.

The recorded adapter is the router's model, and the stand-in is the baseline. The two were measured with different thresholds, so their hand-off counts are not the same kind of result. Compare them on the same messages, with each threshold named beside its count. The comparison below does that for `test_200`, reading the stand-in's recorded answers for the same messages.

Example `compare-with-standin` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Set the stand-in's recorded answers beside the recorded adapter's live predictions on the same 200 test messages.
The last two lines count where the models disagree."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402
import math  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
STANDIN_THRESHOLD = 0.5  # chapter 1's placeholder; the stand-in's own threshold is never chosen on dev
ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"

with open(router_lib.HERE / "data" / "recorded" / "standin-baseline" / "answers.jsonl") as f:
    answers = {row["index"]: row for row in map(json.loads, f)}  # keyed by the test-row index that read_sample returns
threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]

test = router_lib.read_rows("test")
sample = router_lib.read_sample("test_200")
golds = [test[i][1] for i in sample]
standin_preds = [answers[i]["label"] for i in sample]
standin_confs = [math.exp(answers[i]["logprob"]) for i in sample]  # the recording stores log-probabilities; exp turns one back into the probability its answer had

tokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")
adapter_preds, adapter_confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])

standin = router_lib.metrics(standin_preds, golds, standin_confs, STANDIN_THRESHOLD)
adapter = router_lib.metrics(adapter_preds, golds, adapter_confs, threshold)
n = standin["n"]
s_right = [p == g for p, g in zip(standin_preds, golds)]
a_right = [p == g for p, g in zip(adapter_preds, golds)]

print(f"messages on test_200: {n}")
print(f"stand-in correct: {standin['correct']}/{n} ({standin['accuracy']:.1%})")
print(f"adapter banking77-lora-v1 correct: {adapter['correct']}/{n} ({adapter['accuracy']:.1%})")
print(f"stand-in placeholder threshold: {STANDIN_THRESHOLD:.2f}")
print(f"stand-in handed off: {standin['handed_off']}/{n} ({standin['handoff_rate']:.1%})")
print(f"adapter threshold from manifest: {threshold:.2f}")
print(f"adapter handed off: {adapter['handed_off']}/{n} ({adapter['handoff_rate']:.1%})")
print(f"right for the adapter, wrong for the stand-in: {sum(a and not s for a, s in zip(a_right, s_right))}/{n}")
print(f"right for the stand-in, wrong for the adapter: {sum(s and not a for a, s in zip(a_right, s_right))}/{n}")
```

Output:

```text
messages on test_200: 200
stand-in correct: 109/200 (54.5%)
adapter banking77-lora-v1 correct: 172/200 (86.0%)
stand-in placeholder threshold: 0.50
stand-in handed off: 9/200 (4.5%)
adapter threshold from manifest: 0.63
adapter handed off: 33/200 (16.5%)
right for the adapter, wrong for the stand-in: 69/200
right for the stand-in, wrong for the adapter: 6/200
```

Matching the two models message by message shows which messages each one misses. The stand-in's answers come from its recording, and the adapter runs live on the same 200 messages. The stand-in's confidence is the probability its recorded answer had, from the recording's log-probability. The stand-in's threshold is chapter 1's placeholder, 0.50, and the output labels it as one. Its hand-off count, 9/200 (4.5%), says what that placeholder does to the stand-in, and nothing about the stand-in's own threshold, which was never chosen on dev. The adapter hands off 33/200 (16.5%) at 0.63.

The adapter is right on 172/200 (86.0%) of these messages, the stand-in on 109/200 (54.5%). Where the two disagree, the adapter alone is right on 69/200, and the stand-in alone on 6/200. Two accuracies can be equal while the two models miss different messages, and the disagreement counts show where that happens.

The comparison is one sample of 200 messages. [Chapter 1](route-a-message#score-the-router-on-a-fixed-sample) said that its small sample can show that a router fails, but cannot tell two routers apart with confidence. Chapter 1's sample was smaller. The discordant counts here, 69/200 against 6/200, are stronger evidence than that sample could give. For a firmer claim, run the same comparison on the full `test_1001` sample, which the stand-in's recording also covers.

**Working rule:** Compare two models only on the same messages, and name the threshold each one used beside its count. When one threshold is a placeholder, say so beside its count, and do not read its hand-off count as the model's own result.

## Choose the threshold on dev {#choose-the-threshold-on-dev}

The threshold is the cut that sends a message to a person. Chapter 2 set its measures, and chapter 1 left it as a placeholder. The book's rule chooses it on the dev split only: the lowest confidence threshold, on a grid of 0.01, at which accuracy on messages not handed off is at least the target of 0.90. The rule is written in the manifest with its value, so a reader can see the rule as well as the number it produced. The test sample plays no part, [as chapter 2 requires](measure-the-routing-task#dev-test-and-fixed-samples).

Example `choose-threshold-on-dev` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Find the hand-off threshold on dev by the manifest's rule, then check it against the manifest's value.
Read the chosen threshold beside the three threshold lines. test_200 is not read here."""

import json

import torch

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
manifest_threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]
TARGET = 0.90  # the book's target accuracy on messages not handed off: a choice the book makes, not one the data sets

tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")
rows = router_lib.read_rows("train")
dev = router_lib.read_sample("dev")
golds = [rows[i][1] for i in dev]
preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev])  # batches by length to pad less; results come back in dev order
n = len(dev)
chosen = router_lib.choose_threshold(preds, golds, confs, TARGET)  # the rule lives in router_lib: the lowest 0.01 grid value that meets TARGET; None if no value meets it, and the prints below would then fail

print(f"dev messages: {n}")
print(f"target accuracy on messages not handed off: {TARGET:.2f}")
print(f"chosen threshold: {chosen:.2f}")
print(f"threshold in manifest: {manifest_threshold:.2f}")
print(f"chosen threshold matches manifest: {'yes' if chosen == manifest_threshold else 'no'}")  # both come from the same 0.01 grid, so equal means the same threshold
for t in sorted({0.50, chosen, 0.90}):  # 0.50 is chapter 1's placeholder; a set drops the chosen value if it is one of the other two
    r = router_lib.metrics(preds, golds, confs, t)
    print(f"threshold {t:.2f} handed off: {r['handed_off']}/{n} ({r['handoff_rate']:.1%})")
    print(f"threshold {t:.2f} accuracy on messages not handed off: {r['correct_not_handed_off']}/{r['not_handed_off']} ({r['accuracy_not_handed_off']:.1%})")
```

Output:

```text
dev messages: 1003
target accuracy on messages not handed off: 0.90
chosen threshold: 0.63
threshold in manifest: 0.63
chosen threshold matches manifest: yes
threshold 0.50 handed off: 80/1003 (8.0%)
threshold 0.50 accuracy on messages not handed off: 811/923 (87.9%)
threshold 0.63 handed off: 148/1003 (14.8%)
threshold 0.63 accuracy on messages not handed off: 770/855 (90.1%)
threshold 0.90 handed off: 352/1003 (35.1%)
threshold 0.90 accuracy on messages not handed off: 620/651 (95.2%)
```

Scoring dev with the recorded adapter and applying the rule gives a threshold of 0.63. The check against the manifest's value prints yes, and a mismatch would mean the manifest and the rule disagree.

Three thresholds show the trade-off, and the table below sets them side by side, with each count and rate. Raising the threshold never hands off fewer messages, and the rule picks the lowest value that meets the target. Each row is the same adapter on the same dev messages, so only the threshold changes.

| Threshold | Handed off, on dev | Accuracy on messages not handed off, on dev |
| --- | --- | --- |
| 0.50 | 80/1003 (8.0%) | 811/923 (87.9%) |
| 0.63, chosen | 148/1003 (14.8%) | 770/855 (90.1%) |
| 0.90 | 352/1003 (35.1%) | 620/651 (95.2%) |

At 0.50, the router keeps more messages, but its kept messages are less accurate, and the target is missed. At 0.90, the kept messages are the most accurate of the three, but the most dev messages go to a person. The chosen value is the lowest of the three thresholds that meets the target, so it hands off more messages than 0.50 and fewer than 0.90. The table shows what it costs in hand-offs and what it buys in accuracy. Where to set the target is a business choice the rule cannot make. The book uses 0.90 accuracy on kept messages, and a bank running its own router would read its own target off this trade.

The adapter stays the same in production, but the messages it sees do not. Suppose a product problem sends many messages about one team in one week. The threshold was chosen on the dev mix of messages, so the hand-off rate can move on its own while the adapter stays exactly the same. A rise in hand-offs with no change to the adapter is the sign to check the mix of messages first. Moving the threshold by eye until the hand-off rate looks right gives up the property the rule was chosen for: at least the target accuracy on the dev messages the router keeps.

The confidence is not a probability that the team is right. [Guo et al. (2017)](https://arxiv.org/abs/1706.04599) report that their networks are "poorly calibrated", meaning their confidence does not match how often they are right. This book checks its own adapter on dev. The dev messages fall into five confidence bands, and each band gets a count, an accuracy and a mean confidence, the two quantities Guo et al. compare when they measure calibration. Grouping dev by confidence shows whether a 0.9 means nine in ten.

Example `calibration-bands-on-dev` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Group the dev messages into five bands by the adapter's confidence, and print each band's count, accuracy and mean confidence.
Read the top band's mean confidence against its accuracy. Only dev is read."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
BANDS = [(0.0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)]

tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")
rows = router_lib.read_rows("train")
dev = router_lib.read_sample("dev")
golds = [rows[i][1] for i in dev]
preds, confs = router_lib.classify_by_length(model, tokenizer, [rows[i][0] for i in dev])  # results come back in dev order, so index i matches golds[i]

print(f"dev messages: {len(dev)}")
counted = 0
for lo, hi in BANDS:
    last = hi == 1.0
    members = [i for i, c in enumerate(confs) if lo <= c < hi or (last and c == 1.0)]  # lower edges are in, upper edges out; the top band also takes 1.0, so each message is counted once
    counted += len(members)
    name = f"band {lo:.1f}-{hi:.1f}"
    print(f"{name} messages: {len(members)}")
    if members:
        correct = sum(preds[i] == golds[i] for i in members)
        print(f"{name} correct: {correct}/{len(members)} ({correct / len(members):.1%})")
        print(f"{name} mean confidence: {sum(confs[i] for i in members) / len(members):.1%}")
    else:
        print(f"{name} correct: not defined (0 messages)")
        print(f"{name} mean confidence: not defined (0 messages)")
print(f"messages in bands: {counted}")
```

Output:

```text
dev messages: 1003
band 0.0-0.2 messages: 1
band 0.0-0.2 correct: 1/1 (100.0%)
band 0.0-0.2 mean confidence: 19.9%
band 0.2-0.4 messages: 41
band 0.2-0.4 correct: 10/41 (24.4%)
band 0.2-0.4 mean confidence: 32.9%
band 0.4-0.6 messages: 92
band 0.4-0.6 correct: 49/92 (53.3%)
band 0.4-0.6 mean confidence: 51.0%
band 0.6-0.8 messages: 123
band 0.6-0.8 correct: 84/123 (68.3%)
band 0.6-0.8 mean confidence: 70.6%
band 0.8-1.0 messages: 746
band 0.8-1.0 correct: 693/746 (92.9%)
band 0.8-1.0 mean confidence: 96.4%
messages in bands: 1003
```

Most dev messages fall in the top band, from 0.8 to 1.0, which holds 746 of them. The bottom band, from 0.0 to 0.2, holds 1, too few for its accuracy to say anything about low confidence.

On dev the confidence orders the messages usefully, but at the top it runs above the accuracy: the top band's mean confidence is 96.4%, against an accuracy of 693/746 (92.9%). Four of the five bands have an accuracy inside their range, and the bottom band is the one exception, with its single message. From the second band up, accuracy rises. In the two middle bands, accuracy and mean confidence sit close together, while in the band from 0.2 to 0.4 the mean confidence is well above the accuracy.

The bands cannot show more: the band from 0.2 to 0.4 holds 41 messages. Every number here comes from the one dev sample that also chose the threshold, so none of them shows how calibrated the adapter will be on new messages.

The router the rule chooses is an empirical cut on dev, recorded with its rule and data. It is a cut, not a model of the messages, so its numbers describe the dev mix. A reader takes its hand-off rate and accuracy on kept messages as the router's result, and re-checks the dev rule when the mix of messages moves.

**Working rule:** Choose the threshold on the dev split by the stated rule, and write its value and the rule into the adapter's manifest. Report the hand-off rate beside accuracy on the messages kept, each with its count, and show at least one setting either side of the chosen value. Check calibration on dev before you trust the confidence as a cut, and when the mix of messages changes, choose the threshold again on a fresh dev draw from the training file.

# Distill a larger model's answers into a small one

By the end of this chapter, you can train a small student on the labels a larger teacher gives your training rows, measure how often those labels agree with the gold labels, and decide from the same messages whether the student is worth using beside a student trained on gold labels.

A router learns to name the team from labelled messages. This chapter asks whether a larger model can do that labelling for the training rows, so that a small model learns from its answers. The gold labels come from [BANKING77](measure-the-routing-task#the-data-and-its-licence). [Chapter 5](fine-tune-with-lora) trained a head on the gold labels of the training rows, and the adapter it saved, `banking77-lora-v1`, is the human-labelled adapter. Its test result is the bar for everything in this chapter, and its threshold is the one its manifest records, chosen on dev by the rule [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) gave.

Gold labels cost people time to make, and distillation is the attempt to get around that: a larger model's answers stand in for the gold labels. The larger model here is the [stand-in](route-a-message#ask-a-model-for-one-team). It is larger than SmolLM2-135M-Instruct, the base model this book's router is built on, and it answers the same question the router asks.

## What distillation copies {#what-distillation-copies}

Distillation trains a small model to reproduce a large one. Hinton, Vinyals and Dean's paper, which gave distillation its name, proposed training a small model on the probabilities that a large model gives every class, as soft targets ([Hinton et al., 2015, arXiv:1503.02531](https://arxiv.org/pdf/1503.02531)). Those probabilities carry more information than a single label, the paper argues, and a small model trained this way can often learn from far less data than the large model needs. That is the paper's own claim, from its own experiments, and it is not a rule about every task.

This chapter does not use soft targets. The stand-in returns one team for each message, the team it judges most likely, and the student sees only that team. Kim and Rush used the same simplification for translation, where they replaced the teacher's distribution with its most likely output: "Our simplest approximation is to replace the teacher distribution q with its mode" ([Kim and Rush, 2016, arXiv:1606.07947](https://arxiv.org/pdf/1606.07947)). Their setting is translation, not classification. We borrow the simplification, and we measure it on this task instead of assuming it carries over. Here the teacher's greedy answer plays the part of the mode: it is the one team the teacher returns for each message.

Two things follow from training on one label per message. A student learns the teacher's errors along with its correct answers, because the training loop treats a wrong label as a label, the same as a right one. And the student's errors can differ from the teacher's, which is the reason to measure the student against gold labels as well as against the teacher.

**Working rule:** Before you train a student on a teacher's answers, measure how often the teacher's answer matches the gold label on rows that you have labelled, and read some of the disagreements. Expect the student to reproduce the teacher's errors, and count them on messages the student has not trained on.

## A teacher whose terms allow it {#a-teacher-whose-terms-allow-it}

The teacher has to be a model whose terms allow training on its outputs. The stand-in's GGUF build in this book is the repository `unsloth/Qwen3-4B-Instruct-2507-GGUF`, file `Qwen3-4B-Instruct-2507-Q4_K_M.gguf`, at the revision the teacher's manifest records. The model is Qwen3-4B-Instruct-2507, from Qwen, and the build is by unsloth. We credit both. The model card of the build declares `license: apache-2.0` and names `Qwen/Qwen3-4B-Instruct-2507` as its base model ([model card at the pinned revision](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/raw/a06e946bb6b655725eafa393f4a9745d460374c9/README.md)). The base model's licence, at [a pinned commit](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/resolve/cdbee75f17c01a7cc42f958dc650907174af0554/LICENSE), is Apache License 2.0. The Apache License 2.0, in its second section, grants the right "to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works" ([Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)).

We read Apache-2.0 as placing no restriction on using a model's outputs. Its text governs the Work and such Derivative Works, as section 2 above sets out, and it does not mention outputs. This is our reading of the licence, not a quotation from it, and it holds for this model's licence only. The adapters this book ships are distributed under Apache-2.0, and [the notice](https://github.com/rollingedition/book-small-open-models/blob/36d69d814d8008aeb49bee64a43c02f5513b2f25/examples/data/recorded/NOTICE.md) beside them gives the licence and the attribution.

Licences differ. An open-weights licence such as Apache-2.0 governs the model's files; a hosted model is governed by its provider's terms of service, which can forbid using the service or its outputs to train a competing model. Terms change, so read the current terms of any model you would train on, on the day you use it. This book never uses a hosted model as a teacher.

The stand-in is an open model, so its licence can be read and its file can be fixed by a hash, which the teacher's manifest records. A hosted model's answers can change when its provider changes the model, and its terms may forbid this use. Those are reasons to prefer an open teacher whose file is fixed, even before its terms are read.

**Working rule:** Read the teacher's licence and its terms of use on the day you train. Name the teacher, its licence and its revision in the chapter, and point the student's manifest to the teacher's recording. Do not train on the outputs of a model whose terms forbid it. A licence that places no restriction on outputs answers the question for that licence, so read its text before you assume the answer for another model.

## Label training rows with the teacher {#label-training-rows-with-the-teacher}

The teacher labels training rows only. Dev rows are kept for choosing thresholds, and test rows are kept for reporting, as [chapter 2](measure-the-routing-task#dev-test-and-fixed-samples) set out. The rows to label are drawn from the training file, outside dev. The labelled rows are the first 1,000 of a seeded random draw of 2,000 such rows, a random subset of them, and the recording sorts them by index before labelling. The labelling run was planned for all 2,000, but it could not finish that many within its time limit, so the book reports the 1,000 rows it labelled.

The draw is seeded. Labels already saved for a chosen row are kept and not labelled again, so a run that stops and continues adds only the rows it has not labelled. We did not test whether a resumed run gives the same labels as one full run.

The stand-in answers each message with the router prompt from [chapter 1](route-a-message). It decodes greedily, at temperature zero. In a probe on 40 test messages, two runs gave identical answers, and that probe is the only check we made of repeatability. Its answer is constrained by a grammar to the 77 team names, so each label is a real team.

Example `teacher-agreement` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Measure how often the teacher's label matches gold on the training rows it labelled, and its accuracy on test_200
from its recorded answers. The point rests on the agreement line; the example reads recorded answers only."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded"

train = router_lib.read_rows("train")
teacher_rows = [json.loads(line) for line in (RECORDED / "teacher-labels" / "labels.jsonl").read_text().splitlines()]
# row["index"] is the row's position in the training file; field 1 of a row is its gold team
agree = sum(row["label"] == train[row["index"]][1] for row in teacher_rows)
n = len(teacher_rows)
print(f"training rows labelled by the teacher: {n}")
print(f"teacher agrees with gold on training rows: {agree}/{n} ({agree / n:.1%})")
print(f"teacher disagrees with gold on training rows: {n - agree}/{n} ({(n - agree) / n:.1%})")

test = router_lib.read_rows("test")
sample = router_lib.read_sample("test_200")
standin = router_lib.standin_answers()
# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text
golds = [test[i][1] for i in sample]
teacher_preds = [standin[i] for i in sample]
correct = sum(p == g for p, g in zip(teacher_preds, golds))
print(f"messages on test_200 with a teacher answer: {sum(i in standin for i in sample)}/{len(sample)}")
print(f"teacher accuracy on test_200: {correct}/{len(sample)} ({correct / len(sample):.1%})")
```

Output:

```text
training rows labelled by the teacher: 1000
teacher agrees with gold on training rows: 549/1000 (54.9%)
teacher disagrees with gold on training rows: 451/1000 (45.1%)
messages on test_200 with a teacher answer: 200/200
teacher accuracy on test_200: 109/200 (54.5%)
```

The teacher's label matches the gold label on 549/1000 (54.9%) of the 1000 training rows it labelled, and it disagrees on 451/1000 (45.1%). A student trained on those disagreeing rows learns the wrong label as the answer to each one. The last two lines of the output come from the stand-in's recorded answers on `test_200`, the 200 test messages this chapter reports on. There the teacher is right on 109/200 (54.5%) messages.

Read the line `teacher agrees with gold on training rows` before any student is trained. The line `teacher accuracy on test_200` is the teacher's own accuracy on messages no one has used for training, and it is the reference for every copy measure below.

Counts do not show what the disagreements look like. Three training rows, each labelled `card_arrival` in the gold file, show the pattern. Each message is about a card that has not arrived, described here without its text, since messages are never quoted in this book. The teacher named `card_delivery_estimate` for two of them and `lost_or_stolen_card` for the third, so the teacher names a different team from the gold label each time.

**Working rule:** Measure the teacher's agreement with gold on a labelled sample of rows before you train a student on the labels. Label only rows outside dev and test, and record the teacher's model, its version and its settings in the teacher's recording manifest, and point the student's manifest to it, so each label can be traced.

## Train the student on the teacher's labels {#train-the-student-on-the-teachers-labels}

The student uses the recipe of [chapter 5](fine-tune-with-lora). It has the same base model, the same LoRA targets, the same head, the same seed and the same batch size. Its labels come from the teacher instead of the gold file. The rows and steps differ from chapter 5 as well: the student trains on 1000 rows for 125 steps, and the human-labelled adapter trained on 9000 rows for 1125 steps. The control in [the agreement section](#measure-agreement-not-only-accuracy) keeps the student's rows and steps and changes only the labels. The student's adapter is `banking77-distilled-v1`. Its manifest records its settings and points to the teacher's recording, `teacher-labels/manifest.json`, which names the teacher's model and version, its settings and the rows it labelled, so anyone can trace where each label came from.

The recipe is held fixed on purpose. If the student changed its recipe and its labels at once, a gap in the results could come from either, and a reader could not say which. Keeping the recipe fixed makes the labels the only difference that the results can point to.

A student's threshold is chosen on dev by the same rule as the human-labelled adapter's. The rule takes the lowest threshold, on a grid of hundredths, at which the messages not handed off are at least 0.90 accurate on dev. The rule has a fallback for a student that cannot reach that target. The fallback takes the threshold with the highest accuracy on the messages not handed off, the lowest of any tie, and it skips any threshold that hands off every message. The output says that the fallback was used. This is the book's rule for a result that misses, so it is reported as a fallback and not as a student that met the target. A fallback threshold is chosen from the messages a threshold keeps on dev, and at a high threshold that can be a handful of messages. Treat a fallback value as provisional, and report it with the hand-off rate it costs on test, as this chapter does.

The student's threshold is chosen on part of the dev split, 500 of 1003 messages, because a run over every dev message did not fit the time budget of two CPU cores. Both student examples use that same part, and the control uses it for its own threshold, so the two thresholds are chosen on the same dev messages. The human-labelled adapter's threshold, from chapter 5, was chosen on all 1,003 dev messages.

**Working rule:** Choose a student's threshold on dev by the same rule as the human-labelled adapter, and when no threshold reaches the target, say that the threshold is the fallback. Report the hand-off rate the threshold costs beside accuracy, with both counts, before you compare the student with anything.

## Measure agreement, not only accuracy {#measure-agreement-not-only-accuracy}

Three measures run through this chapter, and each answers its own question. The teacher against gold on the training rows asks whether the teacher is right on the rows it labels, and it is the one [above](#label-training-rows-with-the-teacher). The student against the teacher on `test_200` asks how much of the teacher the student copies. The student against gold on `test_200` asks how often the student names the right team, which is the figure a router would be judged on. Each is printed with its denominator: the first is read on the 1,000 training rows, and the other two on the same 200 test messages.

Agreement and accuracy answer different questions. Agreement with the teacher says how much of the teacher the student has learned, whether the teacher is right or wrong. Accuracy against gold says how often the student names the right team. A student can agree with a wrong teacher and be wrong, and it can disagree with a wrong teacher and be right. Printing both lets a reader see which of the two is happening, instead of inferring it from one figure.

Example `student-recorded-adapter` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Score the distilled student banking77-distilled-v1 on test_200 beside the human-labelled adapter, with the student's
threshold chosen on dev. The point rests on the student's accuracy line and its copy lines; both adapters are loaded
and scored live, and nothing is trained."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded"

tokenizer, student = router_lib.load_classifier(RECORDED / "distill-student" / "adapter")
DEV_N = 500  # the first 500 of the 1,003 dev messages: the run's time budget under two CPUs
# reached is False when no threshold meets the target, and the threshold returned is then the fallback
threshold, reached = router_lib.dev_threshold(student, tokenizer, target=0.90, n=DEV_N)
print(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")
print(f"dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")
print(f"student threshold chosen on dev: {threshold:.2f}")

test = router_lib.read_rows("test")
sample = router_lib.read_sample("test_200")
# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text
golds = [test[i][1] for i in sample]
texts = [test[i][0] for i in sample]
student_preds, student_confs = router_lib.classify(student, tokenizer, texts)
router_lib.print_copy_report("banking77-distilled-v1", student_preds, student_confs, golds, threshold)

# chapter 5 chose the adapter's threshold on dev, and its manifest records the value used here
lora = json.loads((RECORDED / "lora-full" / "manifest.json").read_text())
_, lora_model = router_lib.load_classifier(RECORDED / "lora-full" / "adapter")
lora_preds, lora_confs = router_lib.classify(lora_model, tokenizer, texts)
router_lib.print_copy_report("banking77-lora-v1", lora_preds, lora_confs, golds, lora["handoff_threshold"])

teacher = router_lib.standin_answers()
teacher_preds = [teacher[i] for i in sample]
matches = sum(s == t for s, t in zip(student_preds, teacher_preds))
# positions within sample where the student's answer is not the gold team
wrong = [k for k in range(len(sample)) if student_preds[k] != golds[k]]
wrong_like_teacher = sum(student_preds[k] == teacher_preds[k] for k in wrong)
print(f"student answers equal to teacher answers on test_200: {matches}/{len(sample)} ({matches / len(sample):.1%})")
print(f"student errors on test_200: {len(wrong)}/{len(sample)} ({len(wrong) / len(sample):.1%})")
print(f"student errors equal to the teacher's answer: {wrong_like_teacher}/{len(wrong)}")
```

Output:

```text
dev messages used: 500 of 1003
dev threshold reaching 0.90 accuracy not handed off: no
student threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-lora-v1 accuracy on test_200: 172/200 (86.0%)
banking77-lora-v1 handed off: 33/200 (16.5%)
banking77-lora-v1 accuracy on messages not handed off: 156/167 (93.4%)
student answers equal to teacher answers on test_200: 100/200 (50.0%)
student errors on test_200: 131/200 (65.5%)
student errors equal to the teacher's answer: 36/131
```

The student is well behind the human-labelled adapter, and the gap is not only in hand-offs, as the overall figures below show. On `test_200`, the student is right on 69/200 (34.5%) messages and hands off 150/200 (75.0%), while the adapter is right on 172/200 (86.0%) messages and hands off 33/200 (16.5%). The student's threshold is the [fallback described above](#train-the-student-on-the-teachers-labels). Each run's accuracy on the messages it keeps is at its own threshold, and the output gives both beside this paragraph.

The copy measures come next. The student's answer equals the teacher's on 100/200 (50.0%) messages, and the teacher itself is right on 109/200 (54.5%). A student that copies the teacher closely therefore copies the teacher's wrong answers too. The student makes 131/200 (65.5%) errors on `test_200`. Of those errors, 36/131 repeat the teacher's answer. The rest differ from it, so a student's mistakes are not only the teacher's mistakes.

The comparison with the adapter alone mixes two things. The adapter learned from the gold labels of the full training split, and the student learned from the teacher's labels on a smaller set of rows. A gap between them could come from the labels, from the number of rows or from both. To separate them, the book records the same recipe on the same rows with gold labels. The control, `banking77-gold-1000-v1`, is the student's recipe trained on gold labels, and its rows are the rows the teacher labelled.

Example `student-vs-gold-control` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Put the student banking77-distilled-v1 beside its control banking77-gold-1000-v1, which has the same recipe and the
same 1,000 training rows but gold labels, on the same test_200 messages. The point rests on the two counts of messages
each adapter gets right that the other gets wrong, which need no threshold; both adapters are scored live, and nothing
is trained."""

import json
import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded"
DEV_N = 500  # the first 500 of the dev messages, as student-recorded-adapter uses: the time budget under two CPUs
print(f"dev messages used: {DEV_N} of {len(router_lib.read_sample('dev'))}")

test = router_lib.read_rows("test")
sample = router_lib.read_sample("test_200")
# sample holds test row positions; field 1 of a test row is its gold team, field 0 its text
golds = [test[i][1] for i in sample]
texts = [test[i][0] for i in sample]

runs = {}
for name, folder in [("banking77-distilled-v1", "distill-student"), ("banking77-gold-1000-v1", "gold-same-rows")]:
    tokenizer, model = router_lib.load_classifier(RECORDED / folder / "adapter")
    threshold, reached = router_lib.dev_threshold(model, tokenizer, target=0.90, n=DEV_N)
    print(f"{name} dev threshold reaching 0.90 accuracy not handed off: {'yes' if reached else 'no'}")
    print(f"{name} threshold chosen on dev: {threshold:.2f}")
    preds, confs = router_lib.classify(model, tokenizer, texts)
    router_lib.print_copy_report(name, preds, confs, golds, threshold)
    runs[name] = [p == g for p, g in zip(preds, golds)]

# dict order is insertion order: the student first, then the control
student, control = runs.values()
n = len(golds)
print(f"test_200 messages the student gets right and the control gets wrong: {sum(s and not c for s, c in zip(student, control))}/{n}")
print(f"test_200 messages the control gets right and the student gets wrong: {sum(c and not s for s, c in zip(student, control))}/{n}")

# Recipe sizes from the recorded manifests: the student and its control share rows and steps;
# the human-labelled adapter does not. Its manifest names its row count train_rows, not rows, which is why rows_key differs.
for name, folder, rows_key in [
    ("banking77-distilled-v1", "distill-student", "rows"),
    ("banking77-gold-1000-v1", "gold-same-rows", "rows"),
    ("banking77-lora-v1", "lora-full", "train_rows"),
]:
    settings = json.loads((RECORDED / folder / "manifest.json").read_text())["settings"]
    print(f"{name} training rows: {settings[rows_key]}")
    print(f"{name} training steps: {settings['steps']}")
```

Output:

```text
dev messages used: 500 of 1003
banking77-distilled-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-distilled-v1 threshold chosen on dev: 0.68
banking77-distilled-v1 accuracy on test_200: 69/200 (34.5%)
banking77-distilled-v1 handed off: 150/200 (75.0%)
banking77-distilled-v1 accuracy on messages not handed off: 28/50 (56.0%)
banking77-gold-1000-v1 dev threshold reaching 0.90 accuracy not handed off: no
banking77-gold-1000-v1 threshold chosen on dev: 0.97
banking77-gold-1000-v1 accuracy on test_200: 77/200 (38.5%)
banking77-gold-1000-v1 handed off: 197/200 (98.5%)
banking77-gold-1000-v1 accuracy on messages not handed off: 3/3 (100.0%)
test_200 messages the student gets right and the control gets wrong: 24/200
test_200 messages the control gets right and the student gets wrong: 32/200
banking77-distilled-v1 training rows: 1000
banking77-distilled-v1 training steps: 125
banking77-gold-1000-v1 training rows: 1000
banking77-gold-1000-v1 training steps: 125
banking77-lora-v1 training rows: 9000
banking77-lora-v1 training steps: 1125
```

The control also has no threshold that meets the dev target, so it uses the fallback too, and its threshold is provisional for that reason. It hands off 197/200 (98.5%) of `test_200`, and it is right on 3/3 (100.0%) of the messages it keeps. Three messages are not a measure. Overall accuracy needs no threshold, because it counts every message whatever the threshold, so the control's 77/200 (38.5%) sits directly beside the student's 69/200 (34.5%).

The fair comparison is on the answers themselves, and it needs no threshold, because it compares the teams. The control is right and the student wrong on 32/200 messages, and the student is right and the control wrong on 24/200. The control holds no margin over the student that this sample can settle, so we judge 200 messages too few to call one label source better than the other.

The adapter trained on 9000 rows for 1125 steps, while the control and the student share the 1000 rows and 125 steps, so the difference between the control and the adapter mixes row count and steps with anything else the two runs differ in, and this chapter does not separate them.

**Working rule:** Compare two adapters only on the same messages and the same rows. Count the messages each one gets right that the other gets wrong, and print both counts with their denominator. Do not set accuracies taken at different thresholds side by side as though they were the same measure.

## When the teacher is wrong {#when-the-teacher-is-wrong}

A student trained on one label per message learns the labels it is given. Where the teacher is wrong on a row, the student has been shown the wrong team for that row, and it has no way to know otherwise. The teacher's agreement with gold on the training rows, [measured above](#label-training-rows-with-the-teacher), is the share of the teacher's training labels that are right. The student is measured on test messages and not on its training rows, because its figures on rows it trained on would show what it memorised rather than what it learned.

On this task, the student that learns from the stand-in's labels is not the one to ship, as the [figures above](#measure-agreement-not-only-accuracy) show. [Chapter 7](quantize-and-measure) quantizes `banking77-lora-v1`, the human-labelled adapter, and not the student.

Read the copy measure next to accuracy. If the student copies the teacher on most messages and the teacher is wrong on many, the student inherits those errors, and its accuracy against gold falls with the teacher's. If the student disagrees with the teacher on many messages and is still wrong, it is learning something other than the labels, and the first suspect is the number of training rows. That is a rule of thumb from this book's own figures, and this chapter does not test it.

Distilling is worth its cost in some cases. We recommend trying it when gold labels are scarce, when the teacher is strong on your task, and when a sample of gold labels shows that the teacher agrees with them often enough to make the student's errors acceptable. When you already have gold labels for your training rows, train on them, and use a teacher only to check them. A teacher model also changes. Treat a new version of the teacher as a new recording: its labels change, so the student's manifest, its threshold and its measures have to be made again.

In production the failure is quiet. A student that copies its teacher returns a steady stream of teams and a steady hand-off rate, and neither alarms anyone. Check the student against gold labels on a sample of messages that a person has labelled, on a fixed schedule. [Chapter 9](version-watch-and-fall-back#sample-handed-off-messages-for-people) describes the same kind of sample for the adapter's hand-offs as a next step, and this book does not build it. Compare the student's answers with those labels, not with the teacher's, because agreement with the teacher says nothing about the team a message belongs to.

The decisions this chapter made are these:

- The teacher labels training rows only, never dev or test rows.
- The teacher is the stand-in, Qwen3-4B-Instruct-2507, under Apache-2.0, and no hosted model is used.
- The student is trained with the same recipe as the human-labelled adapter, and its threshold follows the same rule, with the fallback reported as a fallback.
- The student is compared with a control trained on gold labels for the same rows, on the same messages, and the two share their rows, so a difference between them comes from the labels and not from the rows.

A teacher's labels carry its mistakes into the student, so agreement with gold decides whether distilling is worth it.

**Working rule:** Train a student on a teacher's labels only when you have too few gold labels for the task and a sample of gold labels shows that the teacher agrees with them well enough. Compare the student with the human-labelled adapter, and with the same recipe on gold labels, on the same messages, before you choose it. When the student loses, ship the adapter and say so in the write-up.

# Quantize the router and measure what changes

By the end of this chapter, you can store the fine-tuned router's backbone weights as 8-bit integers in one call, count the predictions that change on a fixed sample, and say what that count does and does not show. You will also see how speed is measured, and what one machine's recording found.

A quantized copy of the router would take less space; the question is whether it still answers the same way. [Chapter 5](fine-tune-with-lora) left the router as one adapter, `banking77-lora-v1`, whose weights are float32. Its hand-off threshold is 0.63, chosen on the dev split by the rule in [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev). Quantizing stores the weights in fewer bits. A quantized model takes less space for its weights, and it may return different answers. This chapter measures the second part on the fixed sample of 200 test messages that chapter 5 scored, `test_200`. It does not claim that quantizing is free.

The copy could take the float32 adapter's place in the [inbox](route-a-message) that routes the bank's support messages, and three questions come before it does. How many bytes do the copy's weights take, compared with the adapter's? Does the copy answer the same messages the same way? And is it faster on the machine that serves the inbox? The first two can be measured here, from the model's shapes and from a fixed sample. The third comes from a recording on a named machine, and the section on speed says what that recording found and what it does not cover. Each answer is a count with its denominator, or a ratio with its inputs named.

## One call to quantize the linear layers {#one-call-to-quantize-the-linear-layers}

A linear layer is a weight matrix that maps one vector to another. The router's backbone is made of them. Quantizing a layer stores each of its weights as an 8-bit integer, with a scale that maps the integers back to approximate float values. Fewer bits per weight means fewer bytes for the layer, and the answers can move, because the values the layer computes are no longer the exact ones.

The scale is what makes the scheme work for a whole matrix. A single scale for an entire matrix has to cover its largest weight, so the rows whose weights are small get a coarse grid. Jacob et al. (2018) report that a shared resolution gives channels with narrow ranges much larger relative error ([arXiv:1712.05877](https://arxiv.org/abs/1712.05877)). That is a CNN result, so here it is a reason to test per-row scales, not a measurement. A per-row scale lets each output row use the full range of the 8-bit integers for its own weights. That is why the byte count adds one float32 scale for each output row. The scales are few, but they are not free, so the byte count below includes them.

The helper in this book, `int8_classifier`, names its scheme in its docstring: per-row int8 weights, with float32 activations. The scheme is torchao's `Int8WeightOnlyConfig`, which the [torchao source at v0.18.0](https://raw.githubusercontent.com/pytorch/ao/v0.18.0/torchao/quantization/quant_api.py) describes as "int8 weight-only symmetric per-channel quantization to linear layers". The helper applies it in three steps. It merges the adapter's weights into the base model with PEFT's `merge_and_unload`. It calls `quantize_` from `torchao.quantization` with `Int8WeightOnlyConfig`, and passes a filter that selects every `nn.Linear` layer except the one named `score`. Then it returns the model in evaluation mode. The int8 copy, named `banking77-lora-v1-int8` in the recordings, is the model the helper returns: a derived model built from the adapter, not a second adapter. The output layer, `score`, is the head that chapter 5 trained, and the section on it explains why the helper leaves it out.

The merge comes first for a reason. [Chapter 5](fine-tune-with-lora#adapters-train-a-small-part-of-the-model) described the adapter as a small set of matrices beside the base model's layers. Merging folds the adapter's update into the base weights, so each layer is one weight matrix, quantized once, and the layer count describes the router. Without the merge, the same call would also quantize the adapter's own matrices as separate int8 layers.

Older PyTorch code uses `torch.ao.quantization.quantize_dynamic`, which also stores Linear weights as int8 and quantizes activations as it runs, a different scheme. PyTorch 2.15 deprecates it, and its warning points to torchao's `quantize_` ([PyTorch v2.15.0, `torch/ao/quantization/utils.py`](https://raw.githubusercontent.com/pytorch/pytorch/v2.15.0/torch/ao/quantization/utils.py)), which the [torchao README at v0.18.0](https://raw.githubusercontent.com/pytorch/ao/v0.18.0/README.md) shows as the migration. The examples in this book call no deprecated API themselves.

Example `int8-layers-and-bytes` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Counts the backbone Linear layers the int8 copy quantizes, and the weight bytes they store, from their shapes.
The prose's saving rests on the last printed line, the int8 share of float32 weight bytes; the head assert checks that the score head is not quantized."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)  # the counts come from shapes, so the thread count changes no printed line

import router_lib  # noqa: E402

router_lib.setup_runtime()  # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtype
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"

_, model = router_lib.int8_classifier(RECORDED / "adapter")  # merges the adapter, then quantizes every backbone Linear, not score
layers = router_lib.backbone_linears(model)  # (name, layer) pairs, all except score
quantized = [layer for _, layer in layers if router_lib.is_int8(layer)]  # a torchao int8 weight, not a float one
weights = sum(layer.weight.numel() for _, layer in layers)
rows = sum(layer.weight.shape[0] for _, layer in layers)  # weight shape is (output rows, input size)

print(f"backbone Linear layers: {len(layers)}")
print(f"backbone Linear layers quantized to int8: {len(quantized)}")
head_quantized = router_lib.is_int8(model.score)
assert not head_quantized, "the score head must stay float32"
print(f"score head quantized: {'yes' if head_quantized else 'no'}")
print(f"score head weight dtype: {model.score.weight.dtype}")
print(f"weights in backbone Linear layers: {weights}")
print(f"output rows in backbone Linear layers: {rows}")
float_bytes = 4 * weights
int8_bytes = 1 * weights + 4 * rows  # one byte a weight, plus one float32 scale (4 bytes) for each output row
print(f"weight bytes, int8 as a share of float32: {int8_bytes / float_bytes:.1%}")
```

Output:

```text
backbone Linear layers: 210
backbone Linear layers quantized to int8: 210
score head quantized: no
score head weight dtype: torch.float32
weights in backbone Linear layers: 106168320
output rows in backbone Linear layers: 155520
weight bytes, int8 as a share of float32: 25.1%
```

Building the int8 copy and counting its layers and their weights shows what the helper quantized. The backbone has 210 linear layers, and the int8 copy quantizes 210 of them. Those layers hold 106168320 weights, in 155520 output rows.

The last line is arithmetic on shapes, and it is not a measurement of a file or a process. A float32 weight takes 4 bytes. An int8 weight takes 1 byte, and the calculation adds one float32 scale, 4 bytes, for each output row, because that is the convention the helper's scheme names. Over the weights in those layers, the int8 copy takes 25.1% of the float32 bytes. A quarter of that is the weights themselves, and the rest is the scales. The figure covers only the weights of these layers. It leaves out every other tensor the model holds, so it says nothing about the size of a file on disk, and nothing about the memory a running router uses.

The bytes matter on the serving machine only if its memory has to hold the weights. Whether it does depends on the machine and the rest of the process, which this chapter does not measure. The ratio tells a reader how much of the weight budget the copy frees.

Importing torchao prints warnings to standard error, and an example's first run may print a load report from transformers. They come from the libraries' own import code, not from a call the examples make, and nothing in the examples suppresses or acts on them. Some lines say torchao could not load its GPU-only kernel libraries, which the CPU path does not use. A `SyntaxWarning` from torchao's own source shows only on the first import, before Python caches the bytecode. Notices that `register_constant()` is deprecated come from torch and are triggered by torchao's import. The load report lists `score.weight` as MISSING: the base checkpoint has no classification head, so transformers starts a new one, and the adapter's saved head replaces it. None of these changes an output line. The book pins torchao at 0.18.0 in its requirements file, and every example uses that pin.

**Working rule:** Quantize only the layers you can name, leave the output layer in float32, and count the bytes from the shapes before you claim any saving. Report the saving as a ratio of weight bytes, with the count of layers beside it, and never as a file size or as a speed.

## Count the predictions that change {#count-the-predictions-that-change}

Quantizing changes the numbers a layer computes, so the answers can change too. The test of a quantized copy is whether it answers the same messages the same way.

Scoring `test_200` with both copies at the threshold in chapter 5's manifest shows whether they answer alike. For each copy the run counts the correct answers, the hand-offs and the accuracy on the messages not handed off. It then counts the messages whose team differs between the two copies, and the hand-off decisions that differ. The same run, `int8-changed-predictions`, also scores a third copy, with the head quantized too, for [the section on the score head](#keep-the-score-head-in-float32).

The two copies are not treated alike on purpose. The float32 side is the adapter as chapter 5 loaded it, unmerged, while the int8 side is merged first, because the helper merges before it quantizes. So the count below holds the merge and the quantization together. The run never scores a merged float32 copy on its own, and the chapter does not claim that the merge alone changes nothing.

Example `int8-changed-predictions` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Scores test_200 with the float32 adapter, its int8 copy and the copy with the head in int8, at the threshold in chapter 5's manifest.
The prose's point rests on the line that counts predictions changed by int8, read with the hand-off line under it."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)  # two threads for these forward passes; the speed section's recording uses four

import router_lib  # noqa: E402

router_lib.setup_runtime()  # fixes the seeds, requires deterministic algorithms and sets float32 as the default dtype
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]  # chosen on dev in chapter 5, stored in the manifest
test = router_lib.read_rows("test")  # (text, label index) pairs
sample = router_lib.read_sample("test_200")  # fixed row indices, so every copy scores the same messages
golds = [test[i][1] for i in sample]  # index 1 of each row is its gold team
texts = [test[i][0] for i in sample]

tokenizer, float_model = router_lib.load_classifier(RECORDED / "adapter")  # the adapter as chapter 5 loaded it, unmerged
float_preds, float_confs = router_lib.classify_by_length(float_model, tokenizer, texts)  # results in the order of texts
_, int8_model = router_lib.int8_classifier(RECORDED / "adapter")  # merged first, then backbone quantized, head float32
int8_preds, int8_confs = router_lib.classify_by_length(int8_model, tokenizer, texts)
_, head_model = router_lib.int8_classifier_all_linears(RECORDED / "adapter")  # as above, with the score head quantized too
head_preds, head_confs = router_lib.classify_by_length(head_model, tokenizer, texts)

print(f"handoff threshold from manifest: {threshold:.2f}")
print(f"messages on test_200: {len(texts)}")
router_lib.print_copy_report("float32", float_preds, float_confs, golds, threshold)  # a handed-off message is wrong if its team is wrong; the last measure divides by kept messages only
router_lib.print_copy_report("int8", int8_preds, int8_confs, golds, threshold)
changed, handoffs = router_lib.count_changes(float_preds, float_confs, int8_preds, int8_confs, threshold)  # a changed prediction is a different team, handed off or not; a changed hand-off is a confidence on the other side of the threshold
print(f"predictions changed by int8: {changed}/{len(texts)} ({changed / len(texts):.1%})")
print(f"hand-off decisions changed by int8: {handoffs}/{len(texts)}")
changed_head, _ = router_lib.count_changes(float_preds, float_confs, head_preds, head_confs, threshold)
print(f"predictions changed with the head in int8 too: {changed_head}/{len(texts)}")
```

Output:

```text
handoff threshold from manifest: 0.63
messages on test_200: 200
float32 accuracy on test_200: 172/200 (86.0%)
float32 handed off: 33/200 (16.5%)
float32 accuracy on messages not handed off: 156/167 (93.4%)
int8 accuracy on test_200: 172/200 (86.0%)
int8 handed off: 33/200 (16.5%)
int8 accuracy on messages not handed off: 156/167 (93.4%)
predictions changed by int8: 0/200 (0.0%)
hand-off decisions changed by int8: 0/200
predictions changed with the head in int8 too: 1/200
```

On this sample the int8 copy changes 0/200 (0.0%) predictions and 0/200 hand-off decisions. The copy that also quantizes the head is counted in [the section on the score head](#keep-the-score-head-in-float32). The output beside this paragraph gives each copy's accuracy, its hand-offs and its accuracy on the messages not handed off, each with its denominator.

The three measures are the ones [chapter 2](measure-the-routing-task#three-measures-each-with-its-denominator) defined, and each has its denominator. Accuracy divides the correct messages by all 200, so a handed-off message counts as wrong if its team is wrong. The hand-off rate divides the messages below the threshold by all 200. Accuracy on the messages not handed off divides the correct ones among the kept messages, and it is not defined when none are kept. The changed-prediction count uses the same 200 messages and asks a question the three measures do not: did the two copies agree on each one? Agreement is not correctness, which is why the output also prints the accuracy.

A changed prediction is not automatically a loss. It can turn a wrong team into a right one, or a right team into a wrong one, and the count alone does not say which. To read the direction, compare each copy's team with the gold label on every message the copies disagree on. Accuracy can hide this. Two copies can have the same accuracy while different messages are right on each, so the same total says nothing about which messages moved. When the count is zero, there are no flips to read in either direction. A reader who gets a non-zero count should read the flips before reading the totals.

A flip has a concrete cost once a copy serves the queue. Suppose the float32 adapter sends a message to `card_arrival` and the int8 copy sends the same message to `card_delivery_estimate`. Both are teams, and only one of them can be the message's gold label, so the change either fixes an answer or breaks one, unless both teams are wrong. The inbox sends the message to one team either way. The change would reach the support staff who work that team's queue, and nobody would see it in a total. The count is what makes that change visible before the copy serves any message.

Read the changed-prediction count as one result, from one adapter at one threshold, and not as a guarantee. A different sample, adapter or threshold could show a change, and a sample of 200 can miss a change that is rare in a full stream of messages.

The count is about predictions and nothing else. The copy also keeps its activations in float32, as the helper's docstring says, so this count tells you nothing about a copy that quantizes activations as well. This chapter does not measure that copy.

**Working rule:** Count the changed predictions on the same messages, at the same threshold, for both copies, and print each count with its denominator beside the accuracy and the hand-off count. When the count is zero, report it as zero on that sample, and name the sample and the adapter it was taken from.

## Keep the score head in float32 {#keep-the-score-head-in-float32}

The score head is the layer with 77 outputs, one for each team. Its softmax gives the confidence that chapter 5 compares with the threshold, and its largest probability is the team the router returns. The helper leaves the head out of the call, so the head stays in float32. This section says why the book makes that choice, and what the count shows for it.

The copy this chapter serves keeps its head in float32, and that choice rests on one count, from `int8-changed-predictions`. With the head in int8 as well, 1/200 predictions change on this sample. The count does not rank layers by sensitivity, and the chapter does not claim that the head is, or is not, where a change comes from.

The messages at risk are the ones whose confidence sits a little above the threshold. A change in the head can carry one of them below it, and the message then goes to a person rather than to a team. The count of changed hand-off decisions in [the previous section](#count-the-predictions-that-change) shows whether a copy moves any, and in this run the int8 copy moves 0/200 hand-off decisions.

What the helper did with the head is in `int8-layers-and-bytes`. The head is not quantized, as that example asserts, and its weights are torch.float32. Those printed lines come from the same run as the layer counts in [the first section](#one-call-to-quantize-the-linear-layers), so the two can be compared directly.

The changed-prediction count covers the whole int8 copy and the copy with the head quantized as well. It does not split either one layer by layer, so it cannot say which layer a change comes from. Finding that takes quantizing one layer at a time and counting again on the dev split.

**Working rule:** Keep the score head in float32 unless a count on the dev split, at the chosen threshold, shows that a quantized head changes no team and no hand-off that you rely on. Say which threshold you counted at.

## Measure speed in a recording {#measure-speed-in-a-recording}

Fewer bytes for the weights does not by itself make a model faster, and this chapter does not claim that it does. Whether the int8 copy computes faster on a given CPU depends on the arithmetic routines that run on that CPU, so the book measures it on one named machine.

No example prints a timing. A timing changes with the machine and with the load on it, so an example that printed one would give a different output on every run, and the book's examples have to give the same output every time. A speed ratio belongs in a recording instead.

Weight-only int8 saved weight bytes, and whether the copy also computes faster on a CPU is a separate question. The `throughput` recording tests that. It times this chapter's int8 copy, `banking77-lora-v1-int8`, built by the same helper the examples use, against the float32 adapter and against the stand-in, on the machine named in its manifest. On that machine, with four threads and batches of 16, the int8 copy ran at 0.9 times the float32 classifier's messages a second, so the recording gives no speed reason to serve int8. [Chapter 10](when-a-small-model-is-the-wrong-choice#count-the-bill-and-the-machine) reports the rates with the machine they were measured on. The recording did not time the 360M model, and this chapter claims no speed for any machine but that one.

A recording of that kind pins its model and its inputs, runs them once on the named machine, and saves the outputs with a manifest that names the script. Examples that use a recording read its saved outputs and do not run the model themselves. For the speed comparison, the recording runs the same messages through both copies, with the same number of threads for each, and saves the timing of each copy as its own output. The ratio is worked out from those saved outputs, and the reader can see the machine and the thread count beside it. A reader who wants the ratio for their own machine should run the same recording there, and report the machine with the result.

**Working rule:** Read a speed ratio only from a recording that names the machine it ran on, and never infer one from the weight bytes. The ratio above describes its own machine only, so run the recording on the machine you would serve from before you read it as your speed.

## Decide with the hand-off in view {#decide-with-the-hand-off-in-view}

The decision is which copy the router serves. [Chapter 8](serve-the-router-on-a-cpu) serves the float32 adapter by default, and nothing in this chapter changes that default. The int8 copy's count of changed predictions is 0/200 (0.0%), and its count of changed hand-off decisions is 0/200. A zero count keeps the copy a candidate. The saving it offers is a ratio of weight bytes in some layers. The recording found no speed gain for the int8 copy on its machine, and the book has no measured memory need that favours it, so a copy with no changed answers in this count and no measured gain has nothing to recommend it yet.

Keeping float32 is a choice with a cost. The adapter takes more bytes than its int8 copy would, and on a small machine that can matter. What the book has shown is narrower. On this sample, the count of changed predictions is 0/200 (0.0%), and only a zero count removes prediction change as the reason to refuse the copy. The remaining reasons are ones this chapter has not measured, so they decide the default. If a recording on the serving machine shows a speed gain that matters, and a memory need is measured, the case for int8 becomes an engineering case with a counted risk. Until then the float32 adapter is the one with the fewest open questions.

The hand-off is the part to watch. Chapter 5 chose its threshold so that accuracy on the messages not handed off reaches its target on the dev split. A copy whose confidences move even a little can hand off a different set of messages, and then the threshold no longer does what it was chosen to do. That is why the changed-prediction count is taken at the chosen threshold, and why the hand-off count sits beside it. The same measures apply to any adapter that is quantized, such as a retrained `v2`.

A quantized copy goes wrong in production through a change nobody counted. A library upgrade can change the arithmetic, and a shift in the mix of messages can move some of them across the threshold. Count again on the dev split whenever the libraries or the messages change, and before a quantized copy serves its first message.

The same steps apply to a quantized copy of a reader's own router, and they need nothing in this book beyond the samples and the threshold. Take the threshold from the adapter's manifest, and check that the manifest says it was chosen on the dev split. Quantize the backbone and leave the head in float32, as the helper does. Score the fixed sample with both copies, and count the accuracy, the hand-offs, the accuracy on the messages not handed off and the changed predictions, each with its denominator. Read the flips in each direction before reading the totals. Then count again on the dev split at the chosen threshold, because the threshold was chosen there. A copy that passes these steps still has its speed and its memory measured in a recording before it replaces the float32 adapter.

**Working rule:** Serve the float32 adapter by default. Consider the int8 copy only when a recording on the serving machine names a speed gain, a measured memory need favours it, and then count its changed predictions on the dev split at the chosen threshold before it serves a message.

# Serve the router on a CPU, with the hand-off inside the route

By the end of this chapter, you can serve the router as one `POST /route` endpoint that returns a team, a confidence and a hand-off flag for each message. You can also test that the route never returns a team for a message it has handed off.

The route is where the hand-off rule either holds or fails. [Chapter 1](route-a-message) set out the teams, and [chapter 5](fine-tune-with-lora) built a head that gives a team and a confidence in one forward pass, and it chose the threshold that decides when a message goes to a person. This chapter puts that model behind an HTTP route, which is where the inbox's software sends a message and reads back a team or a hand-off.

If a caller can get a team from the route for a message the router has flagged, the flag does nothing, however carefully chapter 5 chose its threshold. So the route computes the flag itself, from the same answer as the team, and it returns a team only when the flag is false. Nothing else in the chapter is as important as that rule.

This chapter serves the float32 adapter that [chapter 5](fine-tune-with-lora#use-the-recorded-adapter) took from the recording as it was. [Chapter 7](quantize-and-measure) makes a weight-only int8 copy, `banking77-lora-v1-int8`, and measures the copy. This chapter does not. It also measures nothing about speed. That comparison comes from a recording in [chapter 10](when-a-small-model-is-the-wrong-choice), and nothing in this chapter predicts it.

## One route that returns the hand-off {#one-route-that-returns-the-hand-off}

A route is a function that a web framework calls when a request arrives at a path with a method. The framework here is FastAPI, pinned at 0.163.2 in `examples/requirements.txt`. The router has one route, `POST /route`. A caller sends a JSON object with one field, `message`, which holds the text of one inbox message. The route answers with a JSON object of five fields: `team`, `confidence`, `handed_off`, `adapter` and `version`.

Each field has one source, and that matters for the hand-off. The `confidence` is the softmax probability of the team the model chose, which is the meaning [chapter 5 gave it](fine-tune-with-lora#a-head-that-scores-the-teams). The `handed_off` flag is the result of `handoff(confidence, threshold)`, which is true when the confidence is below the threshold. The comparison is written as `not confidence >= threshold`, so a confidence that is not a number, NaN included, is handed off too. The threshold is not typed into the route. The route reads it from the adapter's manifest, where chapter 5 wrote the value it chose on the dev split, so the route and the chapter cannot disagree about the number. The `team` is `None` when `handed_off` is true, and it is the team's label otherwise. The `adapter` and `version` fields tell the caller which model answered. The manifest has no field for the adapter's version, so the version is the suffix of the adapter's name: `banking77-lora-v1` reports `v1`.

The route calls the classifier once per request. One forward pass gives the team and the confidence together, and the flag is computed from that same confidence. The two cannot come from two different calls and drift apart. A reader who opens the module will find one route of its own, and one path to a team, the answer the route builds after the flag is set. FastAPI adds its documentation pages, which return no team. We recommend that a router keep this shape: one route, one model call, one flag, and the team read only after the flag. A second route that skips the flag would be the quickest way to undo the work of chapter 5, and it would be hard to notice, because it would still return correct teams for the messages it does not hand off.

Each message of `test_40`, a [fixed sample](measure-the-routing-task#dev-test-and-fixed-samples) of 40 that the book uses for quick checks, goes to the route, and the test counts the flags and the teams. It prints counts and never the text of a message.

Example `serve-route-testclient` (Python 3.12.15 · fastapi 0.163.2 · peft 0.21.3 · pydantic 2.14.4 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Send the test_40 messages to the one POST /route through TestClient, in process, and print counts that tie the
hand-off flag to the team. The prose's point rests on the "handed off, team is None" line and the "not handed off,
team is the gold label" line; the last checks show the route's answer to a body with no message."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

from fastapi.testclient import TestClient  # noqa: E402

import router_lib  # noqa: E402

router_lib.setup_runtime()                  # seeds, deterministic algorithms and float32, so the counts repeat
client = TestClient(router_lib.make_app(router_lib.HERE / "data" / "recorded" / "lora-full"))  # checks the files, loads the model once, and reads the hand-off threshold from the adapter's manifest

test = router_lib.read_rows("test")         # each row is (message, gold label index)
sample = router_lib.read_sample("test_40")  # row numbers of the committed sample
replies = [client.post("/route", json={"message": test[i][0]}) for i in sample]
bodies = [r.json() for r in replies]
gold = [test[i][1] for i in sample]         # gold label indexes; LABELS turns one into a team name

print(f"responses: {len(replies)}")
print(f"status codes: {sorted({r.status_code for r in replies})}")
print(f"response keys: {sorted(bodies[0])}")
print(f"handed off: {sum(b['handed_off'] for b in bodies)}/{len(bodies)}")
print(f"team names outside the list: {sum(b['team'] is not None and b['team'] not in router_lib.LABELS for b in bodies)}")

sure = [k for k, b in enumerate(bodies) if not b["handed_off"]]    # kept: the route returned a team
unsure = [k for k, b in enumerate(bodies) if b["handed_off"]]      # flagged: the route returned no team
print(f"not handed off, team is the gold label: {sum(bodies[k]['team'] == router_lib.LABELS[gold[k]] for k in sure)}/{len(sure)}")
print(f"handed off, team is None: {sum(bodies[k]['team'] is None for k in unsure)}/{len(unsure)}")
print(f"adapter: {bodies[0]['adapter']}")
print(f"version: {bodies[0]['version']}")

bad = client.post("/route", json={})        # a body with no message field, which the route's body model requires
detail_is_list = isinstance(bad.json()["detail"], list)
assert detail_is_list, "a body with no message should get a 422 whose detail is a list"
print(f"status on an empty body: {bad.status_code}")
print(f"detail is a list: {'yes' if detail_is_list else 'no'}")
```

Output:

```text
responses: 40
status codes: [200]
response keys: ['adapter', 'confidence', 'handed_off', 'team', 'version']
handed off: 8/40
team names outside the list: 0
not handed off, team is the gold label: 30/32
handed off, team is None: 8/8
adapter: banking77-lora-v1
version: v1
status on an empty body: 422
detail is a list: yes
```

The flag and the team agree in this sample. The route answered every request, and the hand-off count is 8/40. Every flagged message came back without a team: 8/8. Of the messages the route kept, 30/32 have the labelled team, the gold label this book scores against. That kept count is the measure [chapter 2](measure-the-routing-task#three-measures-each-with-its-denominator) calls accuracy on the messages not handed off: the correct messages among those the route did not flag. A handed-off message has no team to score, so it stays out of that count. The last count checks the label set: the number of team names outside the list is 0.

The output shows the rule's shape: a flagged message has no team, and a kept message has one. That holds for the 40 messages of the sample, which is evidence, not a proof about every request. The proof is in the code. The route test checks that the code does what the rule says, on a sample the reader can rerun.

**Working rule:** Compute the flag in the route, from the same confidence as the team, with `handoff`, and return a team only when the flag is false. Do not add a second route, a field or an option that returns a team for a message the flag has marked. If a caller needs a team for every message, the answer is a person for the messages the flag marks, not a team from the route.

## Load the model once at startup {#load-the-model-once-at-startup}

A route that loaded the model for each request would repeat the same work on every message. The route in this chapter does not, and the reason is where the code sits. Before it loads anything, the function `make_app` checks the adapter's files against its manifest, and it checks the base model's revision against the pin that the manifest records. It raises an error and builds no app on any mismatch. The check is [chapter 9](version-watch-and-fall-back#check-the-adapters-files-against-its-manifest)'s, and this chapter relies on it. After the check, `make_app` reads the manifest, then loads the tokenizer and the model with `load_classifier`, and only then does it define the route. The route function itself only calls `classify` and then `handoff`. So the model is loaded when the app is built, and each request uses the model that is already in memory.

The route test builds the app once and sends all 40 messages to that one app, so the 40 requests share one load. Loading is part of building the app, not a step a caller takes, and a caller cannot make the route load a second copy by sending a request.

Threads are the other setting that belongs at startup. `make_app` takes the thread count as its `threads` argument, which defaults to two, and it calls `torch.set_num_threads` with that count before it loads the model. So every process that builds the app sets its threads before the first model call. The PyTorch documentation says the call sets the intra-op thread count on the CPU, and that "set_num_threads must be called before running eager, JIT or autograd code" if the count is to hold ([torch 2.15.0+cpu documentation](https://docs.pytorch.org/docs/2.15/generated/torch.set_num_threads.html)).

The `set_num_threads` setting covers the threads used inside one operation, the intra-op threads the documentation names. Two threads is the book's choice, not a measured best. The book's examples have a budget of two vCPUs, so the book sets two threads for them, and a reader on a different machine should choose a count and measure it. The choice is labelled as the book's, because no output in this chapter compares one thread count with another.

**Working rule:** Build the app once per process, with the thread count that process should use, and before the first request. On a two-vCPU budget, as this book uses, keep the default of two threads, and choose another count only after you have measured your own machine with your own messages.

## Know what the route cannot score {#reject-what-the-route-cannot-score}

Two kinds of request cannot be scored, and the route handles them differently. The first is a request that lacks the field the route needs. The route declares its body as a model with one required field, `message`. A body without that field gets status 422. The route test ends by sending such a body, and the reply's `detail` field is a list, which the test asserts. A caller that receives a 422 has no answer to read, and it should not look for a team in that body.

The second kind of request is a message too long to read in full. The model reads a message as tokens, which are pieces of text, and a long message has more of them. The function `classify` calls the tokenizer with truncation on and `max_length` set to 128. A message longer than 128 tokens is cut at 128 before the model reads it. The route does not check the length, and its answer does not say that a cut happened. The same 128 was used in training: the script that recorded `banking77-lora-v1`, `recordings/lora_full.py`, trains with `max_length` set to 128. So the cut at serving is the cut the model was trained with, and that is the reason the route uses it.

A shorter cut, at 64 tokens, is a common choice, and the count below shows what it would touch on the test sample of 200 messages, the sample that [chapter 5](fine-tune-with-lora) compared models on. The example prints how many of the 200 messages are longer than each cut. Only the tokenizer loads here, and no model runs.

Example `truncation-count` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Count the test_200 messages over 64 tokens and over the 128 that classify() cuts at, then show the cut on one long
message. The prose's point rests on the 128-token line and the before and after lengths; only the tokenizer loads."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

from transformers import AutoTokenizer  # noqa: E402

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
repo = json.loads((RECORDED / "adapter" / "adapter_config.json").read_text())["base_model_name_or_path"]  # the adapter's base model
tokenizer = AutoTokenizer.from_pretrained(repo, revision=router_lib.pinned_revision(repo))  # the pinned revision the model uses

test = router_lib.read_rows("test")         # each row is (message, gold label index)
sample = router_lib.read_sample("test_200") # row numbers of the committed sample
lengths = [len(tokenizer(test[i][0]).input_ids) for i in sample]  # full length in tokens, before any cut
n = len(lengths)
print(f"messages over 64 tokens: {sum(x > 64 for x in lengths)}/{n}")
print(f"messages over 128 tokens where classify() cuts: {sum(x > 128 for x in lengths)}/{n}")

# The cut itself: repeat the text of the first test_200 message until it is longer than 128 tokens,
# then tokenize it with classify()'s settings (truncation on, max_length 128).
long_text = test[sample[0]][0]
while len(tokenizer(long_text).input_ids) <= 128:  # repeat the first message until it runs past the 128-token cut
    long_text = long_text + " " + test[sample[0]][0]
before = len(tokenizer(long_text).input_ids)
after = len(tokenizer(long_text, truncation=True, max_length=128).input_ids)  # the settings classify() passes
print(f"repeated message tokens before the cut: {before}")
print(f"repeated message tokens after the cut: {after}")
print(f"truncation side: {tokenizer.truncation_side}")  # "right" cuts from the end of the text
```

Output:

```text
messages over 64 tokens: 1/200
messages over 128 tokens where classify() cuts: 0/200
repeated message tokens before the cut: 132
repeated message tokens after the cut: 128
truncation side: right
```

On this sample, a cut at 64 tokens would touch 1/200 messages, and the cut at 128 tokens, the one `classify` applies, would touch 0/200. Like the route test, this is one sample. It does not say how often long messages occur in a different inbox, and that rate has to be counted on the reader's own messages.

The example's second half shows what the cut does to a message that is long. It repeats the text of the first `test_200` message until the text is longer than 128 tokens, then tokenizes it with the settings `classify` uses. The message reaches the model as 128 tokens, down from 132. The tokenizer's truncation side is right, so the cut drops the end of the text and keeps its start.

A reader whose messages run longer than this sample has to count them before serving, and the route will not count them for the reader. The route does not reject a long message. It cuts the message, scores what is left, and returns a team when the flag is false, and nothing in the answer shows that a cut happened. So the check belongs before the route, in the caller's hands. We recommend treating a message the route would cut as a message for a person, outside the route, and not as a message to guess. A guessed team would look like a routed message and carry no flag, so no one would know to check it. A message the cut touches is scored on its beginning only, and the caller is not told. That is a quiet failure, and the reader should decide what happens to such messages before the route takes them. The safe outcome, which this book recommends, is that a long message goes to a person, outside the route, with the count reported beside the rest of the results.

**Working rule:** Count the messages longer than the cut before you serve a set of messages, with the tokenizer of the adapter's base model and the same cut the route uses. The route cuts a longer message at 128 tokens and scores only what is left, so when the count is above zero, send those messages to a person outside the route, and report the count with the hand-off rate.

## Test the route in the same process {#test-the-route-without-a-socket}

The route's rule, a team only when the flag is false, has to be checked on real answers, and FastAPI's test client lets a test do that without starting a server. The FastAPI testing tutorial says that its test client is used like `httpx`, and that calls to it are ordinary calls, with no `await` ([FastAPI 0.163.2 testing tutorial](https://raw.githubusercontent.com/fastapi/fastapi/0.162.5/docs/en/docs/tutorial/testing.md)). So a test is an ordinary Python loop, and each call returns an answer it can read.

The route test uses that client. It builds the route from the recorded folder, sends it each of the 40 messages of `test_40`, and reads back the JSON of every answer. It does not start uvicorn, and it does not print message text. Every answer came back with status 200, and the test prints the keys of the first answer, which are the five fields listed in [the first section](#one-route-that-returns-the-hand-off). The counts it prints are the ones the route's rule predicts: the handed-off count, the flagged messages with no team, the kept messages with the gold team, and the team names outside the list. It also sends a body with no `message`, prints that reply's status, and asserts that its `detail` field is a list.

The test does not start a server, and that is a choice. A server started inside a test would need a free port, a wait for start-up and a clean shutdown, and each of those can fail for reasons that have nothing to do with the hand-off rule. A reader who wants a check against a running server can send the same 40 messages to that server and compare its counts with these. The output is the same on every run, because `setup_runtime`, which the test calls before it builds the route, fixes the seeds, requires deterministic algorithms and sets the default type to float32. A reader who runs the test again and gets different counts has found a change in the model, the data or the environment, and should stop and find out which before trusting the route.

Flag and team are checked together, for every answer in the sample: a flagged answer has no team, and a kept answer has one. A test that checked only the status code would pass a route that returned a team for every message, and it would miss the flag altogether. So the test prints both counts.

**Working rule:** Test the route through the test client on a committed sample, and print the counts that tie the flag to the team: the number of handed-off messages, the number of those with no team, and the number of team names outside the list. A test that checks only the status code does not show that the hand-off rule holds. Each claim the test makes should have a printed line behind it, because a line is what the book can quote and a reader can rerun. A claim without a line is a claim the reader has to take on trust.

## What goes wrong in production {#what-goes-wrong-in-production}

A route that passes its test can still fail once it runs under a server, and the test does not show these failures. The test runs once, on a committed sample, with the code as it stands. Production runs for as long as the service does, on messages nobody has sampled, and often under more than one process. Each of the failures below is a way the sample can be right while the service is wrong. The first is the copies of the model. The deployment page for uvicorn, the server this chapter's route is meant to run under, says that "Unlike gunicorn, uvicorn does not use pre-fork, but uses spawn" ([uvicorn 0.54.0 deployment guide](https://raw.githubusercontent.com/encode/uvicorn/0.54.0/docs/deployment/index.md)). A spawned worker is a new Python process. The module below builds the app when uvicorn imports it, so each worker that imports it loads its own copy of the model. We read the spawn quote as meaning that, and the chapter does not measure the memory a copy costs. A reader should count the copies: the number of worker processes, times the model's footprint on their own machine, before choosing how many workers to run.

Run from the examples folder, where `router_lib` and `data` are, the module that uvicorn imports is one line of setup and one line of app:

```python
# serve_route.py
import router_lib

app = router_lib.make_app("data/recorded/lora-full", threads=2)
```

The command is `uvicorn serve_route:app`. uvicorn takes its application as `"<module>:<attribute>"`, and its [settings page at the same tag](https://raw.githubusercontent.com/encode/uvicorn/0.54.0/docs/settings.md) gives `APP` in that form.

The second is a conflict between two options. The same page says, "The `--reload` and `--workers` arguments are mutually exclusive." A reader who develops with automatic reloading cannot run several workers in the same command. A reader who runs several workers is running without reload, so a change to the code needs a restart. Both facts come from the server's own page, and both should be in the reader's runbook.

The third is threads. Each worker process sets the thread count its `make_app` call asks for, so the total number of threads grows with the number of workers. On the two vCPUs this book budgets for, two threads per worker means one worker. We recommend that a reader count the total threads before adding a worker, and keep the total at or below the cores the machine has. That is a rule of thumb. This chapter does not measure what happens when the total goes above the core count, and it does not claim a number for it.

The fourth is the hand-off rate. The flag depends on the threshold that chapter 5 chose on the dev split, and on the messages the inbox sends. The route returns the flag on every answer, so the rate of flagged messages can be counted from the route's own output. If the rate moves, the route cannot say why. Its answers do not show whether the inbox's messages changed, whether the model changed, or whether the threshold was edited. [Chapter 9](version-watch-and-fall-back) watches this rate, and it adds versions and a fallback to people.

The fifth is the long messages, which [the section on what the route cannot score](#reject-what-the-route-cannot-score) covers. The cut at 128 tokens does not announce itself, so the count above has to be taken again on the messages the route actually receives.

The sixth is the lack of a record. A route that returns the flag but logs only the status code cannot show, after a change, how the hand-off rate moved. We recommend that each answer be logged with its flag, its adapter and its version, so that the rate can be counted from the logs as well as from the test.

What the chapter settled:

- The route returns the flag in every answer, and a team only when the flag is false.
- The route serves `banking77-lora-v1` in float32, with the threshold read from its manifest.
- Before it loads the model, `make_app` checks the adapter's files and its base pin against the manifest, and it builds no app on a mismatch ([chapter 9](version-watch-and-fall-back#check-the-adapters-files-against-its-manifest)).
- The route cuts each message at 128 tokens, the length the model was trained with.
- Each worker process holds one model and builds it with two threads, the book's choice for a two-vCPU budget.
- The version the route reports is the suffix of the adapter's name.

The hand-off holds or fails in the route, so the route computes the flag.

**Working rule:** Run one model per worker process, built with the thread count you have measured for your machine. Keep two threads per worker, as this book does, until your own measurement says otherwise. Count the worker processes before you count the model's memory on the machine, and keep `--reload` and `--workers` apart. Watch the hand-off rate the route returns, beside its errors, from the first day the route serves messages.

# Version the router, watch its hand-offs and fall back to people

By the end of this chapter, you can name a router by the parts that make its answers, check those parts before the route loads them, count the hand-off rate for each window of messages against a stated baseline, and say where each handed-off message goes.

[Chapter 8](serve-the-router-on-a-cpu) put the router behind a route, and each answer carries a team or a flag that sends the message to a person. A route that takes the bank's support messages for months raises three questions that the route cannot answer by itself. Which model is answering? Are its files the ones that were checked? Are the messages it hands off becoming more or fewer? This chapter gives each question a control and a printed count. The controls are the book's own.

This chapter covers what running your own model adds: versions, a fallback to people and retraining. Watching a service in general, with its dashboards, logs and alerts, is ground for a book on production monitoring. This chapter does not repeat it.

## Name the router by its parts {#name-the-router-by-its-parts}

A router is more than its model. It is the base model, the adapter trained on that base, the threshold that sends a message to a person, and the data and recipe that produced the adapter. Two routers that differ in any one of these can answer the same message differently. A log line that says only "the router" cannot tell them apart.

The adapter's files do not include the base model. The [PEFT checkpoint guide](https://raw.githubusercontent.com/huggingface/peft/v0.21.2/docs/source/developer_guides/checkpoint.md) says the saved weights hold only the adapter module's parameters, nothing from the base. The adapter therefore has to name the base it was trained on, and the name has to be exact. This book names the base by its commit, the pin in `examples/models.toml`, because a short commit hash will not do. The [Hugging Face download guide, at v2.4.0](https://huggingface.co/docs/huggingface_hub/v2.4.0/en/guides/download) requires the full-length hash whenever a commit is given. With no revision named, the download takes whatever the `main` branch holds. A branch name therefore moves with the branch, and it is not a pin. Chapter 5 recorded the base revision in the adapter's manifest, next to the threshold, the data commit and the recipe hash.

A version is written into the name. `banking77-lora-v1` is the recorded LoRA adapter, trained for one epoch on the training rows outside the dev split, version one. A new recording of the same recipe is `v2`, and it gets its own name and its own manifest. The manifest has no separate version field. The name is the version, and [chapter 8](serve-the-router-on-a-cpu#one-route-that-returns-the-hand-off)'s route reports the version as the suffix of the adapter's name. For `banking77-lora-v1` that suffix is `v1`. The int8 copy from [chapter 7](quantize-and-measure) is a different model, `banking77-lora-v1-int8`, and this route does not serve it. Its suffix would read `int8`, so a copy served by name needs a version field of its own.

The first check reads the manifest and compares its base revision with the pin. It prints the adapter's name first, followed by the comparison:

Example `adapter-manifest-check` (Python 3.12.15 · fastapi 0.163.2 · peft 0.21.3 · pydantic 2.14.4 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Show what the adapter check does before a route loads an adapter, and what it does to two edited copies.
The point rests on the "route may load" line and on the two "refused by make_app" lines."""
import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
import tempfile  # noqa: E402
import warnings  # noqa: E402
from pathlib import Path  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
# check_adapter hashes the manifest, the outputs and the LoRA settings; pinned is the manifest's digest matching its pin
manifest, s_ok, s_n, f_ok, f_n, pinned = router_lib.check_adapter(RECORDED)
# the base pin in models.toml, compared with the manifest's base revision
base = router_lib.pinned_revision(manifest["model"]) == manifest["model_version"]
print(f"adapter: {manifest['adapter']}")
print(f"base revision matches pin: {'yes' if base else 'no'}")
print(f"manifest matches pin: {'yes' if pinned else 'no'}")
print(f"settings matching: {s_ok}/{s_n}")
print(f"files matching: {f_ok}/{f_n}")
print("route may load: " + ("yes" if base and pinned and s_ok == s_n and f_ok == f_n else "no"))
with tempfile.TemporaryDirectory() as tmp:
    # changes a manifest field that is neither a LoRA setting nor an output file, so only the manifest digest sees it
    threshold = router_lib.edited_copy(RECORDED, Path(tmp) / "threshold", "manifest.json", "handoff_threshold", 0.0)
    # adds a key that LoraConfig does not know, to the config the settings are read from
    extra = router_lib.edited_copy(RECORDED, Path(tmp) / "extra", "adapter/adapter_config.json", "future_key", 1)
    for label, folder in [("threshold copy", threshold), ("extra-key copy", extra)]:
        _, s_ok, s_n, f_ok, f_n, pinned = router_lib.check_adapter(folder)
        print(f"{label}: manifest matches pin {'yes' if pinned else 'no'}, settings {s_ok}/{s_n}, files {f_ok}/{f_n}")
        print(f"{label} refused by make_app: {'yes' if router_lib.refused_by_make_app(folder) else 'no'}")
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        router_lib.load_classifier(extra / "adapter")
    named = [w for w in caught if "future_key" in str(w.message)]  # only the warning about the extra key
    print(f"warnings naming the extra key while PEFT loads the copy: {len(named)}")
    # the warning's first sentence: the text up to the first ". " in its message
    print(f"that warning: {named[0].category.__name__}: {str(named[0].message).split('. ')[0]}.")
```

Output:

```text
adapter: banking77-lora-v1
base revision matches pin: yes
manifest matches pin: yes
settings matching: 3/3
files matching: 3/3
route may load: yes
threshold copy: manifest matches pin no, settings 3/3, files 3/3
threshold copy refused by make_app: yes
extra-key copy: manifest matches pin yes, settings 3/3, files 2/3
extra-key copy refused by make_app: yes
warnings naming the extra key while PEFT loads the copy: 1
that warning: UserWarning: Unexpected keyword arguments ['future_key'] for class LoraConfig, these are ignored.
```

The name carries the version, banking77-lora-v1, and the base revision matches the pin (yes). These are the two parts a reader can check against the pins before any file is read.

**Working rule:** Name a router by its base revision, its adapter's name with the version in that name, its data commit, its threshold and its recipe hash, and write all five into the adapter's manifest before the route serves it. Compare the base revision in the manifest with the pin in `examples/models.toml` each time you load the adapter. A pin that has moved means a different base model, and the adapter needs a new check before it serves.

## Check the adapter's files against its manifest {#check-the-adapters-files-against-its-manifest}

A load is not a version check. The [PEFT checkpoint guide](https://raw.githubusercontent.com/huggingface/peft/v0.21.2/docs/source/developer_guides/checkpoint.md) notes that most entries in a LoRA config are not needed to load the model, so a load can succeed with a config that someone has changed by hand. A successful load does not show that the adapter is the pinned one. The extra-key copy below still loads, and that load raises a warning naming the extra key. A mismatched base does fail at load, with a state-dict error, in the research probe; this example does not print that case.

So the check compares the adapter with its manifest, in three parts. It computes the SHA-256 of `manifest.json` itself, and compares it with the digest pinned for the adapter's name in `examples/adapter-pins.json`. The pin covers every field the manifest holds, including the threshold, the base revision and the output digests. It computes the SHA-256 of each output file the manifest lists, and compares it with the digest the manifest records. It also reads the three LoRA settings in the config, the rank, the alpha that sets the scaling and the target layers, and compares them with the manifest. The printed lines give how many of each agree, and whether the route may load the folder.

For the recorded adapter, the manifest matches its pin (yes), 3/3 settings and 3/3 files agree, so the route may load it.

The check also runs on two edited copies, and each one gets its own line. The first has its `handoff_threshold` changed to 0 in the manifest, and its line is manifest matches pin no, settings 3/3, files 3/3. Its settings and files still agree, because the threshold is not a setting and no output file changed. Only the manifest's own digest sees the change, so the copy is refused. The second has one extra key added to its config, and its line is manifest matches pin yes, settings 3/3, files 2/3. Its settings still agree, and the file check refuses it, because the config file's hash no longer matches. PEFT loads that copy anyway, and of the warnings the load raises, 1 names the extra key. In full, it reads UserWarning: Unexpected keyword arguments ['future_key'] for class LoraConfig, these are ignored. The load returns a model with the key ignored, and that warning is all PEFT says about it. A warning is not a refusal. The check is what refuses the copy.

The route in [chapter 8](serve-the-router-on-a-cpu#load-the-model-once-at-startup) runs this check. Its `make_app` calls `check_adapter` before it loads the model, and it also compares the manifest's base revision with the pin. It raises an error on any mismatch, so a copy that fails is refused before the model is in memory. The threshold copy is refused by `make_app` (yes), and so is the extra-key copy (yes). The check trusts the repository. Its pinned digest sits in `examples/adapter-pins.json`, next to the code, so a reader who edits the folder and not the pins file is refused, and a reader who edits both is not. The pins file is therefore reviewed like any other code.

**Working rule:** Run the check before any route loads an adapter. Pin the SHA-256 of the manifest with the code, count the matching settings and files, print each count with its total, and refuse the load when any count falls short or the manifest digest does not match. Take the hashes from the files on disk, and check and load from the same files, with nothing writing to them in between.

## Watch the hand-off rate {#watch-the-hand-off-rate}

The hand-off rate is the number of messages below the threshold divided by all the messages in a window. [Chapter 2](measure-the-routing-task#three-measures-each-with-its-denominator) defined it, and this chapter watches it over time. A watch needs three things, a window, a baseline and a rule for when a window has moved too far from the baseline.

The baseline is the hand-off rate on the dev split at the adapter's threshold. `dev_handed_off` computes it by routing all the dev messages once and counting the hand-offs, which comes to 148/1003 (14.8%). The threshold example in [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) prints the same count, 148/1003 (14.8%). The two agree, so the baseline is the one that chapter chose the threshold on.

A baseline from dev is not a forecast for the inbox. The dev split is drawn from training rows, team by team. The test sample is drawn from the test split. Neither is a sample of the inbox's traffic, and the mix of teams in live messages is not known. A gap between a window and the baseline is therefore a question to ask, and it does not answer the question itself.

The drift rule is this book's choice, not a figure from a source. A window drifts when its hand-off rate is more than a stated number of points from the baseline, in either direction. The rule as printed reads more than 10 points from the baseline. Whoever runs the watch should choose a number of their own and write it down before reading the windows.

The committed test sample, `test_1001`, has no dates, so the windows are cut from its first 100 messages. The first two windows, of 25 messages each, are the first messages of the sample, in sample order. Weeks 3 and 4 are a replay, not new traffic. For those two weeks, the example takes the sample's next messages, sorts them by the router's confidence from lowest to highest, and splits the sorted list in two. The example marks the replay as simulated in its printed line, simulated replay of sample messages 51 to 100. The replay is built to produce a spread of confidences, so it shows what the rule does with a skewed window, and like the baseline it says nothing about the inbox.

Example `handoff-windows` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Watch the router's hand-off rate week by week on a fixed, ordered sample of test_1001, against the dev baseline.
The point rests on the "drift" lines and on the "lowest confidence" and "right" lines of weeks 3 and 4, which the output
labels as a simulated replay."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
# the threshold is read from the manifest, not typed here
threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]
tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")

handed_dev, n_dev = router_lib.dev_handed_off(model, tokenizer, threshold)
base = 100 * handed_dev / n_dev
print(f"dev hand-off rate (baseline): {handed_dev}/{n_dev} ({base:.1f}%)")
print(f"drift rule: more than {router_lib.DRIFT_POINTS:.0f} points from the baseline")
print("weeks 3 and 4: simulated replay of sample messages 51 to 100")
test = router_lib.read_rows("test")
# simulated=True (the default) adds weeks 3 and 4: messages 51 to 100, lowest confidence first, ties by position
for w, (rows, preds, confs) in enumerate(router_lib.watch_weeks(model, tokenizer, threshold), 1):
    n = len(rows)
    handed = sum(router_lib.handoff(c, threshold) for c in confs)
    rate = 100 * handed / n
    # the messages the router keeps, each with its gold team: test row i holds (text, gold team)
    kept = [(p, test[i][1]) for i, p, c in zip(rows, preds, confs) if not router_lib.handoff(c, threshold)]
    right = sum(p == g for p, g in kept)
    print(f"week {w} messages: {n}")
    print(f"week {w} handed off: {handed}/{n} ({rate:.1f}%)")
    # abs makes the rule two-sided, so a fall in the rate drifts as a rise does
    print(f"week {w} drift: {'yes' if abs(rate - base) > router_lib.DRIFT_POINTS else 'no'}")
    print(f"week {w} lowest confidence: {min(confs):.3f}")
    print(f"week {w} right among messages not handed off: {right}/{len(kept)}")
```

Output:

```text
dev hand-off rate (baseline): 148/1003 (14.8%)
drift rule: more than 10 points from the baseline
weeks 3 and 4: simulated replay of sample messages 51 to 100
week 1 messages: 25
week 1 handed off: 6/25 (24.0%)
week 1 drift: no
week 1 lowest confidence: 0.479
week 1 right among messages not handed off: 17/19
week 2 messages: 25
week 2 handed off: 4/25 (16.0%)
week 2 drift: no
week 2 lowest confidence: 0.378
week 2 right among messages not handed off: 19/21
week 3 messages: 25
week 3 handed off: 9/25 (36.0%)
week 3 drift: yes
week 3 lowest confidence: 0.246
week 3 right among messages not handed off: 16/16
week 4 messages: 25
week 4 handed off: 0/25 (0.0%)
week 4 drift: yes
week 4 lowest confidence: 0.981
week 4 right among messages not handed off: 25/25
```

The windows are small because the sample is small, with four windows of 25 messages each. In a window this size, one message is 4 points of the rate, so one message can move a window across the rule's line. A watch on a real inbox needs windows large enough that one message does not decide the flag. The window size belongs in the report, stated before the windows are read.

Weeks 1 and 2 stay inside the rule, and the replay drifts in both directions. The four windows come out like this, each with its count and its flag.

- Week 1 hands off 6/25 (24.0%), and its drift flag is no.
- Week 2 hands off 4/25 (16.0%), and its drift flag is no.
- Week 3 hands off 9/25 (36.0%), and its drift flag is yes.
- Week 4 hands off 0/25 (0.0%), and its drift flag is yes.

Week 4 is the case a reader should look at closely. A rate that falls to zero drifts under this rule, because the rule treats a fall as a change in the same way it treats a rise. The rate is zero since the lowest confidence in week 4 is 0.981, above the threshold, so no message in the window is handed off. The router was right on 25/25 of the messages it kept. The window is one of confident, correct answers, and the drift flag reports that its hand-off rate has moved, not that the router has made mistakes.

Week 3 is the other kind of window. Its lowest confidence is 0.246, its hand-offs rise to 9/25 (36.0%), and the router is right on 16/16 of the messages it keeps. A rise in hand-offs means more messages reach a person. It does not, by itself, show which cause produced the rise, whether a change in the messages, in the teams that appear or in the model. The route cannot say which, because its answers carry no record of the messages, and [chapter 8](serve-the-router-on-a-cpu#what-goes-wrong-in-production) says the same about the rate it returns.

**Working rule:** Count the hand-offs in each window of the same size, print each count with its denominator, and compare each rate with the dev baseline at the adapter's threshold. Set the drift rule, and its number of points, before you read the windows, and write it into your report. Treat a drifting window as a reason to read its messages and their teams, not as a verdict on the router.

## Sample handed-off messages for people {#sample-handed-off-messages-for-people}

The [hand-off watch](#watch-the-hand-off-rate) counts how many messages go to a person. It leaves two questions open, whether the router was right on the messages it handed off and whether it was right on the messages it kept. This section describes how the first would be answered. It is a next step. This chapter draws no sample, reports no labels and gives no number for it.

A person would label a sample of the handed-off messages, drawn with a fixed seed, and the labels would say which team the person chooses for each. Set against the team the router's model would have chosen, they would show how often the two differ on the messages the router was unsure of. A router that is unsure about a message is the one whose choices people should check most, so this sample is the one to draw first. A reader who builds it can use the gold labels of a public dataset as a stand-in for a person's labels, and should say so in the report.

A sample of handed-off messages does not say how the router does on the messages it keeps. The keeps are the messages whose answers the route returned, and the hand-off never sends them to a person. Their accuracy can be estimated only when people check some of them too, after the route has already answered them. The check on kept messages is a second sample, drawn in the same way, and it is not a by-product of the sample of handed-off messages. The measure is accuracy on the messages not handed off, with its count and its denominator, as [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) set it.

**Working rule:** When you build the sample, draw it with a fixed seed, print how many messages were labelled, and report how often the router's team and the person's team agree, with its count. Label a sample of kept messages on the same schedule, and read the router's accuracy on kept messages only from that sample. Never read the kept rate off the handed-off sample.

## Fall back to a person {#fall-back-to-a-person}

The route returns a flag and no team for a handed-off message, as [chapter 8](serve-the-router-on-a-cpu#one-route-that-returns-the-hand-off) built it. Something must still happen to that message. The rule in this chapter is that a handed-off message always goes to a person. The recorded answer of the stand-in, Qwen3-4B-Instruct-2507, may travel with a handed-off message as a suggestion, labelled as the stand-in's, and the person decides whether to use it. The stand-in's answers on `test_1001` are recorded in `standin-baseline` ([chapter 2](measure-the-routing-task#the-baseline-from-recorded-answers)). The router's team is never returned for a handed-off message.

The route's JSON has `team`, `confidence`, `handed_off`, `adapter` and `version`, and it has no field that says who answered. A fallback answer must therefore never go out in that shape. A reader who wants a suggestion in the response adds a field of their own, on purpose, and says in the response what the field is.

The book does not teach a call to a hosted model as the fallback, and it sets no threshold for the stand-in. Its answers come from a recording, which the fallback script reads. That script measures how often the suggestion is right, and it does not decide anything from that measure.

Example `fallback-to-larger-model` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Hand each handed-off message to a person, with the stand-in's recorded answer as a labelled suggestion.
The point rests on the "team returned for a handed-off message" line and the "suggestions right" line."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
# the threshold is read from the manifest, not typed here
threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]
tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")
test = router_lib.read_rows("test")
standin = router_lib.standin_rows()  # a dict from test row index to the stand-in's answer, for the rows it answered
rows, handed = [], []
# simulated=False keeps weeks 1 and 2 only: the replay belongs to the watch, not to this example
for weekly_rows, _, confs in router_lib.watch_weeks(model, tokenizer, threshold, simulated=False):
    rows += weekly_rows
    handed += [i for i, c in zip(weekly_rows, confs) if router_lib.handoff(c, threshold)]

# the list of records stands in for the person's queue, which the book does not build
# the suggestion is None where the stand-in has no recorded answer, and the team is always None
queue = [{"index": i, "team": None, "suggestion": standin[i]["label"] if i in standin else None} for i in handed]
suggested = [r for r in queue if r["suggestion"] is not None]
right = sum(r["suggestion"] == test[r["index"]][1] for r in suggested)  # index 1 of a test row is its gold team

print(f"messages routed: {len(rows)}")
print(f"handed off by the router: {len(handed)}/{len(rows)} ({100 * len(handed) / len(rows):.1f}%)")
print(f"sent to a person: {len(queue)}/{len(handed)}")
print(f"suggestions attached: {len(suggested)}/{len(handed)}")
print(f"suggestions right: {right}/{len(suggested)}")
print(f"team returned for a handed-off message: {sum(r['team'] is not None for r in queue)}")
```

Output:

```text
messages routed: 50
handed off by the router: 10/50 (20.0%)
sent to a person: 10/10
suggestions attached: 10/10
suggestions right: 4/10
team returned for a handed-off message: 0
```

The example routes the first two windows, 50 messages in all, and the router hands off 10/50 (20.0%) of them. Each handed-off message gets one record, and the output counts the records sent to a person. Where the stand-in has a recorded answer, the record carries it as a suggestion. Of the records, 10/10 carry a suggestion, and the count of teams returned for a handed-off message is 0. The list of records stands in for the person's queue. The book does not build that queue, and a reader who builds one should keep each record with its message.

The suggestions are right on 4/10 of the handed-off messages that carry one. That is fewer than half, and it is the reason the suggestion is shown to a person and never used as the answer. The fallback script does not measure the stand-in on messages the router keeps. Every handed-off message in this sample carries a suggestion, and the chapter does not claim the same for the inbox.

**Working rule:** Send each handed-off message to a person. A recorded answer from the stand-in may come with it, labelled as a suggestion. Never return a team for it, and never put a suggestion in the route's response shape without a field that says who answered. Count the suggestions that are right, with the count and its denominator, before a reader decides how much weight to give them.

## Retrain on a trigger {#retrain-on-a-trigger}

A drifting window is a signal to look, and it does not on its own start a retraining. The book sets three triggers, and this section states each one. The first is drift, the rule in this chapter's watch applied to consecutive windows. The book retrains when two consecutive windows drift, a number no source gives, and you may choose another, provided you state it in advance. The second trigger is a team added to the 77 the router returns. The router has no training rows for it, and people's labels would show it as a team outside the 77. The third is a rise in wrong teams on messages the router keeps, confirmed in the kept-message sample from the previous section. It is not the sample of handed-off messages, whose teams are never returned, so wrong teams there are expected and do no harm. The retrain script below exercises only the first. The recorded data holds no added team and no person's labels, so the second and third are rules without a test here.

Example `retrain-trigger` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Decide whether to retrain, as a rule over the weekly drift flags from handoff-windows. The rule is the book's choice,
not a sourced threshold: retrain when two consecutive weeks drift, that is, their hand-off rate is more than
router_lib.DRIFT_POINTS from the dev baseline. The point rests on the "retrain trigger" line."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
RECORDED = router_lib.HERE / "data" / "recorded" / "lora-full"
# the threshold is read from the manifest, not typed here
threshold = json.loads((RECORDED / "manifest.json").read_text())["handoff_threshold"]
tokenizer, model = router_lib.load_classifier(RECORDED / "adapter")
handed_dev, n_dev = router_lib.dev_handed_off(model, tokenizer, threshold)
base = 100 * handed_dev / n_dev

drifting, run, longest = [], 0, 0
# the default simulated=True: weeks 3 and 4 are the replay, the same four windows the watch reads
for rows, _, confs in router_lib.watch_weeks(model, tokenizer, threshold):
    rate = 100 * sum(router_lib.handoff(c, threshold) for c in confs) / len(rows)
    flag = abs(rate - base) > router_lib.DRIFT_POINTS  # two-sided, as in the watch
    drifting.append(flag)
    run = run + 1 if flag else 0
    longest = max(longest, run)
    print(f"week {len(drifting)} drift: {'yes' if flag else 'no'}")

print(f"weeks drifting: {sum(drifting)}/{len(drifting)}")
print(f"longest run of drifting weeks: {longest}")
print("rule: retrain when two consecutive weeks drift")
print("retrain trigger: " + ("fired" if longest >= 2 else "not fired"))
```

Output:

```text
week 1 drift: no
week 2 drift: no
week 3 drift: yes
week 4 drift: yes
weeks drifting: 2/4
longest run of drifting weeks: 2
rule: retrain when two consecutive weeks drift
retrain trigger: fired
```

The retrain script applies the drift rule to the same four windows as the watch, in order, and finds 2/4 windows that drift, with a longest run of 2. Weeks 3 and 4 drive that result, since they are the replay from the watch. The trigger fired because of how the replay was built, so the run shows the rule working as stated, not anything the inbox did.

When a retraining does start, it follows the same path as the first adapter. It is a recording in `recordings/`, as `banking77-lora-v1` was, written by a script there, and it writes a manifest with the same fields. It is named as the next version of the same recipe, so the retrained adapter is `v2`. Its threshold is chosen again on the dev split by the rule in [chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev), never on the test sample. The retrained folder then needs its manifest's digest added to `examples/adapter-pins.json`, as a change to the repository that someone reviews. Until then, `make_app` refuses to build the app from it. The old adapter stays in place until `make_app` builds an app from the new folder.

**Working rule:** Retrain when the trigger you stated is met, and not on one drifting window. Write the trigger, its number and the rule beside the windows that met it. A retrained adapter is a recording named as the next version, with its manifest digest pinned and its threshold chosen again on dev. Keep the previous adapter until the new one passes the check in `make_app`.

## What goes wrong in production {#failures-in-production}

- **A manifest is edited by hand.** The manifest's digest is pinned, so the check refuses it. A check that trusts the repository can be defeated only by editing the pins file too, which is a change someone reviews.
- **A pin moves.** The base pin in `examples/models.toml` can change under an adapter whose manifest still names the old revision. `make_app` compares the two and refuses the build.
- **A fallback routes on the wrong scale.** A confidence from a model that was never calibrated to the threshold is not a gate. The book's fallback therefore never routes, and the suggestion travels only with a person's message.
- **The window is too small.** In a window of 25 messages, one message is 4 points. A flag then reflects one or two messages, and a reader who acts on it has acted on noise.
- **The rate moves and nothing explains it.** The route's answers carry no record of the messages, so the rate's movement is a question to answer from the messages and their teams, with the samples this chapter describes as next steps.

You notice the first kind of failure when the app is built. A refused folder raises an error at startup, so a deployment that builds the app when it starts sees a bad folder before any message fails. The other failures show in the counts. A moved hand-off rate shows in a window's count, and the drift rule reports it. A fallback that answers the wrong messages shows in the suggestion count, and a sample of people's labels shows the rest.

**Working rule:** Print each control's count where a reader can see it at startup and in each window, and keep the counts that would reveal a failure in the same report as the rate they explain.

## The decisions this chapter made {#the-decisions-this-chapter-made}

- A handed-off message always goes to a person. The stand-in's answer, when there is one, is a labelled suggestion, and the chapter counts how often it is right.
- The version check pins the SHA-256 of each adapter's `manifest.json` in `examples/adapter-pins.json`, and `make_app` refuses to build on any mismatch or a moved base pin.
- The drift rule is 10 points from the dev baseline, in either direction, on windows of 25 messages.
- A retrain is due when two consecutive windows drift. The second and third triggers are stated rules, not tested here.

[Chapter 10](when-a-small-model-is-the-wrong-choice) uses the same measures as this chapter.

All four are the book's choices; see [the drift rule](#watch-the-hand-off-rate). Together the controls check which model answers, count what it hands off and send each handed-off message to a person. The fallback rule follows the rule from [chapter 1](route-a-message#what-the-router-returns) that a flagged message goes to a person. The pin rule trades a review of the repository for a check that a load alone cannot make.

**Working rule:** Write each decision above into the report with its number and whether it is the book's choice or a sourced figure. Change a decision only by changing the rule it names, and rerun the examples that print its counts.

# When a small model is the wrong choice

By the end of this chapter, you can say where a small router still loses to the stand-in and where a hosted model is expected to do better, count each option's monthly cost from inputs you state with their dates, find the volume at which a machine of your own becomes the cheaper route, and name the requirements that decide the question before any cost does.

A small router can beat a larger model on most of a set of messages and still be the wrong choice for some messages, some teams or some volumes. The bank's support inbox ([chapter 1](route-a-message)) has 77 teams. Over nine chapters this book built a classifier head for it, served the head with a route that hands off the unsure messages and with the checks that version the router and send its hand-offs to a person, and tried a distilled student and an int8 copy, measuring both without shipping either. The chapters that measure the router score it against the stand-in ([chapter 1](route-a-message#ask-a-model-for-one-team)).

## Where the small model still loses {#where-the-small-model-still-loses}

[Chapter 2](measure-the-routing-task#the-baseline-from-recorded-answers) set the stand-in's score as the baseline. This section puts the router beside it on the same messages: all 1001 messages of `test_1001`, the fixed sample that every chapter's samples come from ([chapter 2](measure-the-routing-task#dev-test-and-fixed-samples)). The router is the classifier `banking77-lora-v1`, run live.

Example `router-on-test-1001` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""The router and the stand-in on all 1,001 messages of test_1001, the committed test order. The prose rests on the
lines for the router's wrong answers sent as answers and for the messages the stand-in gets right and the router wrong.
The stand-in's answers are recorded; the router runs live on 2 threads, and its first 200 messages are the test_200 sample."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
# Runs the router live on the sample and reads the stand-in's answers from its recording, then scores both.
c = router_lib.compare_router_and_standin("test_1001")
n = c["messages"]
print(f"messages on test_1001: {n}")
print(f"router banking77-lora-v1 correct: {c['router_correct']}/{n} ({c['router_correct'] / n:.1%})")
print(f"stand-in correct: {c['standin_correct']}/{n} ({c['standin_correct'] / n:.1%})")
print(f"router handed off: {c['handed_off']}/{n} ({c['handed_off'] / n:.1%})")
print(f"right for the stand-in, wrong for the router: {c['standin_right_router_wrong']}/{n}")
print(f"of those, handed off by the router: {c['of_those_handed_off']}/{c['standin_right_router_wrong']}")
print(f"wrong for the router and sent as an answer: {c['router_wrong_sent']}/{n} ({c['router_wrong_sent'] / n:.1%})")
print(f"right for the router, wrong for the stand-in: {c['router_right_standin_wrong']}/{n}")
```

Output:

```text
messages on test_1001: 1001
router banking77-lora-v1 correct: 847/1001 (84.6%)
stand-in correct: 560/1001 (55.9%)
router handed off: 146/1001 (14.6%)
right for the stand-in, wrong for the router: 48/1001
of those, handed off by the router: 22/48
wrong for the router and sent as an answer: 76/1001 (7.6%)
right for the router, wrong for the stand-in: 335/1001
```

On the whole set the router is ahead, right on 847/1001 (84.6%) against the stand-in's 560/1001 (55.9%), which is the baseline from chapter 2. Yet the stand-in is right and the router wrong on 48/1001 messages. The router hands off some of those, so a person sees them, and the rest reach the caller as wrong answers. The reverse disagreement, where the router is right and the stand-in wrong, has its own count in the output.

The count that matters most for the inbox is the wrong answers the router sends to a caller: 76/1001 (7.6%). Those are messages it kept, because their confidence cleared the threshold. [Chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev) chose that threshold on the dev split, and [chapter 9](version-watch-and-fall-back#fall-back-to-a-person) says where what the router hands off goes, which is a person.

Example `where-the-router-still-loses` (Python 3.12.15 · peft 0.21.3 · torch 2.15.0+cpu · torchao 0.18.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""Where the router still loses: test_200 scored for the router and the stand-in. The prose rests on the count of
messages the stand-in gets right and the router wrong, and on the wrong answers the router sends as answers.
The stand-in's answers are recorded; the router runs live on 2 threads."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import json  # noqa: E402

import torch  # noqa: E402
torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
ADAPTER = router_lib.HERE / "data" / "recorded" / "lora-full"
# The threshold chapter 5 chose on dev, as the adapter's manifest records it.
threshold = json.loads((ADAPTER / "manifest.json").read_text())["handoff_threshold"]
# Keyed by test row index, not by position in the sample, so each lookup uses the sample's row numbers.
answers = router_lib.standin_rows()
test = router_lib.read_rows("test")  # (text, label index) per row
sample = router_lib.read_sample("test_200")
golds = [test[i][1] for i in sample]  # index 1 is the gold team, as an index
standin_preds = [answers[i]["label"] for i in sample]  # the label is the team's index, the same form as the gold label
tokenizer, model = router_lib.load_classifier(ADAPTER / "adapter")
preds, confs = router_lib.classify(model, tokenizer, [test[i][0] for i in sample])
n = len(sample)
router_right = [p == g for p, g in zip(preds, golds)]
standin_right = [p == g for p, g in zip(standin_preds, golds)]
handed_off = [router_lib.handoff(c, threshold) for c in confs]  # true below the threshold; a NaN confidence is handed off too
lost = [i for i in range(n) if standin_right[i] and not router_right[i]]
sent_wrong = sum(not r and not h for r, h in zip(router_right, handed_off))  # wrong and not handed off: what a caller receives
print(f"messages on test_200: {n}")
print(f"router banking77-lora-v1 correct: {sum(router_right)}/{n} ({sum(router_right) / n:.1%})")
print(f"stand-in correct: {sum(standin_right)}/{n} ({sum(standin_right) / n:.1%})")
print(f"router threshold from manifest: {threshold:.2f}")
print(f"router handed off: {sum(handed_off)}/{n} ({sum(handed_off) / n:.1%})")
print(f"right for the stand-in, wrong for the router: {len(lost)}/{n} ({len(lost) / n:.1%})")
print(f"of those, handed off by the router: {sum(handed_off[i] for i in lost)}/{len(lost)}")
print(f"wrong for the router and sent as an answer: {sent_wrong}/{n} ({sent_wrong / n:.1%})")
print(f"right for the router, wrong for the stand-in: {sum(r and not s for r, s in zip(router_right, standin_right))}/{n}")
```

Output:

```text
messages on test_200: 200
router banking77-lora-v1 correct: 172/200 (86.0%)
stand-in correct: 109/200 (54.5%)
router threshold from manifest: 0.63
router handed off: 33/200 (16.5%)
right for the stand-in, wrong for the router: 6/200 (3.0%)
of those, handed off by the router: 3/6
wrong for the router and sent as an answer: 11/200 (5.5%)
right for the router, wrong for the stand-in: 69/200
```

`test_200` is the first 200 messages of `test_1001`, a prefix the count script checks, so these counts are part of the full run and not a second sample. On it, the stand-in is right and the router wrong on 6/200 (3.0%) messages, and the router hands off 3/6 of them.

The hand-off holds back the wrong answers. The router sends 11/200 (5.5%) wrong answers on these 200 messages. On the messages it keeps, its accuracy is 156/167 (93.4%), and 33/200 (16.5%) of all 200 go to a person.

Forcing a small model onto the team list keeps every answer a team, and does not make the answer right. [Chapter 4](prompt-and-constrain#measure-the-constrained-answers) asked the two smaller models for a team that way. The table gives each prompted variant's count on the 200 messages of `test_200`:

| Prompted variant | Right on `test_200` |
| --- | --- |
| SmolLM2-135M-Instruct, raw prompt | 9/200 (4.5%) |
| SmolLM2-135M-Instruct, chat template | 11/200 (5.5%) |
| SmolLM2-360M-Instruct, raw prompt | 12/200 (6.0%) |

The raw 135M model gave one answer more often than any other (card_arrival, 165/200), and that team is the gold team for 0/200 of the messages. A model that leans on one answer is a prior, not a router. [Chapter 4](prompt-and-constrain#why-the-prior-wins) shows why a prompt cannot make up for a model this size.

The stand-in fails in patterns too. Across all 1001 messages, its answers never name 4 of the 77 teams, so every message whose gold team is one of them goes to another team. Two confusions tie for its most common error. The clearest is the team `Refund_not_showing_up`, which is answered as `request_refund` on 13/13 of its messages. A pair like that is where a labelled set has to be read before any model is trusted on it, and [chapter 2](measure-the-routing-task#where-the-labels-collide) lists the pairs for every candidate.

A student copies its teacher, including the teacher's mistakes. [Chapter 6](distill-a-larger-model#when-the-teacher-is-wrong) distilled the stand-in's answers into a student. The teacher agreed with the gold label on 549/1000 (54.9%) of the training rows it labelled, so the student learned from labels that were wrong almost half the time. The student reached 69/200 (34.5%) on `test_200`, and the same recipe trained on the gold labels of the same rows reached 77/200 (38.5%).

The student also falls below its teacher, which reached 109/200 (54.5%) on `test_200`, and below the gold-label control, by a gap this sample cannot settle ([chapter 6](distill-a-larger-model#measure-agreement-not-only-accuracy)). The student is right where the control is wrong on 24/200 messages, and the control where the student is wrong on 32/200.

A new team is another place the small model loses. A classifier head has one output for each team it was trained on, so a team without an output cannot be routed by it. Adding one means labelled messages for that team, a training run with the recipe from [chapter 5](fine-tune-with-lora), and the checks from [chapter 9](version-watch-and-fall-back#retrain-on-a-trigger) before the router sends a message to it. Whether a prompt change could add a new team is a question for the reader to test. This book did not measure a hosted model, and the table later in this chapter lists the question as open.

Put together, the small models here did three things: a prompted model leaned on one answer, a student copied its teacher's mistakes, and a team with no output could not be routed. None of these is a fault in the router's arithmetic. The stand-in did one thing more: it never named 4 teams and confused a pair. That is the stand-in's, not either option's. A larger threshold trades the router's 11/200 (5.5%) wrong answers on `test_200` for hand-offs ([chapter 5](fine-tune-with-lora#choose-the-threshold-on-dev)), but it gives no team an output and fixes neither the teacher's errors nor the stand-in's. Each changes what a caller sees: a wrong team, a handed-off message, or no team at all.

**Working rule:** Before you route with a small model, run it and the hosted model on the same fixed sample. Count the messages each gets right that the other gets wrong, and the wrong answers that would reach a caller. Count a prompted model's answers against the team list, and count how often its most common answer is the gold team. Do not route a team that has no labelled messages of its own, and do not train a student on a teacher's labels before you have counted how often the teacher agrees with the gold labels.

## Count the bill and the machine {#count-the-bill-and-the-machine}

A hosted model is paid per message. A hosted model's monthly bill is the number of messages times what one message costs, and one message costs its input tokens at the input price plus its output tokens at the output price. A price list written per million tokens gives the price per token when divided by one million. A routing message is long on the input side, because the prompt and the message are both counted, and short on the output side, because the answer is one team name.

A machine is paid for whether it is busy or not. A machine's monthly bill is its hours times its hourly rate. The number of messages a machine can route in a month is its throughput, in messages per second, times the seconds in a month, times its utilisation, which is the share of the month it is busy. The machine's cost per message therefore depends on how many messages it routes, and the comparison needs a volume before it gives an answer.

The three formulas below hold for any pair of options.

```
hosted cost per month = messages × (input tokens × input price + output tokens × output price)
machine cost per month = hours × machine cost per hour
machine capacity per month = messages per second × seconds per month × utilisation
```

Each input needs a value, and each price needs a date and a source. The book quotes no hosted price, because it has none with both a source and a date that a reader could check. Read your provider's price page on the day you run the comparison, and write the price, its date and its address beside the number.

Input tokens also need a tokenizer. A hosted provider's count may differ from an open model's count of the same text, and this book did not measure one. The routing prompt is what a hosted model would receive for each message: the list of 77 team names and the message, as `router_prompt` builds it. The monthly example counts it with the tokenizer of SmolLM2-135M-Instruct, which gives 587 tokens per message, and the example labels that count as a stand-in for the provider's.

Throughput is the input that a recording has to supply, because it depends on the machine. The recording names its machine and saves the rates it measured. On a cloud machine with 4 vCPUs and 15 GiB of memory, at 4 threads, the stand-in routed 0.32 messages a second at one message per call. The router's classifier, `banking77-lora-v1`, in float32, routed 19.0 messages a second at the same call size. Its rate in batches of 16 is in the output as well.

Only the one-per-call pair is like for like. The classifier in batches of 16 takes 16 messages in one forward pass, and the stand-in takes one message per call with its whole router prompt. So the batched ratio of 104.0 mixes model size with batching, and this chapter does not use it for the comparison. The comparison uses the one-per-call ratio of 60.8. Each copy got one timed pass over the 200 messages of `test_200`, after a warm-up of one message, with tokenization inside the timed span. The ratios come from unrounded rates, so they are not exactly the quotient of the printed rates.

Example `cost-ratio-from-recording` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""The throughput rates and ratios as recorded, and the machine they were measured on. The prose rests on the
one-per-call ratio line, the like-for-like comparison; the batched lines differ in call size. Reads the recording only; no model runs."""

import json

import router_lib

folder = router_lib.HERE / "data" / "recorded" / "throughput"
ratios = json.loads((folder / "ratios.json").read_text())
rates = router_lib.recorded_rates()
manifest = json.loads((folder / "manifest.json").read_text())

print(f"machine: {manifest['machine'].split(', ')[0]}")  # the machine's description is cut at its first comma, which keeps the name and drops the rest
print(f"threads: {manifest['settings']['n_threads']}")
print(f"messages on test_200: {ratios['messages']}")
print(f"stand-in rate, one per call (messages a second): {rates['standin_one_message_per_call']}")
print(f"classifier fp32 rate, one per call (messages a second): {rates['classifier_fp32_one_message_per_call']}")
print(f"classifier fp32 rate, batches of 16 (messages a second): {rates['classifier_fp32_batches_of_16']}")
print(f"classifier int8 rate, batches of 16 (messages a second): {rates['classifier_int8_batches_of_16']}")
# Each ratio is computed from unrounded rates, so it is not the quotient of the printed rates.
print(f"fp32 over stand-in, one per call, throughput ratio: {ratios['classifier_fp32_one_at_a_time_over_standin']}")
print(f"fp32 over stand-in, batches of 16, throughput ratio: {ratios['classifier_fp32_over_standin']}")
print(f"int8 over stand-in, batches of 16, throughput ratio: {ratios['classifier_int8_over_standin']}")
print(f"int8 over fp32, batches of 16, throughput ratio: {ratios['classifier_int8_over_fp32']}")
print(f"predictions agreeing, fp32 and int8: {ratios['agreement_fp32_int8']}/{ratios['messages']}")
```

Output:

```text
machine: a cloud machine with 4 vCPUs and 15 GiB of memory
threads: 4
messages on test_200: 200
stand-in rate, one per call (messages a second): 0.32
classifier fp32 rate, one per call (messages a second): 19.0
classifier fp32 rate, batches of 16 (messages a second): 33.0
classifier int8 rate, batches of 16 (messages a second): 29.0
fp32 over stand-in, one per call, throughput ratio: 60.8
fp32 over stand-in, batches of 16, throughput ratio: 104.0
int8 over stand-in, batches of 16, throughput ratio: 91.8
int8 over fp32, batches of 16, throughput ratio: 0.9
predictions agreeing, fp32 and int8: 200/200
```

On this machine, at this thread count, the int8 copy was no faster than float32. Its ratio over float32 in batches of 16 is 0.9. [Chapter 7](quantize-and-measure#measure-speed-in-a-recording) said that fewer bytes do not settle speed, and this recording measures the speed. The two copies agreed on 200/200 predictions. The recording ran at that thread count, and this book's serving budget is two vCPUs, so a reader's rates should come from a recording on their own machine at their own thread count.

The bill needs a rate for the router, and the recording gives one: the classifier at one message per call, timed as the stand-in was. The route handles one message per request, so the bill uses that rate. A reader replaces it with one taken on their own machine.

Example `monthly-bill-from-labelled-inputs` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""The monthly bill for a hosted model and for the router on one machine, from inputs the reader sets. The hosted bill,
the machine's bill and the cheaper-route line are the point the prose rests on. The prices, volume, machine cost
and utilisation are example values, not published prices; the token counts are live and the router's rate is recorded."""

import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
inputs = router_lib.bill_inputs_recorded()
test = router_lib.read_rows("test")
messages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1

# The hosted prompt, counted with the 135M model's tokenizer as a stand-in for the provider's.
input_tokens = router_lib.router_prompt_tokens(messages)
# Prices are per million tokens, hence the division at the end.
per_message = (input_tokens * inputs["price_per_million_input_tokens"]
               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000
hosted_month = inputs["messages_per_month"] * per_message

machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]
capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]

print(f"example input, messages per month: {inputs['messages_per_month']}")
print(f"example input, price per million input tokens, USD: {inputs['price_per_million_input_tokens']}")
print(f"example input, price per million output tokens, USD: {inputs['price_per_million_output_tokens']}")
print(f"example input, output tokens per message: {inputs['output_tokens_per_message']}")
print(f"example input, machine cost per hour, USD: {inputs['machine_cost_per_hour']}")
print(f"example input, hours per month: {inputs['hours_per_month']}")
print(f"example input, utilisation: {inputs['utilisation']}")
print(f"recorded input, router messages per second, one per call: {inputs['router_messages_per_second']}")
print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")
print(f"example hosted price per message (example prices), USD: {per_message:.6f}")
print(f"example hosted bill per month (example prices), USD: {hosted_month:.2f}")
print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")
print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")
print(f"example machine cost per message at capacity (example cost per hour), USD: {machine_month / capacity:.6f}")
print(f"example volume within machine capacity: {'yes' if inputs['messages_per_month'] <= capacity else 'no'}")  # above capacity, one machine cannot carry the volume and this bill does not hold
print(f"example values, cheaper route at the input volume: {'hosted' if hosted_month < machine_month else 'machine'}")
```

Output:

```text
example input, messages per month: 100000
example input, price per million input tokens, USD: 1.0
example input, price per million output tokens, USD: 4.0
example input, output tokens per message: 6
example input, machine cost per hour, USD: 0.4
example input, hours per month: 730
example input, utilisation: 0.5
recorded input, router messages per second, one per call: 19.0
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example hosted bill per month (example prices), USD: 61.10
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example machine cost per message at capacity (example cost per hour), USD: 0.000012
example volume within machine capacity: yes
example values, cheaper route at the input volume: hosted
```

At these example inputs, the hosted route is the cheaper one. The hosted bill is 61.10 USD a month for the example volume, and a machine of your own costs 292.00 USD a month whatever the volume.

At the recorded rate and the example utilisation, the machine can route 24966000 messages a month, and at full capacity its cost per message is 0.000012 USD. The example volume of 100000 messages a month sits far below that capacity, and the break-even section finds where the two bills cross. The cost per hour in these inputs suits the recorded machine; a two-vCPU machine needs its own recorded rate.

Every input in this bill can stop being true. Prices change, so a price read in one month is a fact about that month. The utilisation here is an assumption about traffic that has not happened yet, and an inbox's volume moves with the products the bank sells and with the events that fill its queue. A machine that was idle half the month in one quarter can be busy all of the next. So count again each month, with the actual volume, the price on its date and the utilisation the machine measured. A comparison made once is a record of the month it was made in.

**Working rule:** Compare the two bills with inputs you have checked and a machine you have priced. Write each price with its source and the date you read it. Count the input tokens with the tokenizer of the hosted model you compare with, and take the throughput from a recording on the machine you would buy. Where a value is an example, say so beside it, and replace it before you decide.

## Keep the data inside the building {#keep-the-data-inside-the-building}

This section is not a cost comparison, and none of the numbers in this book decide it. Each message the inbox receives is a customer's text. When a hosted model answers it, the text leaves the bank's machines and goes to the provider's. For each message, a hosted call sends the message text and the list of 77 team names. The bank may not want that, for reasons of its own: its customers' data, its contracts, its rules for who may read what. Those reasons hold even when the hosted bill is lower.

The messages in this book are BANKING77's public messages, so the book's own runs show nothing about privacy. The requirement belongs to the bank, and only the bank can say whether it applies.

A small model that stays on the bank's machines meets the requirement by where it runs. The router, its weights and its logs stay under the bank's controls, and the versioning and file checks from [chapter 9](version-watch-and-fall-back#check-the-adapters-files-against-its-manifest) apply to every adapter the route loads. A derived copy, such as the int8 copy, has no manifest or pin of its own, so it needs both before it serves. Staying inside has a cost of its own, which is not a bill. The bank runs the machine, keeps it patched, and keeps the route's checks running, including the hand-off that stays in the route whatever else changes ([chapter 8](serve-the-router-on-a-cpu#one-route-that-returns-the-hand-off)).

A bill is a number that moves with the volume, the price and the date. A requirement does not move. So settle the requirement first, and then compare only the options that meet it. A small model that runs on the bank's machines is one such option. A larger open model that runs there is another. The stand-in is one, and on this machine it routed 0.32 messages a second one per call, against 19.0 for the classifier. That gap is its measured cost in throughput, 60.8 times over, and it is not a cost in money. If no option meets the requirement at the volume the inbox expects, the cost comparison has nothing to decide. A small model that meets the requirement still has to pass the accuracy and hand-off counts in the first section, and meeting the requirement does not excuse it from them.

**Working rule:** Settle the data requirement before the cost. Write it down as a requirement, with the people who own it, and do not count it as a saving or as a cost. When the messages must stay on the bank's machines, choose a model that runs there, and then count its accuracy and its hand-offs as in the first section of this chapter.

## Choose with a table {#choose-with-a-table}

The questions above are not all about cost, and most of them can be answered before anyone runs a recording. The table below sets the questions side by side. For each, it gives what a small router looks like, what a hosted model looks like, and what to count on your inbox.

| Question | Small router | Hosted model | What to measure |
| --- | --- | --- | --- |
| Must the messages stay on the bank's machines? | Meets it: runs on the bank's machines | Doesn't: the text goes to the provider | Requirement, not a count: decide it first |
| Must every answer be a team? | Yes, by a head or by constrained decoding ([chapter 4](prompt-and-constrain)) | Only if a prompt and a check keep it to the list | Answers outside the team list, on a fixed sample |
| How often does the team list change? | A new team needs labelled messages and a training run ([chapter 9](version-watch-and-fall-back)) | Whether a prompt change is enough: a question to test | The labelled messages and steps a new team takes to route |
| How many messages a month? | A fixed monthly bill, up to its capacity | A bill per message, growing with volume | The break-even count in the last section |
| How many labelled messages does each team have? | Enough to train on and to test each team ([chapter 2](measure-the-routing-task)) | Few or none, for a prompt | Labelled messages per team, counted on the training split |
| Does the task need open text or broad knowledge? | Limited to the team list, and its open-text answers failed in this book's first trial | A question to test | Accuracy on the same fixed sample, with its count |

The table is a list of what to count, not a score. Read its rows in order: the requirement in the first row, then the team list, then the volume, because each one can end the question before the next is asked. In the last column, a cell that begins "Requirement" names a decision that comes before the counts. Every other cell names a count, taken on your sample, with its denominator printed. The open-text row rests on the first trial in [chapter 4](prompt-and-constrain#why-free-text-fails): the 135M model's unconstrained answers were team names on 0/20 messages.

**Working rule:** Answer every row from your inbox before you choose. Where a row asks for a count, take it on a fixed sample and print its denominator. Where a row names a requirement, let the requirement decide before the counts do, and write it into the decision as a requirement.

## Know the break-even {#know-the-break-even}

The break-even is the volume at which the hosted model's monthly bill and the machine's are equal. Below it, the hosted model costs less at the inputs you set. Above it, the machine costs less, provided the volume is inside the machine's capacity. The machine's bill is paid whether it is busy or idle, so the break-even divides that bill by the hosted price per message:

break-even messages per month = machine bill per month ÷ hosted price per message

The formula does not subtract the machine's cost per message from the hosted price. The machine's cost per message at capacity is its fixed bill divided by its capacity, and that bill is already counted once in the numerator. Subtracting the same cost again would count the machine twice. The formula also holds only within capacity. Above it, one machine cannot carry the volume, and the count is made again for a second machine, with its own bill and its own capacity.

Example `break-even-from-recorded-ratio` (Python 3.12.15 · torch 2.15.0+cpu · transformers 5.23.0), passing in Edition 2026.11.9:

```python
"""The monthly volume at which one machine's bill equals the hosted bill for the router's prompts. The break-even line
is the point the prose rests on, and it holds only inside the capacity line, which the last line checks. The prices and
the machine's cost are example values, as in the bill; the router's rate is recorded on the machine."""

import math
import os

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

import torch  # noqa: E402

torch.set_num_threads(2)

import router_lib  # noqa: E402

router_lib.setup_runtime()
inputs = router_lib.bill_inputs_recorded()  # example prices and cost per hour, with the router's rate from the recording
test = router_lib.read_rows("test")  # (text, label index) for every test row, in file order
messages = [test[i][0] for i in router_lib.read_sample("test_200")]  # index 0 is the text; the label is index 1

# Mean count of the whole router prompt, team list included, with SmolLM2-135M's tokenizer.
input_tokens = router_lib.router_prompt_tokens(messages)
# Prices are per million tokens, hence the division at the end.
per_message = (input_tokens * inputs["price_per_million_input_tokens"]
               + inputs["output_tokens_per_message"] * inputs["price_per_million_output_tokens"]) / 1_000_000
machine_month = inputs["machine_cost_per_hour"] * inputs["hours_per_month"]
capacity = inputs["router_messages_per_second"] * 3600 * inputs["hours_per_month"] * inputs["utilisation"]

# The machine's bill is the same each month up to its capacity, so the hosted bill
# equals it at messages = machine bill / hosted price per message.
# Rounded up: the first whole message count at which the machine is no dearer.
breakeven = math.ceil(machine_month / per_message)

print(f"input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): {input_tokens}")
print(f"example hosted price per message (example prices), USD: {per_message:.6f}")
print(f"example machine bill per month (example cost per hour), USD: {machine_month:.2f}")
print(f"machine capacity per month (recorded rate, example utilisation): {capacity:.0f}")
print(f"example break-even messages per month (example prices and cost per hour): {breakeven}")
print(f"break-even within machine capacity: {'yes' if breakeven <= capacity else 'no'}")
```

Output:

```text
input tokens per message, hosted prompt (counted with SmolLM2-135M-Instruct, a stand-in tokenizer): 587
example hosted price per message (example prices), USD: 0.000611
example machine bill per month (example cost per hour), USD: 292.00
machine capacity per month (recorded rate, example utilisation): 24966000
example break-even messages per month (example prices and cost per hour): 477906
break-even within machine capacity: yes
```

With the example values, the machine's bill of 292.00 USD breaks even at 477906 messages a month. That count sits inside the machine's capacity of 24966000 messages a month (the check prints yes). That capacity uses the recorded one-per-call rate with the example utilisation.

The break-even does not depend on the rate, only on the bill and the hosted price. The volume of 100000 messages a month sits well below it, so the hosted bill is the lower one.

A cost point is not a decision, and the inbox still decides the rest. It answers the cost question only after the requirement in [keep the data inside the building](#keep-the-data-inside-the-building) is settled. If the messages must stay inside, the question is which small model meets the accuracy and hand-off counts, and the break-even says what that machine costs against a hosted model at each volume. If the requirement lets a hosted model in, and the volume is below the break-even, the hosted model is the lower-cost route at the example prices. Above the break-even, the machine is the lower-cost route, provided the volume is inside its capacity and the router's wrong answers reaching a caller are numbers the support staff can accept. Hand-offs go to a person whatever the bill says: on the first 200 messages, the router hands off 33/200 (16.5%) of them.

**Working rule:** Compute the break-even with your machine's bill and the hosted price you read, with that price's date and source written beside it. Check that the volume you expect sits inside the capacity you measured on your machine before you read the break-even as a saving. Above that capacity, count a second machine and compute again. Read the break-even as a cost point, not a decision. The requirements and the accuracy counts from the earlier sections still decide whether the router goes live.

## The decisions this book made for the inbox {#the-decisions-this-book-made-for-the-inbox}

On the samples and machines the book names, these are the numbers that bear on the inbox. Each one carries the chapter that counts it.

- **Accuracy.** On all of `test_1001`, the router is right on 847/1001 (84.6%), and the stand-in on 560/1001 (55.9%) ([chapter 2](measure-the-routing-task#the-baseline-from-recorded-answers)). The router hands off the rest to a person, and 76/1001 (7.6%) wrong answers still reach a caller. Those are the count the support staff must accept before the router goes live.
- **Speed.** On the recorded machine, at one message per call, the classifier ran at 19.0 messages a second against the stand-in's 0.32, a ratio of 60.8 ([chapter 7](quantize-and-measure#measure-speed-in-a-recording) says how speed is measured). The int8 copy gave no speed reason to serve: its ratio over float32 in batches of 16 was 0.9. So the float32 adapter stays the one the route serves ([chapter 8](serve-the-router-on-a-cpu#one-route-that-returns-the-hand-off)).
- **Cost.** At the example inputs, 100000 messages a month cost 61.10 USD on the hosted side and 292.00 USD on a machine of your own. The break-even is 477906 messages a month, inside a capacity of 24966000. These are example prices and an example cost per hour, so replace them with your own, each with its date and source, before you read the break-even as a decision.
- **The data requirement.** If the messages must stay on the bank's machines, the hosted model is out, and the choice is among options that run there. Settle that first, as the section on keeping the data inside the building says.
- **Where the small model is not the choice.** A team with no labelled messages has no output in the head, and a new team needs the steps in [chapter 9](version-watch-and-fall-back#retrain-on-a-trigger). Open-text answers failed in this book's first trial ([chapter 4](prompt-and-constrain#why-free-text-fails)). A hosted model is expected to do better on open text and broad knowledge, which is a question to test. And at a volume below the break-even, where a hosted model is allowed, the hosted model is the cheaper route at the example prices.
- **What was learned.** The data requirement and the team list decide which options are left. Only after that does the bill choose among them.

**The verdict for the inbox.** If the messages must stay on the bank's machines, and every team has labelled messages, a small router on the bank's own machine is the choice, provided the volume is above the break-even at the bank's own prices and inside the machine's capacity. The hand-off to a person stays in the route whatever the bill says. Count the wrong answers that reach a caller on the bank's own sample before it goes live.

**Working rule:** Write this list again from your inbox, with your numbers, before you choose. A verdict that does not come from your counts, dates and requirements is not the book's verdict for your inbox.
